Build fails when using Bison 3.2

[jasonboggess]

Attempted to install via pip on macos 10.14 using python 3.7 and pip3 version 18.0

Got the following error:

    /private/var/folders/_9/fz23mhs14q135p8kh6f4dg980000gn/T/pip-install-g5ph25wz/yaramod/build/src/yaramod/yy/yy_parser.hpp:738:102: error: too many arguments provided to function-like macro invocation
          basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);

[s3rvac]

Hi. Could you please try building yaramod and its Python bindings manually by following these two steps?

1. First, ensure that you have Flex >= 2.6 and Bison >= 3.0 and set proper paths (ensure that they exist on your system):

   # Apple ships old Flex & Bison, so Homebrew versions should be used.
   export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
   export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
   export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"
   

2. Clone the repo and run the build:

   git clone https://github.com/avast-tl/yaramod.git
   cd yaramod
   mkdir build
   cd build
   cmake -DYARAMOD_PYTHON=ON ..
   make
   

If the build fails, please post the output here. If the build passes, could you please try running pip install [..] again but after setting the above variables?

BTW, which of the following two commands did you run?

pip install git+https://github.com/avast-tl/yaramod.git
pip install yaramod

[jasonboggess]

Hi! Thanks for the super fast reply. I did the above steps (cloned the repo and tried to build manually) and still got the same errors.

Used homebrew to install flex and bison.

Here are what versions I'm using and what my variables ended up to be:

$ flex --version
flex 2.6.4
$ bison --version
bison (GNU Bison) 3.2
Written by Robert Corbett and Richard Stallman.

Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ echo $CMAKE_INCLUDE_PATH
/usr/local/opt/flex/include
$ echo $CMAKE_LIBRARY_PATH
/usr/local/opt/flex/lib;/usr/local/opt/bison/lib
$ cmake --version
cmake version 3.12.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

And when running make:

$ make
[  3%] Building CXX object src/CMakeFiles/yaramod.dir/builder/yara_file_builder.cpp.o
In file included from yaramod/src/builder/yara_file_builder.cpp:8:
In file included from yaramod/include/yaramod/parser/parser_driver.h:13:
In file included from yaramod/include/yaramod/parser/lexer.h:13:
yaramod/build/src/yaramod/yy/yy_parser.hpp:738:102: error: too many arguments provided to function-like macro invocation
      basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                                                                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:80:10: note: macro 'YY_RVREF' defined here
# define YY_RVREF(Type)    Type&&
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:738:49: error: unknown type name 'YY_RVREF'
      basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1209:76: error: too many arguments provided to function-like macro invocation
    make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l);
                                                                           ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:81:10: note: macro 'YY_COPY' defined here
# define YY_COPY(Type)     Type
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1209:24: error: unknown type name 'YY_COPY'
    make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l);
                       ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1632:128: error: too many arguments provided to function-like macro invocation
   Parser ::basic_symbol<Base>::basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l)
                                                                                                                               ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:80:10: note: macro 'YY_RVREF' defined here
# define YY_RVREF(Type)    Type&&
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1632:75: error: unknown type name 'YY_RVREF'
   Parser ::basic_symbol<Base>::basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l)
                                                                          ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2717:84: error: too many arguments provided to function-like macro invocation
   Parser ::make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l)
                                                                                   ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:81:10: note: macro 'YY_COPY' defined here
# define YY_COPY(Type)     Type
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2717:32: error: unknown type name 'YY_COPY'
   Parser ::make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l)
                               ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1484:7: error: no matching constructor for initialization of 'yaramod::yy::Parser::by_type'
    : Base (YY_MOVE (other))
      ^     ~~~~~~~~~~~~~~~
yaramod/build/src/yaramod/yy/yy_parser.hpp:2103:12: note: in instantiation of member function 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>::basic_symbol' requested here
    return symbol_type (token::END, YY_MOVE (l));
           ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2043:22: note: candidate constructor not viable: no known conversion from 'typename remove_reference<basic_symbol<by_type> &>::type' (aka 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>') to 'const yaramod::yy::Parser::by_type' for 1st argument
   Parser ::by_type::by_type (const by_type& other)
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2048:22: note: candidate constructor not viable: no known conversion from 'typename remove_reference<basic_symbol<by_type> &>::type' (aka 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>') to 'yaramod::yy::Parser::token_type' (aka 'yaramod::yy::Parser::token::yytokentype') for 1st argument
   Parser ::by_type::by_type (token_type t)
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2038:22: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
   Parser ::by_type::by_type ()
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1488:13: error: cannot initialize object parameter of type 'const yaramod::yy::Parser::by_type' with an expression of type 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>'
    switch (other.type_get ())
            ^~~~~
10 errors generated.
make[2]: *** [src/CMakeFiles/yaramod.dir/builder/yara_file_builder.cpp.o] Error 1
make[1]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make: *** [all] Error 2

FYI I ran:

pip3 install yaramod

[metthal]

I am not sure if it'll help because the errors are from bison generated parser, but I see some nonstd::optional in the output so I tried to update optional-lite to the newest version in new-optional-lite branch. Can you please try to compile this branch? Thanks.

[jasonboggess]

@metthal I deleted the build directory, and checked out your branch, but same errors.

[metthal]

It may have something to do with this. From what I see, you use bison 3.2 which was rolled out just few days ago so this might be some bug in the new release which showed itself only with our parser definition. If this is an option for you, could you please try to revert bison to 3.1 or 3.0.x? I'll try to upgrade to bison 3.2 on my machine as soon as I get to it. If it's really problem in the new release, we might need to file a bug there.

[s3rvac]

@metthal Good catch. The issue is really caused by Bison 3.2. I was able to reproduce it even on Ubuntu inside Docker via the following Dockerfile:

FROM ubuntu:bionic

RUN apt-get -y update && \
    apt-get install -y build-essential cmake git flex wget

RUN wget https://ftp.gnu.org/gnu/bison/bison-3.2.tar.gz && \
    tar xf bison-3.2.tar.gz && \
    cd bison-3.2 && \
    ./configure && \
    make -j$(nproc) && \
    make install

RUN git clone https://github.com/avast-tl/yaramod && \
    mkdir yaramod/build && \
    cd yaramod/build && \
    cmake .. && \
    make -j$(nproc)

When you run

$ docker build -t yaramod .

the build fails with

In file included from /yaramod/include/yaramod/parser/lexer.h:13:0,
                 from /yaramod/include/yaramod/parser/parser_driver.h:13,
                 from /yaramod/src/builder/yara_file_builder.cpp:8:
/yaramod/build/src/yaramod/yy/yy_parser.hpp:738:134: error: macro "YY_RVREF" passed 2 arguments, but takes just 1
       basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                                                                                                      ^
[..]

[jasonboggess]

I manually installed 3.1.0 and 3.0.5 of bison and I still see the same issue. I made sure that bison --version was correct and that my env variables were set:

export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

[s3rvac]

This is weird. With 3.1.0 and 3.0.5, the build succeeds for me. It only fails with 3.2.0.