base64 and widebase64 modifiers (VirusTotal#1185)

Add two new modifiers: base64 and widebase64.

These modifiers take the given text string and generate 3 different strings
being careful to trim off the bytes which are dependent upon leading or
trailing bytes in the larger search space.

I've implemented it by slightly cheating. I generate all the search strings in
a list and then creating one large string suitable for the RE compiler to deal
with. For example, the string "This program cannot" generates these three
base64 encoded strings:

VGhpcyBwcm9ncmFtIGNhbm5vd
RoaXMgcHJvZ3JhbSBjYW5ub3
UaGlzIHByb2dyYW0gY2Fubm90

Those three strings are then transformed into a RE that looks like this:

(VGhpcyBwcm9ncmFtIGNhbm5vd|RoaXMgcHJvZ3JhbSBjYW5ub3|UaGlzIHByb2dyYW0gY2Fubm90)

That string is then passed to the RE compiler for parsing and AST generation.
The AST is then emitted into the appropriate spot and YARA believes it was
given a regex from that point on.

I've also implemented support for specifying custom alphabets:

base64("!@#$%^&*(){}[].,|ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstu")

This means that I have to be careful to escape any special RE characters in the
string passed to the compiler. The base64 alphabet has to be 64 bytes long, and
does support escaped bytes properly too.

To avoid the need to deal with escaping I had a first implementation which
attempted to generate the AST by hand, which was mostly working but was very
cumbersome to maintain. In doing this I ended up improving yr_re_print_node()
so that it indents the tree properly, which made debugging that attempt easier.

Author: wxsBSD

docs/writingrules.rst

@@ -66,6 +66,8 @@ keywords are reserved and cannot be used as an identifier:
      - uint32be
      - wide
      - xor
+     - base64
+     - base64wide
      -
 
 Rules are generally composed of two sections: strings definition and condition.
@@ -362,7 +364,7 @@ The following rule will search for every single byte xor applied to the string
             $xor_string = "This program cannot" xor
 
         condition:
-           $xor_string
+            $xor_string
     }
 
 The above rule is logically equivalent to:
@@ -435,6 +437,65 @@ If you want more control over the range of bytes used with the xor modifier use:
 The above example will apply the bytes from 0x01 to 0xff, inclusively, to the
 string when searching. The general syntax is ``xor(minimum-maximum)``.
 
+base64 strings
+^^^^^^^^^^^^^^
+
+The ``base64`` modifier can be used to search for strings that have been base64
+encoded. A good explanation of the technique is at:
+
+https://www.leeholmes.com/blog/2019/12/10/searching-for-content-in-base-64-strings-2/
+
+The following rule will search for the three base64 permutations of the string
+"This program cannot":
+
+.. code-block:: yara
+
+    rule Base64Example1
+    {
+        strings:
+            $a = "This program cannot" base64
+
+        condition:
+            $a
+    }
+
+This will cause YARA to search for these three permutations:
+
+VGhpcyBwcm9ncmFtIGNhbm5vd
+RoaXMgcHJvZ3JhbSBjYW5ub3
+UaGlzIHByb2dyYW0gY2Fubm90
+
+The ``base64wide`` modifier works just like the base64 modifier but the results
+of the base64 modifier are converted to wide.
+
+The interaction between ``base64`` (or ``base64wide``) and ``wide`` and
+``ascii`` is as you might expect. ``wide`` and ``ascii`` are applied to the
+string first, and then the ``base64`` and ``base64wide`` modifiers are applied.
+At no point is the plaintext of the ``ascii`` or ``wide`` versions of the
+strings included in the search. If you want to also include those you can put
+them in a secondary string.
+
+The ``base64`` and ``widebas64`` modifiers also support a custom alphabet. For
+example:
+
+.. code-block:: yara
+
+    rule Base64Example2
+    {
+        strings:
+            $a = "This program cannot" base64("!@#$%^&*(){}[].,|ABCDEFGHIJ\x09LMNOPQRSTUVWXYZabcdefghijklmnopqrstu")
+
+        condition:
+            $a
+    }
+
+The alphabet must be 64 bytes long.
+
+The ``base64`` and ``base64wide`` modifiers are only supported with text
+strings. Using these modifiers with a hexadecmial string or a regular expression
+will cause a compiler error. Also, the ``xor`` and ``nocase`` modifiers used in
+combination with ``base64`` or ``base64wide`` will cause a compiler error.
+
 Searching for full words
 ^^^^^^^^^^^^^^^^^^^^^^^^
 

libyara/Makefile.am

@@ -95,6 +95,7 @@ yarainclude_HEADERS = \
 	include/yara/ahocorasick.h \
 	include/yara/arena.h \
 	include/yara/atoms.h \
+	include/yara/base64.h \
 	include/yara/bitmask.h \
 	include/yara/compiler.h \
 	include/yara/error.h \
@@ -158,6 +159,7 @@ libyara_la_SOURCES = \
 	ahocorasick.c \
 	arena.c \
 	atoms.c \
+	base64.c \
 	bitmask.c \
 	compiler.c \
 	endian.c \

libyara/atoms.c

@@ -812,7 +812,7 @@ static int _yr_atoms_case_insensitive(
 // _yr_atoms_xor
 //
 // For a given list of atoms returns another list after a single byte xor
-// has been applied to it (0x01 - 0xff).
+// has been applied to it.
 //
 
 static int _yr_atoms_xor(
@@ -1411,7 +1411,11 @@ int yr_atoms_extract_from_re(
         *atoms = NULL;
       });
 
-  if (modifier.flags & STRING_GFLAGS_WIDE)
+  // Don't do convert atoms to wide here if either base64 modifier is used.
+  // This is to avoid the situation where we have "base64 wide" because
+  // the wide has already been applied BEFORE the base64 encoding.
+  if (modifier.flags & STRING_GFLAGS_WIDE &&
+      !(modifier.flags & STRING_GFLAGS_BASE64 || modifier.flags & STRING_GFLAGS_BASE64_WIDE))
   {
     FAIL_ON_ERROR_WITH_CLEANUP(
         _yr_atoms_wide(*atoms, &wide_atoms),