Edhoedt callback on include (#749)

* Full structure implemented, callback available. TODO: actually use the callback's result to import rules files.

* Importing files via external callback fully implemented and working

* Removing useless comments

* Adding documentation

* Updating doc for include callbacks

* Fixing compiler_set_include_callback's signature

* Fixing issues according to @plusvic 's advice
 * Limited lines to 80 chars
 * include callback is now using its own user_data and does not conflict with other callbacks
 * Copying include callback result to the stack before pushing it into the flex scanner
 * yara's memory management functions are now exposed through the API
 * Making the lexer reponsible for freeing the memory allocated by the include callback
 * Updating docs accordingly

* * fixing memory management issues in callbacks
* adding a callback to allow external code to free the memory used to return the include callback's result
* API: un-exposing yara's memory management functions
* making include callbacks the only way ton include files
* adding a default include callback to open files from the disk (mimicking original behavior)
* updating docs accordingly

* fixing missing pointer cast

* re-generating lexer.c

* removing imports ununsed since commit 92877db

* fixing default file read callback

* Fixing line numbers in errors

* Fix some issues related to includes callback.

* Remove allow_includes field from YR_COMPILER. Includes now can be disallowed by setting the include callback to NULL.
* Fix memory and file handle leaks.
* Calculate size of error message buffer correctly.
* Add test case.

Author: plusvic

docs/capi.rst

@@ -63,6 +63,40 @@ contains the file name and line number where the error or warning occurs.
 you're using :c:func:`yr_compiler_add_string`. The ``user_data`` pointer is the
 same you passed to :c:func:`yr_compiler_set_callback`.
 
+By default, for rules containing references to other files
+(``include "filename.yara"``), yara will try to find those files on disk.
+However, if you want to fetch the imported rules from another source (eg: from a
+database or remote service), a callback function can be set with
+:c:func:`yr_compiler_set_include_callback`.
+The callback receives the following parameters:
+ * ``include_name``: name of the requested file.
+ * ``calling_rule_filename``: the requesting file name (NULL if not a file).
+ * ``calling_rule_namespace``: namespace (NULL if undefined).
+ * ``user_data`` pointer is the same you passed to :c:func:`yr_compiler_set_include_callback`.
+It should return the requested file's content as a string. The memory for this
+string should be allocated by the callback function. Once it is safe to free the
+memory used to return the callback's result, the include_free function passed to
+:c:func:`yr_compiler_set_include_callback` will be called. If the memory does
+not need to be freed, NULL can be passed as include_free instead.
+
+The callback function has the following prototype:
+
+.. code-block:: c
+
+  const char* include_callback(
+      const char* include_name,
+      const char* calling_rule_filename,
+      const char* calling_rule_namespace,
+      void* user_data);
+
+The free function has the following prototype:
+
+.. code-block:: c
+
+  void include_free(
+      const char* callback_result_ptr,
+      void* user_data);
+
 After you successfully added some sources you can get the compiled rules
 using the :c:func:`yr_compiler_get_rules()` function. You'll get a pointer to
 a :c:type:`YR_RULES` structure which can be used to scan your data as
@@ -402,6 +436,15 @@ Functions
   pointer is passed to the callback function.
 
 
+.. c:function:: void yr_compiler_set_include_callback(YR_COMPILER* compiler, YR_COMPILER_INCLUDE_CALLBACK_FUNC callback, YR_COMPILER_INCLUDE_FREE_FUNC include_free, void* user_data)
+
+  Set a callback to provide rules from a custom source when ``include``
+  directive is invoked. The *user_data* pointer is untouched and passed back to
+  the callback function and to the free function. Once the callback's result
+  is no longer needed, the include_free function will be called. If the memory
+  does not need to be freed, include_free can be set to NULL.
+
+
 .. c:function:: int yr_compiler_add_file(YR_COMPILER* compiler, FILE* file, const char* namespace, const char* file_name)
 
   Compile rules from a *file*. Rules are put into the specified *namespace*,
@@ -668,6 +711,7 @@ Functions
   Enables the specified rule. After being disabled with :c:func:`yr_rule_disable`
   a rule can be enabled again by using this function.
 
+
 Error codes
 -----------
 

docs/yarapython.rst

@@ -74,6 +74,36 @@ should be accepted in the source files, for example:
 If the source file contains include directives the previous line would raise
 an exception.
 
+If includes are used, a python callback can be set to define a custom source for
+the imported files (by default they are read from disk). This callback function
+is set through the ``include_callback`` optional parameter.
+It receives the following parameters:
+ *``requested_filename``: file requested with 'include'
+ *``filename``: file containing the 'include' directive if applicable, else None
+ *``namespace``: namespace
+And returns the requested rules sources as a single string.
+
+.. code-block:: python
+  import yara
+  import sys
+  if sys.version_info >= (3, 0):
+      import urllib.request as urllib
+  else:
+      import urllib as urllib
+
+  def mycallback(requested_filename, filename, namespace):
+      if requested_filename == 'req.yara':
+          uf = urllib.urlopen('https://pastebin.com/raw/siZ2sMTM')
+          sources = uf.read()
+          if sys.version_info >= (3, 0):
+              sources = str(sources, 'utf-8')
+          return sources
+      else:
+          raise Exception(filename+": Can't fetch "+requested_filename)
+
+  rules = yara.compile(source='include "req.yara" rule r{ condition: true }',
+                      include_callback=mycallback)
+
 If you are using external variables in your rules you must define those
 external variables either while compiling the rules, or while applying the
 rules to some file. To define your variables at the moment of compilation you

libyara/compiler.c

@@ -56,14 +56,14 @@ YR_API int yr_compiler_create(
 
   new_compiler->errors = 0;
   new_compiler->callback = NULL;
+  new_compiler->include_callback = _yr_compiler_default_include_callback;
+  new_compiler->include_free = _yr_compiler_default_include_free;
   new_compiler->last_error = ERROR_SUCCESS;
   new_compiler->last_error_line = 0;
   new_compiler->current_line = 0;
   new_compiler->last_result = ERROR_SUCCESS;
-  new_compiler->file_stack_ptr = 0;
   new_compiler->file_name_stack_ptr = 0;
   new_compiler->fixup_stack_head = NULL;
-  new_compiler->allow_includes = 1;
   new_compiler->loop_depth = 0;
   new_compiler->loop_for_of_mem_offset = -1;
   new_compiler->compiled_rules_arena = NULL;
@@ -182,38 +182,126 @@ YR_API void yr_compiler_set_callback(
 }
 
 
-int _yr_compiler_push_file(
-    YR_COMPILER* compiler,
-    FILE* fh)
+const char* _yr_compiler_default_include_callback(
+    const char* include_name,
+    const char* calling_rule_filename,
+    const char* calling_rule_namespace,
+    void* user_data)
 {
-  if (compiler->file_stack_ptr < MAX_INCLUDE_DEPTH)
+  char* buffer;
+  char* s = NULL;
+  #ifdef _WIN32
+  char* b = NULL;
+  #endif
+  char* f;
+  FILE* fh;
+  char* file_buffer;
+  size_t file_length;
+
+  if (calling_rule_filename != NULL)
   {
-    compiler->file_stack[compiler->file_stack_ptr] = fh;
-    compiler->file_stack_ptr++;
-    return ERROR_SUCCESS;
+    buffer = (char*) calling_rule_filename;
   }
   else
   {
-    compiler->last_result = ERROR_INCLUDE_DEPTH_EXCEEDED;
-    return ERROR_INCLUDE_DEPTH_EXCEEDED;
+    buffer = "\0";
+  }
+
+  s = strrchr(buffer, '/');
+
+  #ifdef _WIN32
+  b = strrchr(buffer, '\\'); // in Windows both path delimiters are accepted
+  #endif
+
+  #ifdef _WIN32
+  if (s != NULL || b != NULL)
+  #else
+  if (s != NULL)
+  #endif
+  {
+    #ifdef _WIN32
+    f = (b > s)? (b + 1): (s + 1);
+    #else
+    f = s + 1;
+    #endif
+
+    strlcpy(f, include_name, sizeof(buffer) - (f - buffer));
+
+    f = buffer;
+    // SECURITY: Potential for directory traversal here.
+    fh = fopen(buffer, "rb");
+
+    // if include file was not found relative to current source file,
+    // try to open it with path as specified by user (maybe user wrote
+    // a full path)
+    if (fh == NULL)
+    {
+      f = (char*) include_name;
+      // SECURITY: Potential for directory traversal here.
+      fh = fopen(include_name, "rb");
+    }
+  }
+  else
+  {
+    f = (char*) include_name;
+    // SECURITY: Potential for directory traversal here.
+    fh = fopen(include_name, "rb");
+  }
+
+  if (fh == NULL)
+    return NULL;
+
+  fseek(fh, 0, SEEK_END);
+  file_length = ftell(fh);
+  fseek(fh, 0, SEEK_SET);
+
+  file_buffer = (char*) yr_malloc(file_length + 1);
+
+  if (file_buffer == NULL)
+  {
+    fclose(fh);
+    return NULL;
+  }
+
+  if (file_length != fread(file_buffer, 1, file_length, fh))
+  {
+    yr_free(file_buffer);
+    fclose(fh);
+    return NULL;
+  }
+  else
+  {
+    file_buffer[file_length]='\0';
   }
+
+  fclose(fh);
+  return file_buffer;
 }
 
 
-FILE* _yr_compiler_pop_file(
-    YR_COMPILER* compiler)
+void _yr_compiler_default_include_free(
+  const char* callback_result_ptr,
+  void* user_data)
 {
-  FILE* result = NULL;
-
-  if (compiler->file_stack_ptr > 0)
+  if(callback_result_ptr != NULL)
   {
-    compiler->file_stack_ptr--;
-    result = compiler->file_stack[compiler->file_stack_ptr];
+    yr_free((void*)callback_result_ptr);
   }
+}
 
-  return result;
+
+YR_API void yr_compiler_set_include_callback(
+    YR_COMPILER* compiler,
+    YR_COMPILER_INCLUDE_CALLBACK_FUNC include_callback,
+    YR_COMPILER_INCLUDE_FREE_FUNC include_free,
+    void* user_data)
+{
+  compiler->include_callback = include_callback;
+  compiler->include_free = include_free;
+  compiler->incl_clbk_user_data = user_data;
 }
 
+
 int _yr_compiler_push_file_name(
     YR_COMPILER* compiler,
     const char* file_name)

libyara/include/yara/compiler.h

@@ -52,6 +52,17 @@ typedef void (*YR_COMPILER_CALLBACK_FUNC)(
     void* user_data);
 
 
+typedef const char* (*YR_COMPILER_INCLUDE_CALLBACK_FUNC)(
+    const char* include_name,
+    const char* calling_rule_filename,
+    const char* calling_rule_namespace,
+    void* user_data);
+
+typedef void (*YR_COMPILER_INCLUDE_FREE_FUNC)(
+    const char* callback_result_ptr,
+    void* user_data );
+
+
 typedef struct _YR_FIXUP
 {
   void* address;
@@ -98,14 +109,9 @@ typedef struct _YR_COMPILER
   int               loop_depth;
   int               loop_for_of_mem_offset;
 
-  int               allow_includes;
-
   char*             file_name_stack[MAX_INCLUDE_DEPTH];
   int               file_name_stack_ptr;
 
-  FILE*             file_stack[MAX_INCLUDE_DEPTH];
-  int               file_stack_ptr;
-
   char              last_error_extra_info[MAX_COMPILER_ERROR_EXTRA_INFO];
 
   char              lex_buf[LEX_BUF_SIZE];
@@ -114,8 +120,12 @@ typedef struct _YR_COMPILER
 
   char              include_base_dir[MAX_PATH];
   void*             user_data;
+  void*             incl_clbk_user_data;
 
   YR_COMPILER_CALLBACK_FUNC  callback;
+  YR_COMPILER_INCLUDE_CALLBACK_FUNC include_callback;
+  YR_COMPILER_INCLUDE_FREE_FUNC include_free;
+
 
 } YR_COMPILER;
 
@@ -134,14 +144,6 @@ typedef struct _YR_COMPILER
         fmt, __VA_ARGS__);
 
 
-int _yr_compiler_push_file(
-    YR_COMPILER* compiler,
-    FILE* fh);
-
-
-FILE* _yr_compiler_pop_file(
-    YR_COMPILER* compiler);
-
 
 int _yr_compiler_push_file_name(
     YR_COMPILER* compiler,
@@ -151,6 +153,15 @@ int _yr_compiler_push_file_name(
 void _yr_compiler_pop_file_name(
     YR_COMPILER* compiler);
 
+const char* _yr_compiler_default_include_callback(
+    const char* include_name,
+    const char* calling_rule_filename,
+    const char* calling_rule_namespace,
+    void* user_data);
+
+void _yr_compiler_default_include_free(
+    const char* callback_result_ptr,
+    void* user_data);
 
 YR_API int yr_compiler_create(
     YR_COMPILER** compiler);
@@ -166,6 +177,13 @@ YR_API void yr_compiler_set_callback(
     void* user_data);
 
 
+YR_API void yr_compiler_set_include_callback(
+    YR_COMPILER* compiler,
+    YR_COMPILER_INCLUDE_CALLBACK_FUNC include_callback,
+    YR_COMPILER_INCLUDE_FREE_FUNC include_free,
+    void* user_data);
+
+
 YR_API int yr_compiler_add_file(
     YR_COMPILER* compiler,
     FILE* rules_file,