Skip to content

Callback on include #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 15 commits into from
Closed

Callback on include #58

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
e7a4d4a
stable on python 2
edhoedt Aug 24, 2017
c0d40b3
Stable on python 2 and 3 (fixed utf-8 and ascii encoding issues)
edhoedt Aug 25, 2017
80c7c03
* Fixed all encoding issues
edhoedt Aug 25, 2017
06a6da4
Proper error handling
edhoedt Aug 25, 2017
f7dd253
Updating yara submodule to reference yara patched with include callba…
edhoedt Aug 27, 2017
5e3046b
Updating submodule's branch
edhoedt Aug 27, 2017
c14ca76
Updating yara submodule
edhoedt Aug 27, 2017
5b59a8d
Updating yara submodule
edhoedt Aug 27, 2017
22e3245
Updating yara submodule
edhoedt Aug 28, 2017
660f970
Updating yara submodule
edhoedt Aug 28, 2017
555103a
Submodule update
edhoedt Aug 28, 2017
c9aac36
* Fixing memory leaks
edhoedt Sep 5, 2017
fc29ed5
Merge branch 'master' of https://github.com/VirusTotal/yara-python in…
edhoedt Sep 5, 2017
5f0a8ac
making error messages order consistent between include_callback and d…
edhoedt Sep 8, 2017
2dd1f17
Removing exception printing when callback fails
edhoedt Sep 11, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[submodule "yara"]
path = yara
url=https://[email protected]/VirusTotal/yara.git
url=https://[email protected]/edhoedt/yara
branch = callback_on_include
2 changes: 1 addition & 1 deletion yara
Open in desktop
Submodule yara updated 21 files
+4 −0 README.md
+44 −0 docs/capi.rst
+5 −0 docs/writingrules.rst
+30 −0 docs/yarapython.rst
+5 −1 libyara/arena.c
+18 −17 libyara/atoms.c
+11 −0 libyara/compiler.c
+35 −28 libyara/exec.c
+1 −1 libyara/grammar.c
+1 −1 libyara/grammar.y
+1 −1 libyara/include/yara/arena.h
+14 −0 libyara/include/yara/compiler.h
+7 −0 libyara/include/yara/rules.h
+17 −0 libyara/include/yara/types.h
+139 −89 libyara/lexer.c
+106 −56 libyara/lexer.l
+3 −0 libyara/modules.c
+63 −9 libyara/parser.c
+27 −0 libyara/rules.c
+7 −4 libyara/scan.c
+38 −1 tests/test-rules.c
84 changes: 81 additions & 3 deletions yara-python.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1699,14 +1699,79 @@ void raise_exception_on_error_or_warning(

////////////////////////////////////////////////////////////////////////////////

const char* yara_include_callback(
const char* include_name,
const char* calling_rule_filename,
const char* calling_rule_namespace,
void* user_data)
{
PyObject* callback = (PyObject*) user_data;
PyObject* py_incl_name = NULL;
PyObject* py_calling_fn = NULL;
PyObject* py_calling_ns = NULL;

if (include_name != NULL)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before calling any Python function you must acquire the global interpreter lock (GIL) with PyGILState_Ensure and release it with PyGILState_Release.

{
py_incl_name = PY_STRING(include_name);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You must do a Py_DECREF(py_incl_name) when you are done with the Python string in order to let the garbage collector free the memory. The same applies to py_calling_fn and py_calling_ns.

}
else //safeguard: should never happen for 'include_name'
{
py_incl_name = Py_None;
}
if (calling_rule_filename != NULL)
{
py_calling_fn = PY_STRING(calling_rule_filename);
}
else
{
py_calling_fn = Py_None;
}
if (calling_rule_namespace != NULL)
{
py_calling_ns = PY_STRING(calling_rule_namespace);
}
else
{
py_calling_ns = Py_None;
}

PyObject* result = PyObject_CallFunctionObjArgs(callback,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The string returned by the callback is never freed. When PyObject_CallFunctionObjArgs returns the reference count for result is >=1 and your code has the ownership for that string. In order to let the garbage collector free the string you must do a Py_DECREF at some point.

The problem here is that you can't do it in yara_include_callback, because then you are returning to YARA a string that could be freed by the garbage collector at any time. But if you can't do it here, then where?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You must do Py_INCREF(callback) before PyObject_CallFunctionObjArgs and Py_DECREF(callback) after it.

py_incl_name,
py_calling_fn,
py_calling_ns,
NULL);
const char* cstring_result = NULL;

if (result != NULL && result != Py_None && PY_STRING_CHECK(result))
{
cstring_result = PY_STRING_TO_C(result);
}
else
{
PyObject* exception = PyErr_Occurred();
if (exception != NULL){
PyErr_Print();
}
else
{
PyErr_Format(PyExc_TypeError, "'include_callback' callback function must return a yara rule or rules set formated as a single ascii or unicode string");
}
cstring_result = NULL;
}

return cstring_result;
}

////////////////////////////////////////////////////////////////////////////////

static PyObject* yara_compile(
PyObject* self,
PyObject* args,
PyObject* keywords)
{
static char *kwlist[] = {
"filepath", "source", "file", "filepaths", "sources",
"includes", "externals", "error_on_warning", NULL};
"includes", "externals", "error_on_warning", "include_callback", NULL};

YR_COMPILER* compiler;
YR_RULES* yara_rules;
Expand All @@ -1723,6 +1788,7 @@ static PyObject* yara_compile(
PyObject* includes = NULL;
PyObject* externals = NULL;
PyObject* error_on_warning = NULL;
PyObject* include_callback = NULL;

Py_ssize_t pos = 0;

Expand All @@ -1736,7 +1802,7 @@ static PyObject* yara_compile(
if (PyArg_ParseTupleAndKeywords(
args,
keywords,
"|ssOOOOOO",
"|ssOOOOOOO",
kwlist,
&filepath,
&source,
Expand All @@ -1745,7 +1811,8 @@ static PyObject* yara_compile(
&sources_dict,
&includes,
&externals,
&error_on_warning))
&error_on_warning,
&include_callback))
{
error = yr_compiler_create(&compiler);

Expand Down Expand Up @@ -1791,6 +1858,17 @@ static PyObject* yara_compile(
}
}

if (include_callback != NULL)
{
if (!PyCallable_Check(include_callback))
{
return PyErr_Format(
PyExc_TypeError,
"'include_callback' must be callable");
}
yr_compiler_set_include_callback(compiler, yara_include_callback, include_callback);
}

if (externals != NULL && externals != Py_None)
{
if (PyDict_Check(externals))
Expand Down