Skip to content

Synchronise modules (issue #46) #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jan 16, 2020
Merged

Synchronise modules (issue #46) #51

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d08f600
remove symbol signature.name for it has been deleted from downstream
TadeasKucera Jan 8, 2020
7aa6065
Add missing math.min and math.max symbols
TadeasKucera Jan 8, 2020
8b3af84
Add NeededSymbols to module.h to delegate whether to include avast/tv…
TadeasKucera Jan 8, 2020
d21c1f3
Use NeededSymbols: ParserDriver now offers to skip some symbols when …
TadeasKucera Jan 8, 2020
f86ea36
Synchronise symbols in modules with yara upstream
TadeasKucera Jan 8, 2020
84046ba
Add module resetting and tests of avast-specific symbols imports
TadeasKucera Jan 9, 2020
b5ef2d3
Allow to skip VirusTotal-specific symbols (those in yara_file.cpp)
TadeasKucera Jan 13, 2020
ea31ce2
Improve tests of phish and androguard modules
TadeasKucera Jan 13, 2020
591e849
Use enum Features instead of bool avastSpecific and vtSpecific
TadeasKucera Jan 16, 2020
7afaecc
Instead of having global modules, store modules in each ParserDriver …
TadeasKucera Jan 16, 2020
7df3108
Fix Syntax and Typos
TadeasKucera Jan 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion include/yaramod/builder/yara_file_builder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,12 @@ class YaraFileBuilder
public:
/// @name Constructors
/// @{
YaraFileBuilder()
YaraFileBuilder() : YaraFileBuilder(true, true) {}

YaraFileBuilder(bool avastSpecific, bool vtSpecific)
: _tokenStream(std::make_shared<TokenStream>())
, _avastSpecific(avastSpecific)
, _vtSpecific(vtSpecific)
{
}
/// @}
Expand All @@ -60,6 +64,8 @@ class YaraFileBuilder
std::shared_ptr<TokenStream> _tokenStream; ///< Tokens storage
std::vector<TokenIt> _modules; ///< Modules
std::vector<std::shared_ptr<Rule>> _rules; ///< Rules
bool _avastSpecific;
bool _vtSpecific;
};

}
8 changes: 5 additions & 3 deletions include/yaramod/parser/parser_driver.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,9 +253,9 @@ class ParserDriver
/// @name Constructors
/// @{
ParserDriver() = delete;
ParserDriver(ParserMode parserMode);
explicit ParserDriver(const std::string& filePath, ParserMode parserMode = ParserMode::Regular);
explicit ParserDriver(std::istream& input, ParserMode parserMode = ParserMode::Regular);
ParserDriver(ParserMode parserMode, bool avastSpecific = true, bool vtSpecific = true);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be best to avoid just boolean parameters if it's unclear from the context of the function name of what they suppose to mean. Calls to such functions/methods then look like this ParserDriver(..., true, false). What exactly those true and false mean? The best way to do it would be through bitmasks because then you can do something like ParserDriver(..., Features::VirusTotal | Features::Avast).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I implemented this in the last commit.

explicit ParserDriver(const std::string& filePath, ParserMode parserMode = ParserMode::Regular, bool avastSpecific = true, bool vtSpecific = true);
explicit ParserDriver(std::istream& input, ParserMode parserMode = ParserMode::Regular, bool avastSpecific = true, bool vtSpecific = true);
void initialize();
/// @}

Expand Down Expand Up @@ -369,6 +369,8 @@ class ParserDriver
bool _escapedContent = false; ///< flag used to determine if a currently parsed literal contains hexadecimal byte (such byte must be unescaped in getPureText())

ParserMode _mode; ///< Parser mode.
bool _avastSpecific; ///< Used to determine whether to include Avast-specific symbols or skip them
bool _vtSpecific; ///< Used to determine whether to include VirusTotal-specific symbols or skip them

std::stack<std::shared_ptr<TokenStream>> _tokenStreams; ///< _tokenStream contains all parsed tokens
std::stack<Location> _locations; ///< the top location tracks position of currently parsed token within current input file
Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/androguard_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class AndroguardModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/cuckoo_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class CuckooModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/dex_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class DexModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/dotnet_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class DotnetModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/elf_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class ElfModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/hash_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class HashModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/macho_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class MachoModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/magic_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class MagicModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/math_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class MathModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
5 changes: 3 additions & 2 deletions include/yaramod/types/modules/module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ class Module

/// @name Pure virtual initialization method
/// @{
virtual bool initialize() = 0;
virtual bool initialize(bool avastSpecific) = 0;
/// @}

/// @name Getter methods
Expand All @@ -48,7 +48,8 @@ class Module

/// @name Static module loading
/// @{
static std::shared_ptr<Module> load(const std::string& name);
static void reset(const std::string& name);
static std::shared_ptr<Module> load(const std::string& name, bool avastSpecific);
/// @}

protected:
Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/pe_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class PeModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/phish_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class PhishModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
2 changes: 1 addition & 1 deletion include/yaramod/types/modules/time_module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class TimeModule : public Module

/// @name Initialization method
/// @{
virtual bool initialize() override;
virtual bool initialize(bool avastSpecific) override;
/// @}
};

Expand Down
6 changes: 3 additions & 3 deletions include/yaramod/types/yara_file.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,12 @@ class YaraFile

/// @name Addition methods
/// @{
bool addImport(TokenIt import);
bool addImport(TokenIt import, bool avastSpecific);
void addRule(Rule&& rule);
void addRule(std::unique_ptr<Rule>&& rule);
void addRule(const std::shared_ptr<Rule>& rule);
void addRules(const std::vector<std::shared_ptr<Rule>>& rules);
bool addImports(const std::vector<TokenIt>& imports);
bool addImports(const std::vector<TokenIt>& imports, bool avastSpecific);
void insertRule(std::size_t position, std::unique_ptr<Rule>&& rule);
void insertRule(std::size_t position, const std::shared_ptr<Rule>& rule);
/// @}
Expand Down Expand Up @@ -77,7 +77,7 @@ class YaraFile

/// @name Symbol methods
/// @{
std::shared_ptr<Symbol> findSymbol(const std::string& name) const;
std::shared_ptr<Symbol> findSymbol(const std::string& name, bool vtSpecific) const;
/// @}

/// @name Detection methods
Expand Down
2 changes: 1 addition & 1 deletion src/builder/yara_file_builder.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ namespace yaramod {
std::unique_ptr<YaraFile> YaraFileBuilder::get(bool recheck, ParserDriver* external_driver)
{
auto yaraFile = std::make_unique<YaraFile>(std::move(_tokenStream));
yaraFile->addImports(_modules);
yaraFile->addImports(_modules, _avastSpecific);
yaraFile->addRules(_rules);

_modules.clear();
Expand Down
28 changes: 21 additions & 7 deletions src/parser/parser_driver.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,7 +418,7 @@ void ParserDriver::defineGrammar()
.production("IMPORT_KEYWORD", "STRING_LITERAL", [&](auto&& args) -> Value {
TokenIt import = args[1].getTokenIt();
import->setType(IMPORT_MODULE);
if (!_file.addImport(import))
if (!_file.addImport(import, _avastSpecific))
error_handle(import->getLocation(), "Unrecognized module '" + import->getString() + "' imported");
return {};
})
Expand Down Expand Up @@ -1517,8 +1517,14 @@ void ParserDriver::initialize()
* Constructor.
*
* @param parserMode Parsing mode.
* @param avastSpecific set iff we want to use aditional Avast-specific symbols in the imported modules
* @param vtSpecific set iff we want to use aditional VirusTotal-specific symbols in the imported modules
* If you need to use more instances of ParserDriver with different avastSpecific or vtSpecific flags, use
* Module::reset method appropriately.
*/
ParserDriver::ParserDriver(ParserMode parserMode)
ParserDriver::ParserDriver(ParserMode parserMode, bool avastSpecific, bool vtSpecific)
: _avastSpecific(avastSpecific)
, _vtSpecific(vtSpecific)
{
reset(parserMode);
initialize();
Expand All @@ -1529,9 +1535,13 @@ ParserDriver::ParserDriver(ParserMode parserMode)
*
* @param filePath Input file path.
* @param parserMode Parsing mode.
* @param avastSpecific set iff we want to use aditional Avast-specific symbols in the imported modules
* @param vtSpecific set iff we want to use aditional VirusTotal-specific symbols in the imported modules
* If you need to use more instances of ParserDriver with different avastSpecific or vtSpecific flags, use
* Module::reset method appropriately.
*/
ParserDriver::ParserDriver(const std::string& filePath, ParserMode parserMode) : _mode(parserMode),
_valid(true), _filePath(), _currentStrings(), _stringLoop(false), _localSymbols(), _startOfRule(0), _anonStringCounter(0)
ParserDriver::ParserDriver(const std::string& filePath, ParserMode parserMode, bool avastSpecific, bool vtSpecific) : _mode(parserMode), _avastSpecific(avastSpecific)
, _vtSpecific(vtSpecific), _valid(true), _filePath(), _currentStrings(), _stringLoop(false), _localSymbols(), _startOfRule(0), _anonStringCounter(0)
{
initialize();
_tokenStreams.emplace(std::make_shared<TokenStream>());
Expand All @@ -1546,9 +1556,13 @@ ParserDriver::ParserDriver(const std::string& filePath, ParserMode parserMode) :
*
* @param input Input stream.
* @param parserMode Parsing mode.
* @param avastSpecific set iff we want to use aditional Avast-specific symbols in the imported modules
* @param vtSpecific set iff we want to use aditional VirusTotal-specific symbols in the imported modules
* If you need to use more instances of ParserDriver with different avastSpecific or vtSpecific flags, use
* Module::reset method appropriately.
*/
ParserDriver::ParserDriver(std::istream& input, ParserMode parserMode) : _mode(parserMode),
_optionalFirstInput(&input), _valid(true), _filePath(), _currentStrings(), _stringLoop(false), _localSymbols()
ParserDriver::ParserDriver(std::istream& input, ParserMode parserMode, bool avastSpecific, bool vtSpecific) : _mode(parserMode), _avastSpecific(avastSpecific)
, _vtSpecific(vtSpecific), _optionalFirstInput(&input), _valid(true), _filePath(), _currentStrings(), _stringLoop(false), _localSymbols()
{
initialize();
_tokenStreams.emplace(std::make_shared<TokenStream>());
Expand Down Expand Up @@ -1828,7 +1842,7 @@ std::shared_ptr<Symbol> ParserDriver::findSymbol(const std::string& name) const
if (itr != _localSymbols.end())
return itr->second;

return _file.findSymbol(name);
return _file.findSymbol(name, _vtSpecific);
}

/**
Expand Down
3 changes: 2 additions & 1 deletion src/types/modules/androguard_module.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@ AndroguardModule::AndroguardModule() : Module("androguard")
*
* @return @c true if success, otherwise @c false.
*/
bool AndroguardModule::initialize()
bool AndroguardModule::initialize(bool avastSpecific)
{
using Type = Expression::Type;
(void) avastSpecific;

auto androguardStruct = std::make_shared<StructureSymbol>("androguard");

Expand Down
Loading