Merge pull request #401 from fandango-fuzzer/dev

Dev

Author: joszamama

docs/Conversion.md

@@ -11,19 +11,21 @@ kernelspec:
 ---
 
 (sec:conversion)=
-# Conversion and Compression
+# Data Conversions
 
 When defining a complex input format, some parts may be the result of applying an _operation_ on another, more structured part.
-Most importantly, content may be _encoded_, _compressed_, or _sanitized_.
+Most importantly, content may be _encoded_, _compressed_, or _converted_.
 
-Fandango uses a special form of [generators](sec:generators) to handle these, namely generators with _symbols_.
+Fandango uses a special form of [generators](sec:generators) to handle these, called _converters_.
+These are generator expressions with _symbols_, mostly functions that take symbols as arguments.
 Let's have a look at how these work.
 
 
-## Encoding Data
+## Encoding Data During Fuzzing
 
 In Fandango, a [generator](sec:generators) expression can contain _symbols_ (enclosed in `<...>`) as elements.
-When fuzzing, this has the effect of Fandango using the grammar to
+Such generators are called _converters_.
+When fuzzing, converters have the effect of Fandango using the grammar to
 
 * instantiate each symbol from the grammar,
 * evaluate the resulting expression, and
@@ -45,7 +47,8 @@ Of course, these can be decoded again:
 base64.b64decode(encoded)
 ```
 
-Let us assume we have a `<data>` field that contains a number of bytes:
+Let us make use of these functions.
+Assume we have a `<data>` field that contains a number of bytes:
 
 ```{code-cell}
 :tags: ["remove-input"]
@@ -73,10 +76,202 @@ In a third step, we embed the `<item>` into a (binary) string:
 !grep '^<start>' encode.fan
 ```
 
-The resulting [`encode.fan`](encode.fan) spec allows us to encode and embed binary data:
+The full resulting [`encode.fan`](encode.fan) spec looks like this:
 
 ```{code-cell}
-!fandango fuzz -f encode.fan -n 1
+:tags: ["remove-input"]
+!cat encode.fan
+```
+
+With this, we can encode and embed binary data:
+
+```shell
+$ fandango fuzz -f encode.fan -n 1
+```
+
+```{code-cell}
+:tags: ["remove-input"]
+!fandango fuzz -f encode.fan -n 1 --random-seed 7
+assert _exit_code == 0
+```
+
+In the same vein, one can use functions for compressing data or any other kind of conversion.
+
+
+## Sources, Encoders, and Constraints
+
+When Fandango produces an input using a generator, it _saves_ the generated arguments as a _source_ in the produced derivation tree.
+Sources become visible as soon as the input is shown as a grammar:
+
+```shell
+$ fandango fuzz -f encode.fan -n 1 --format=grammar
 ```
 
-In the same vein, one can use functions for compressing data or any other kind of conversion.
+```{code-cell}
+:tags: ["remove-input"]
+!fandango fuzz -f encode.fan -n 1 --format=grammar --random-seed 7
+assert _exit_code == 0
+```
+
+In the definition of `<item>`, we see a generic converter `f(<data>)` as well as the definition of `<data>` that went into the generator.
+(The actual generator code, `base64.b64encode(bytes(<data>))`, is not saved in the derivation tree.)
+
+We can visualize the resulting tree, using a double arrow between `<item>` and its source `<data>`, indicating that their values depend on each other:
+
+```{code-cell}
+:tags: ["remove-input"]
+from Tree import Tree
+
+tree = Tree('<start>',
+  Tree(b'Data: '),
+  Tree('<item>',
+    Tree(b'RmFuZGFuZ29Nyhg='),
+    sources=[
+      Tree('<data>',
+        Tree(b'Fandango'),
+        Tree('<byte>', Tree('<_byte>', Tree(b'M'))),
+        Tree('<byte>', Tree('<_byte>', Tree(b'\xca'))),
+        Tree('<byte>', Tree('<_byte>', Tree(b'\x18')))
+      ),
+    ]
+  )
+)
+tree.visualize()
+```
+
+Since sources like `<data>` are preserved, we can use them in [constraints](sec:constraints).
+For instance, we can produce a string with specific values for `<data>`:
+
+```shell
+$ fandango fuzz -f encode.fan -n 1 -c '<data> == b"Fandango author"'
+```
+
+```{code-cell}
+:tags: ["remove-input"]
+!fandango fuzz -f encode.fan -n 1 -c '<data> == b"Fandango author"' --population-size 1
+assert _exit_code == 0
+```
+
+Is this string a correct encoding of a correct string?
+We will see in the next section.
+
+
+## Decoding Parsed Data
+
+So far, we can only _encode_ data during fuzzing.
+But what if we also want to _decode_ data, say during [parsing](sec:parsing)?
+Our `encode.fan` will help us _parse_ the data, but not decode it:
+
+```shell
+$ echo -n 'Data: RmFuZGFuZ28gYXV0aG9y' | fandango parse -f encode.fan
+```
+
+```{code-cell}
+:tags: ["remove-input"]
+!echo -n 'Data: RmFuZGFuZ28gYXV0aG9y' | fandango parse -f encode.fan
+assert _exit_code == 1
+```
+
+The fact that parsing fails is not a big surprise, as we only have specified an _encoder_, but not a _decoder_.
+As the error message suggests, we need to add a generator for `<data>` - a decoder that converts `<item>` elements into `<data>`.
+
+We can achieve this by providing a generator for `<data>` that builds on `<item>`:
+
+```{code-cell}
+:tags: ["remove-input"]
+!grep '^<data>' encode-decode.fan
+```
+
+Here, `base64.b64decode(bytes(<item>))` takes an `<item>` (which is previously parsed) and decodes it.
+The decoded result is parsed and placed in `<data>`.
+
+The resulting [`encode-decode.fan`](encode-decode.fan) file now looks like this:
+
+```{code-cell}
+:tags: ["remove-input"]
+!cat encode-decode.fan
+```
+
+```{margin}
+Fandango allows generators in both directions so one `.fan` file can be used for fuzzing and parsing.
+```
+
+If this looks like a mutual recursive definition, that is because it is.
+During fuzzing and parsing, Fandango tracks the _dependencies_ between generators and uses them to decide which generators to use first:
+
+* When fuzzing, Fandango operates _top-down_, starting with the topmost generator encountered; their arguments are _produced_.
+  In our case, this is the `<item>` generator, generating a value for `<data>`.
+* When parsing, Fandango operates _bottom-up_, starting with the lowest generators encountered; their arguments are _parsed_.
+  In our case, this is the `<data>` generator, parsing a value for `<item>`.
+
+In both case, when Fandango encounters a recursion, _it stops evaluating the generator_:
+
+* When parsing an `<item>`, Fandango does not invoke the generator for `<data>` because `<data>` is being processed already.
+* Likewise, when producing `<data>`, Fandango does not invoke the generator for `<item>` because `<item>` is being processed already.
+
+Let us see if all of this works and if this input is indeed properly parsed and decoded.
+
+```shell
+$ echo -n 'Data: RmFuZGFuZ28gYXV0aG9y' | fandango parse -f encode-decode.fan -o - --format=grammar
+```
+
+```{code-cell}
+:tags: ["remove-input"]
+!echo -n 'Data: RmFuZGFuZ28gYXV0aG9y' | fandango parse -f encode-decode.fan -o - --format=grammar
+assert _exit_code == 0
+```
+
+We see that the `<data>` element contains the `"Fandango author"` string we provided as a constraint during generation.
+This is what the parsed derivation tree looks like:
+
+```{code-cell}
+:tags: ["remove-input"]
+from Tree import Tree
+
+tree = Tree('<start>',
+  Tree(b'Data: '),
+  Tree('<item>',
+    Tree(b'RmFuZGFuZ28gYXV0aG9y'),
+    sources=[
+      Tree('<data>',
+        Tree(b'Fandango'),
+        Tree('<byte>', Tree('<_byte>', Tree(b' '))),
+        Tree('<byte>', Tree('<_byte>', Tree(b'a'))),
+        Tree('<byte>', Tree('<_byte>', Tree(b'u'))),
+        Tree('<byte>', Tree('<_byte>', Tree(b't'))),
+        Tree('<byte>', Tree('<_byte>', Tree(b'h'))),
+        Tree('<byte>', Tree('<_byte>', Tree(b'o'))),
+        Tree('<byte>', Tree('<_byte>', Tree(b'r')))
+      ),
+    ]
+  )
+)
+tree.visualize()
+```
+
+With a constraint, we can check that the decoded string is correct:
+
+```shell
+$ echo -n 'Data: RmFuZGFuZ28gYXV0aG9y' | fandango parse -f encode-decode.fan -c '<data> == b"Fandango author"'
+```
+
+```{code-cell}
+:tags: ["remove-input"]
+!echo -n 'Data: RmFuZGFuZ28gYXV0aG9y' | fandango parse -f encode-decode.fan -c '<data> == b"Fandango author"'
+assert _exit_code == 0
+```
+
+We get no error - so the parse was successful, and that all constraints hold.
+
+
+## Applications
+
+The above scheme can be used for all kinds of encodings and compressions - and thus allow _translations between abstraction layers_.
+Typical applications include:
+
+* _Compressed_ data (e.g. pixels in a GIF or PNG file)
+* _Encoded_ data (e.g. binary input as ASCII chars in MIME encodings)
+* _Converted_ data (e.g. ASCII to UTF-8 to UTF-16 and back)
+
+Even though parts of the input are encoded (or compressed), you can still use _constraints_ to shape them.
+And if the encoding or compression can be inverted, you can also use it to _parse_ inputs again.

docs/DerivationTree.md

@@ -181,19 +181,22 @@ Invoking methods (`<SYMBOL>.METHOD()`), as well as operators (say, `<SYMBOL> + .
 Since any `<SYMBOL>` has the type `DerivationTree`, one must convert it first into a standard Python type before passing it as argument to a standard Python function.
 
 `str(<SYMBOL>) -> str`
-: Convert `<SYMBOL>` into a Unicode string. If `<SYMBOL>` is a byte string, its contents are converted using `latin-1` encoding.
+: Convert `<SYMBOL>` into a Unicode string. Byte strings in `<SYMBOL>` are converted using `latin-1` encoding.
+
+`bytes(<SYMBOL>) -> bytes`
+: Convert `<SYMBOL>` into a byte string. Unicode strings in `<SYMBOL>` are converted using `utf-8` encoding.
 
 `int(<SYMBOL>) -> int`
 : Convert `<SYMBOL>` into an integer, like the Python `int()` function.
 `<SYMBOL>` must be an `int`, or a Unicode string or byte string representing an integer literal.
 
 `float(<SYMBOL>) -> float`
 : Convert `<SYMBOL>` into a floating-point number, like the Python `float()` function.
-`<SYMBOL>` must be an int, or a Unicode string or byte string representing a float literal.
+`<SYMBOL>` must be an `int`, or a Unicode string or byte string representing a float literal.
 
 `complex(<SYMBOL>) -> complex`
 : Convert `<SYMBOL>` into a complex number, like the Python `complex()` function.
-`<SYMBOL>` must be an int, or a Unicode string or byte string representing a float literal.
+`<SYMBOL>` must be an `int`, or a Unicode string or byte string representing a float or complex literal.
 
 `bool(<SYMBOL>) -> bool`
 : Convert `<SYMBOL>` into a truth value:
@@ -276,6 +279,12 @@ Each element of the list can have a different type, depending on the type the `v
 : Return the parent of the current node, or `None` for the root node.
 
 
+### Accessing Sources
+
+`<SYMBOL>.sources() -> list[DerivationTree]`
+: Return a list containing all sources of `<SYMBOL>`. Sources are symbols used in generator expressions out of which the value of `<SYMBOL>` was created; see [the section on data conversions](sec:conversion) for details.
+
+
 ### Comparisons
 
 `<SYMBOL_1> == <SYMBOL_2>`

docs/Tree.py

@@ -36,18 +36,21 @@ def visualize(self, title="Derivation Tree"):
     def _visualize(self):
         name = f"node-{Tree.id_counter}"
         Tree.id_counter += 1
-        if isinstance(self.symbol, int):
-            label = str(self.symbol)
-        else:
+        if str(self.symbol).startswith("<"):
             label = self.symbol
+        else:
+            label = repr(self.symbol)
 
         # https://graphviz.org/doc/info/colors.html
+        # Colors checked against color vision deficiency
         if isinstance(self.symbol, int):
             color = "bisque4"
+        elif isinstance(self.symbol, bytes):
+            color = "darkblue"
         elif self.symbol.startswith("<"):
             color = "firebrick"
         else:
-            color = "darkblue"
+            color = "olivedrab4"
 
         label = label.replace("<", "\\<")
         label = label.replace(">", "\\>")
@@ -59,6 +62,6 @@ def _visualize(self):
 
         for source in self.sources():
             source_name = source._visualize()
-            Tree.dot.edge(name, source_name)
+            Tree.dot.edge(name, source_name, style="dotted", color="gray", dir="both")
 
         return name

evaluation/experiments/generator_params/generator_params.fan

@@ -5,8 +5,8 @@ def un_convert(input):
     return input[::-1]
 
 <start> ::= <number> <rev_number>
-<rev_number> ::= <number_tail>{0, 2} <number_start> :: convert(<source_number>.to_string())
-<source_number> ::= <number> :: un_convert(<rev_number>.to_string())
+<rev_number> ::= <number_tail>{0, 2} <number_start> := convert(<source_number>.to_string())
+<source_number> ::= <number> := un_convert(<rev_number>.to_string())
 
 <number> ::= <number_start> <number_tail>{0, 2}
 <number_start> ::= '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'

evaluation/experiments/generator_params/generator_params.py

@@ -5,11 +5,11 @@
 
 def count_g_params(tree: DerivationTree):
     count = 0
-    if len(tree.generator_params) > 0:
+    if len(tree.sources) > 0:
         count += 1
     for child in tree.children:
         count += count_g_params(child)
-    for child in tree.generator_params:
+    for child in tree.sources:
         count += count_g_params(child)
     return count
 

evaluation/experiments/transactions/transactions.fan

@@ -15,13 +15,13 @@ def compute_end_balance_sender(start_balance, amount):
 <info> ::= '   <info>\n' <currency> <stmt_date> <amount>'   </info>'
 <currency> ::= '      <currency>' 'EUR' '</currency>\n' | '      <currency>' 'USD' '</currency>\n'
 <stmt_date> ::= '      <stmt_date>' <timestamp> '</stmt_date>\n'
-<timestamp> ::= <digit>+ :: str(int(time.time()))
+<timestamp> ::= <digit>+ := str(int(time.time()))
 <amount> ::= '      <amount>' <am> '</amount>\n'
 <am> ::= <digit>+ ;
 <sender> ::= '\n   <sender>' <name> <account_no> <bank_key> <start_balance> <end_balance> '   </sender>'
 <receiver> ::= '\n   <receiver>' <name> <account_no> <bank_key> <start_balance> <end_balance> '   </receiver>\n'
 <name> ::= '\n      <name>' <name_str> '</name>'
-<name_str> ::= <letter>* :: str(fake.name()).upper();
+<name_str> ::= <letter>* := str(fake.name()).upper();
 <account_no> ::= '\n      <account_no>' <account_number> '</account_no>\n'
 <account_number> ::= <digit><digit><digit><digit><digit> <digit><digit><digit><digit><digit> <digit><digit><digit><digit><digit> <digit><digit><digit><digit><digit> <digit><digit>
 <bank_key> ::= '      <bank_key>' <bank_id> '</bank_key>\n'

evaluation/vs_isla/tar_evaluation/tar.fan

@@ -21,7 +21,7 @@
     <file_name_prefix>
     <header_padding>
 ;
-<file_name> ::= <file_name_first_char> <file_name_chars> <NULs> :: generate_file_name() ;
+<file_name> ::= <file_name_first_char> <file_name_chars> <NULs> := generate_file_name() ;
 <file_name_chars> ::= <file_name_char> <file_name_chars> | "" ;
 <NULs> ::= <NUL> <NULs> | "" ;
 <file_mode> ::= <octal_digit>{6} <SPACE> <NUL>;
@@ -31,10 +31,10 @@
 <mod_time> ::= <octal_digit>{11} <SPACE>;
 <checksum> ::= <octal_digit>{6} <NUL> <SPACE> ;
 <typeflag> ::= '0' ;
-<linked_file_name> ::= <file_name_first_char> <file_name_char_or_nul>{99} | <NUL>{100} :: generate_linked_file_name();
+<linked_file_name> ::= <file_name_first_char> <file_name_char_or_nul>{99} | <NUL>{100} := generate_linked_file_name();
 <file_name_char_or_nul> ::= <file_name_char> | <NUL> ;
-<uname> ::= <uname_first_char> <name_char_dollar_nul>{31} :: generate_uname("<uname>") ;
-<gname> ::= <uname_first_char> <name_char_dollar_nul>{31} :: generate_uname("<gname>") ;
+<uname> ::= <uname_first_char> <name_char_dollar_nul>{31} := generate_uname("<uname>") ;
+<gname> ::= <uname_first_char> <name_char_dollar_nul>{31} := generate_uname("<gname>") ;
 <name_char_dollar_nul> ::= <uname_char> | '$' | <NUL> ;
 <uname_first_char> ::=
     'a' | 'b' | 'c' | 'd' | 'e' | 'f' | 'g' | 'h' | 'i' | 'j' | 'k' | 'l' | 'm'
@@ -49,7 +49,7 @@
 <dev_min_num> ::= <octal_digit>{6} <SPACE> <NUL> ;
 <file_name_prefix> ::= <NUL>{155};
 <header_padding> ::= <NUL>{12};
-<content> ::= <char_or_nul>{512} :: generate_content() ;
+<content> ::= <char_or_nul>{512} := generate_content() ;
 <char_or_nul> ::= <character> | <NUL> ;
 <final_entry> ::= <NUL>{1024};
 <octal_digit> ::= '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' ;

pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fandango-fuzzer"
-version = "0.1.8"
+version = "0.2.0"
 authors = [
     { name = "José Antonio Zamudio Amaya", email = "[email protected]" },
     { name = "Marius Smytzek", email = "[email protected]" },