Implementing the Oil Expression Language

Turn Oil's expression grammar into an AST #387

Demo:

bin/osh -c 'var x = 1 + 2 * 3;`

This already works. (Right now semicolon or newline are accepted, we should also add EOF.)

Code

• https://github.com/oilshell/oil/tree/master/oil_lang
  • grammar.pgen2 is literally Python 3's grammar!!!
  • expr_parse.py contains the public interface that the rest of the code uses. It turns a stream of tokens into an AST, which is two steps under the hood. (tokens -> parse tree, then parse tree -> AST)
  • expr_to_ast.py -- the "transformer" i.e. parse tree -> AST step

Related Code

• https://github.com/oilshell/oil/blob/master/opy/compiler2/transformer.py is a version of this for Python (forked from the Python 2 standard library)
• drwilly is working on find in https://github.com/oilshell/oil/pull/386, which also has a "transformer"

Things We Want to Add

• All the operators. Small changes:
  • // is div
  • ** is ^ (following R and other mathematical languages)
  • ^ is xor
• Literals
  • dict
  • list
  • tuples, although I want to disallow 1-tuples like x,
  • integer
  • float
  • probably sets, although the syntax might be different to allow for dict punning, like {key1, key2} taking their values from surrounding scope
  • string: single quoted are like Python strings, but double quoted allows interpolation. This involves lexer modes. (Already implemented to a large extent)
• Comprehensions (lower priority)
  • list, dict, set
• Function literals (lower priority)