Changes for final CR

Fixes from final CR

Update to pyparsing 2.1.8

Adds build target

Adds build target

Moves testing utilities

Increments version

Minor Makefile change

Fixes some issues that arose when writing patterns

Version bump

Author: evhub

Makefile

@@ -3,14 +3,14 @@
 all: dev pre-commit
 
 dev: venv
-	venv/bin/pip install -r requirements-dev.txt --upgrade
+	venv/bin/pip install -r requirements-dev.txt
 	@echo ""
 	@echo "========================================"
 	@echo "Don't forget to source venv/bin/activate"
 
 venv:
 	test -s venv || { virtualenv -p python2.7 venv; }
-	venv/bin/pip install -r requirements.txt --upgrade
+	venv/bin/pip install -r requirements.txt
 
 clean:
 	find . -name '*.pyc' -delete
@@ -19,6 +19,8 @@ clean:
 	rm -rf .tox
 	rm -rf *.egg-info
 	rm -rf venv
+	rm -rf build
+	rm -rf dist
 
 pre-commit:
 	@tox -e pre-commit -- install -f --install-hooks
@@ -30,3 +32,6 @@ test: pre-commit
 
 docs:
 	tox -e docs
+
+build:
+	python setup.py sdist bdist_wheel

README.md

@@ -1,6 +1,8 @@
 # Undebt
 
-Undebt is a fast, simple, reliable tool for performing massive, automated code refactoring used [@Yelp](https://github.com/Yelp). Undebt lets you define complex find-and-replace rules using simple, straightforward Python that can be applied quickly to an entire code base with a simple command.
+Undebt is a fast, straightforward, reliable tool for performing massive, automated code refactoring used [@Yelp](https://github.com/Yelp). Undebt lets you define complex find-and-replace rules using standard, straightforward Python that can be applied quickly to an entire code base with a simple command.
+
+To learn about what Undebt is and why we created it, check out our [post on the Yelp Engineering Blog](http://engineeringblog.yelp.com/).
 
 To get started using Undebt, install with
 ```

docs/source/cli.rst

@@ -31,9 +31,8 @@ Read it
       --pattern path, -p path
                             paths to pattern definition files
       --extension ext, -e ext
-                            extensions of files to be modified when using
-                            searching a directory (exclude ".", e.g. "py" instead
-                            of ".py")
+                            extensions of files to be modified when searching a
+                            directory (exclude ".", e.g. "py" instead of ".py")
       --multiprocess processes, -m processes
                             number of processes to run in parallel (default is 16)
       --verbose

docs/source/examples.rst

@@ -145,5 +145,5 @@ In files where `from __future__ import unicode_literals` appears, removes unnece
 Example of:
 
 - an advanced style pattern file making use of multi-pass parsing
-- `originalTextFor` to get make a grammar parse to the original text that matched it
+- `originalTextFor` to make grammar elements parse to the original text that matched them
 - `STRING` to match any valid string

docs/source/patterns.rst

@@ -7,9 +7,9 @@ Undebt_: Pattern Files
 Undebt requires a pattern file that describes what to replace and how to replace it. There are two different ways to write pattern files: basic style, and advanced style. Unless you know you need multi-pass parsing, you should use basic style by default.
 
 Basic Style
----------------
+-----------
 
-When writing an basic style pattern, you must define the following names in your pattern file:
+If you don't know what style you should be using, you should be using basic style. When writing a basic style pattern, you must define the following names in your pattern file:
 
 - `grammar` defines what pattern you want to replace, and must be a pyparsing_ grammar object.
 - `replace` is a function of one argument, the tokens produced by `grammar`, that returns the string they should be replaced with, or `None` to do nothing (this is the single-argument form—multi-argument is also allowed as documented `below`_).
@@ -24,9 +24,9 @@ That sounds complicated, but it's actually very simple. To start learning more,
 .. _`pattern utilities`: util.html
 
 Advanced Style
-------------------
+--------------
 
-When writing an advanced style pattern, you need only define one name:
+Unlike basic style, advanced style allows you to use custom multi-pass parsing—if that's not something you need, you should use basic style. When writing an advanced style pattern, you need only define one name:
 
 - `patterns` is a list of `(grammar, replace)` tuples, where each tuple in the list is only run if the previous one succeeded
 
@@ -58,4 +58,4 @@ Or equivalently but more succinctly::
 Multi-Argument Replace
 ----------------------
 
-In both styles, when writing a `replace` function, it is sometimes useful to have access to the location in and/or text of the original file. If your `replace` function takes two arguments, it will be passed `location, tokens`, and for three arguments, it will get `text, location, tokens`. This will work even if you are using one of the `tokens_as_list` or `tokens_as_dict` decorator.
+In both styles, when writing a `replace` function, it is sometimes useful to have access to the parsing location in the file and/or the text of the original file. If your `replace` function takes two arguments, it will be passed `location, tokens`, and for three arguments, it will get `text, location, tokens`. This will work even if you are using one of the `tokens_as_list` or `tokens_as_dict` decorators.

docs/source/util.rst

@@ -41,9 +41,13 @@ Returns the whitespace at the end of `text`.
 
 Modifies a grammar element to always parse to the same fixed `output`.
 
+**debug(item)**
+
+Modifies a grammar element to print the tokens that it matches.
+
 **attach(item, action)**
 
-Modifies a grammar element to parse to the result of calling `action` on the  tokens produced by that grammar element. Used to implement `condense`, `addspace`, and `fixto`.
+Modifies a grammar element to parse to the result of calling `action` on the  tokens produced by that grammar element.
 
 `undebt.pattern.common`
 -----------------------
@@ -90,12 +94,12 @@ Grammar elements that match different types of multi-line strings.
 **COMMA_IND, LPAREN_IND, IND_RPAREN**
 Same as `COMMA`, `LPAREN`, and `RPAREN`, but allow for an `INDENT` after (for `COMMA_IND` and `LPAREN_IND`) or before (for `IND_RPAREN`).
 
-**NO_BS_NL**
-Matches a new line not preceded by a backslash.
-
 **LINE_START**
 Matches the start of a line, either after a new line, or at the start of the file.
 
+**NO_BS_NL**
+Matches a new line not preceded by a backslash.
+
 **START_OF_FILE**
 Grammar element that only matches at the very beginning of the file.
 

requirements.txt

@@ -1,2 +1,2 @@
 argparse==1.4.0
-pyparsing==2.1.7
+pyparsing==2.1.8

tests/__init__.py

@@ -1,33 +1 @@
 # -*- coding: utf-8 -*-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import os.path
-
-from undebt.pattern.interface import create_find_and_replace
-from undebt.pattern.interface import parse_grammar
-
-
-tests_inputs_directory = os.path.join(os.path.dirname(__file__), "inputs")
-
-
-def assert_parse(grammar, text, tokens_list, interval_list):
-    assert len(tokens_list) == len(interval_list)
-    expected = [(tokens_list[i],) + interval_list[i] for i in range(len(tokens_list))]
-    raw_assert_parse(grammar, text, expected)
-
-
-def raw_assert_parse(grammar, text, expected):
-    results = parse_grammar(grammar, text)
-    assert len(results) == len(expected)
-    assert expected == [(tokens.asList(), start, stop) for tokens, start, stop in results]
-
-
-def assert_transform(pattern, text, expected):
-    assert not hasattr(pattern, "patterns"), "assert_transform only works with old/basic style patterns"
-    find_and_replace = create_find_and_replace(pattern.grammar, pattern.replace)
-    results = parse_grammar(find_and_replace, text)
-    assert len(results) == len(expected)
-    assert all(len(tokens) == 1 for tokens, _, _ in results)
-    assert expected == [tokens.asList()[0] for tokens, _, _ in results]

tests/cmd/main_test.py

@@ -7,11 +7,12 @@
 
 import mock
 
-from tests import tests_inputs_directory
 from undebt.cmd.main import main
 from undebt.examples import method_to_function
 
 
+tests_inputs_directory = os.path.join(os.path.dirname(os.path.dirname(__file__)), "inputs")
+
 method_to_function_path = os.path.splitext(method_to_function.__file__)[0] + ".py"
 
 method_to_function_input_path = os.path.join(tests_inputs_directory, "method_to_function_input.txt")

tests/examples/attribute_to_function_test.py

@@ -3,10 +3,10 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_transform
 from undebt.cmd.logic import process
 from undebt.examples import attribute_to_function
 from undebt.pattern.interface import get_patterns
+from undebt.pattern.testing import assert_transform
 
 
 def test_simple():

tests/examples/class_inherit_object_test.py

@@ -3,10 +3,10 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_transform
 from undebt.cmd.logic import process
 from undebt.examples import class_inherit_object
 from undebt.pattern.interface import get_patterns
+from undebt.pattern.testing import assert_transform
 
 
 def test_no_parens():

tests/examples/dbl_quote_docstring_test.py

@@ -3,10 +3,10 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_transform
 from undebt.cmd.logic import process
 from undebt.examples import dbl_quote_docstring
 from undebt.pattern.interface import get_patterns
+from undebt.pattern.testing import assert_transform
 
 
 def test_actual_code():

tests/examples/exec_function_test.py

@@ -3,10 +3,10 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_transform
 from undebt.cmd.logic import process
 from undebt.examples import exec_function
 from undebt.pattern.interface import get_patterns
+from undebt.pattern.testing import assert_transform
 
 
 def test_globals():

tests/examples/hex_to_bitshift_test.py

@@ -3,10 +3,10 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_transform
 from undebt.cmd.logic import process
 from undebt.examples import hex_to_bitshift
 from undebt.pattern.interface import get_patterns
+from undebt.pattern.testing import assert_transform
 
 
 def test_simple():

tests/examples/method_to_function_test.py

@@ -3,10 +3,10 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_transform
 from undebt.cmd.logic import process
 from undebt.examples import method_to_function
 from undebt.pattern.interface import get_patterns
+from undebt.pattern.testing import assert_transform
 
 
 def test_simple():

tests/examples/nl_at_eof_test.py

@@ -3,10 +3,10 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_transform
 from undebt.cmd.logic import process
 from undebt.examples import nl_at_eof
 from undebt.pattern.interface import get_patterns
+from undebt.pattern.testing import assert_transform
 
 
 def test_match():

tests/examples/sqla_count_test.py

@@ -3,10 +3,10 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_transform
 from undebt.cmd.logic import process
 from undebt.examples import sqla_count
 from undebt.pattern.interface import get_patterns
+from undebt.pattern.testing import assert_transform
 
 
 def test_one_liner():

tests/pattern/common_test.py

@@ -3,13 +3,13 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_parse
 from undebt.pattern.common import DOTTED_NAME
 from undebt.pattern.common import INDENT
 from undebt.pattern.common import NL_WHITE
 from undebt.pattern.common import PARENS
 from undebt.pattern.common import STRING
 from undebt.pattern.common import WHITE
+from undebt.pattern.testing import assert_parse
 
 
 def test_INDENT():

tests/pattern/python_test.py

@@ -3,11 +3,12 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_parse
 from undebt.pattern.python import ATOM
+from undebt.pattern.python import BINARY_OP
 from undebt.pattern.python import EXPR
 from undebt.pattern.python import HEADER
 from undebt.pattern.python import OP
+from undebt.pattern.testing import assert_parse
 
 
 def test_OP():
@@ -33,6 +34,15 @@ def test_OP():
     )
 
 
+def test_BINARY_OP():
+    assert_parse(
+        grammar=BINARY_OP,
+        text="==",
+        tokens_list=[["=="]],
+        interval_list=[(0, 2)],
+    )
+
+
 def test_ATOM():
     assert_parse(
         grammar=ATOM,

tests/pattern/util_test.py

@@ -3,7 +3,7 @@
 from __future__ import division
 from __future__ import print_function
 
-from tests import assert_parse
+from undebt.pattern.testing import assert_parse
 from undebt.pattern.util import leading_whitespace
 from undebt.pattern.util import quoted
 from undebt.pattern.util import trailing_whitespace

undebt/__init__.py

@@ -6,4 +6,4 @@
 
 # this should be the only place where the version is kept, if you
 # need to know the version somewhere else just import it from here
-__version__ = "0.1.0"
+__version__ = "0.2.1"

undebt/cmd/main.py

@@ -70,7 +70,7 @@ def _handle_arguments():
         '-e',
         metavar='ext',
         action='append',
-        help='extensions of files to be modified when using searching a directory'
+        help='extensions of files to be modified when searching a directory'
         ' (exclude ".", e.g. "py" instead of ".py")')
     parser.add_argument(
         '--multiprocess', '-m', metavar='processes', type=int, default=16,

undebt/pattern/python.py

@@ -30,7 +30,7 @@
 from undebt.pattern.util import condense
 
 
-ASSIGN_OP = Combine(Optional(Word("~%^&*-+|/")) + Literal("="))
+ASSIGN_OP = Combine((Word("~%^&*-+|/") | ~Literal("==")) + Literal("="))
 
 
 UNARY_OP = addspace(OneOrMore(

undebt/pattern/testing.py

@@ -0,0 +1,31 @@
+# -*- coding: utf-8 -*-
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from undebt.pattern.interface import create_find_and_replace
+from undebt.pattern.interface import parse_grammar
+
+
+def assert_parse(grammar, text, tokens_list, interval_list):
+    """Assert that grammar parses text to tokens_list on interval_list."""
+    assert len(tokens_list) == len(interval_list)
+    expected = [(tokens_list[i],) + interval_list[i] for i in range(len(tokens_list))]
+    raw_assert_parse(grammar, text, expected)
+
+
+def raw_assert_parse(grammar, text, token_interval_list):
+    """Assert grammar parses text to expected tokens and intervals."""
+    results = parse_grammar(grammar, text)
+    assert len(results) == len(token_interval_list)
+    assert token_interval_list == [(tokens.asList(), start, stop) for tokens, start, stop in results]
+
+
+def assert_transform(pattern, text, transformed_tokens):
+    """Assert grammar parses text to expected transformed tokens."""
+    assert not hasattr(pattern, "patterns"), "assert_transform only works with basic style patterns"
+    find_and_replace = create_find_and_replace(pattern.grammar, pattern.replace)
+    results = parse_grammar(find_and_replace, text)
+    assert len(results) == len(transformed_tokens)
+    assert all(len(tokens) == 1 for tokens, _, _ in results)
+    assert transformed_tokens == [tokens.asList()[0] for tokens, _, _ in results]

undebt/pattern/util.py

@@ -33,6 +33,11 @@ def condense(item):
     return attach(item, "".join)
 
 
+def debug(item):
+    """Modifies a grammar element to print whatever it matches."""
+    return attach(item, lambda tokens: print(tokens))
+
+
 def quoted(string):
     """Match a string containing the given string."""
     return originalTextFor(Combine("'" + string + "'") | Combine('"' + string + '"'))