Various improvements (adopt best practices)

[bswck]

A series of PRs will land shortly. 🚀

General overview

• Modernize.
  • Use only new-style class declarations class T(object): → class T:.
  • Use types from the built-in scope, not the deprecated ones like typing.List, typing.Dict etc. See PEP 585 and typing docs on built-in types aliases.
  • Postpone type annotations evaluation in runtime. See PEP 563.
  • Use PEP 604 X | Y union type syntax.
  • Use PEP 613 explicit type aliases.
  • Use typing.TYPE_CHECKING to separate typing-related imports from the actual runtime requirements.
  • Consistently use a zero-argument form of super() inside methods.
  • Use pathlib. Use Path(...).read_(text|bytes)() and Path(...).write_(text|bytes)() instead of the open() context manager where possible.
  • Use f"{i:x}" instead of hex(i)[2:]. Act analogically for other numeral systems. More information here.
• Reformat.
  • Split all whole-line-width parameter lists into multi-line signatures with trailing commas.
  • Split one-line expressions that exceed max line length to multi-line.
  • Consistently use double-quote strings (advised and default format from black).
• Improve performance.
  • Replace list/dict/set comprehensions/generator expressions with map/filter where applicable.
  • Prefer on-demand tuples to on-demand lists/sets.
  • Use itertools where applicable. Don't iterate over data passed to functions that accept iterables.
  • Use type(X) instead of X.__class__ where the __class__'s descriptor behavior doesn't matter.
  • Don't catch StopIteration when using next(), simply pass a default value as the other argument.
  • Don't call exception constructor when raising it with no message.
• Refactor.
  • Tweak type annotations for mypy malduck --strict to pass.
  • Prefer raising exceptions of specialized classes instead of bare Exception.
  • Use typing.NamedTuple instead of collections.namedtuple.
  • Make boolean default arguments keyword-only (breaking change).
  • Consistently use in-place type annotations or stubs, not both.
  • Use exception chaining.
  • Handle some observed corner cases.
  • Use absolute imports to allow separate modules to run independently without having to manually specify the package via python3 -m.
  • In exception scenarios, use logging.Logger.exception() that automatically attaches the relevant traceback, instead of manually concatenating it to the log message on the ERROR logging level.
  • Generally use isinstance(x, T) and issubclass(X, T) instead of raw type(X) is/== type(T) comparisons in order to follow the Liskov substitution principle. Document exceptional cases.
  • Consistently use ABC/ABCMeta for abstract classes.
  • Use f"{obj!r}" instead of f"'{obj}'".
  • Use %-style formatting in logging.

Delivery plan

The fixes from the Refactor section require the most analysis. Those will be delivered in a PR after the first PR (Modernize, Reformat, Improve performance) is accepted and merged.

Refactoring might potentially require re-adjusting the test suite. That PR should be based on the initially-improved codebase to ensure an insightful code review and avoid merge conflicts.

Measuring quality assurance

ruff check malduck --select=ALL, ruff format malduck --check --select=ALL, mypy malduck --strict.

Progress

malduck

• __init__.py
• bits.py
• compression/
  • __init__.py
  • aplib.py
  • components/
    • __init__.py
    • aplib.py
    • lznt1.py
  • gzip.py
  • lznt1.py
• crypto/
  • __init__.py
  • aes.py
  • blowfish.py
  • camellia.py
  • chacha20.py
  • components/
    • __init__.py
    • pyserpent.py
  • des3.py
  • rabbit.py
  • rc.py
  • rsa.py
  • salsa20.py
  • serpent.py
  • winhdr.py
  • xor.py
• disasm.py
• dnpe.py
• extractor/
  • __init__.py
  • config_utils.py
  • extract_manager.py
  • extractor.py
  • extractor.pyi
  • modules.py
• hash/
  • __init__.py
  • crc.py
  • sha.py
• ints.py
• main.py
• pe.py
• procmem/
  • __init__.py
  • binmem.py
  • cuckoomem.py
  • idamem.py
  • procmem.py
  • procmem.pyi
  • procmemdnpe.py
  • procmemelf.py
  • procmempe.py
  • region.py
• py.typed
• string/
  • __init__.py
  • bin.py
  • inet.py
  • ops.py
• structure.py
• verify.py
• yara.py
• yara.pyi