Skip to content

feat!: support decoding multiple ABIs at the same time, including ds-note library logs #757

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 62 commits into from
Jul 27, 2022
Merged
Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
66ff53e
feat: add receipt events and ds-note decoding
banteg Jul 6, 2022
fc20c38
refactor: move ds-note decoding to ecosystem api
banteg Jul 6, 2022
41347d3
fix: consistent exceptions
banteg Jul 6, 2022
4b4539a
feat: optimize ds-log decoding
banteg Jul 6, 2022
2695454
test: add makerdao vat contract
banteg Jul 6, 2022
946536a
test: ds note
banteg Jul 7, 2022
cc8be00
feat: make abi optional for decode_logs, add ds-note decoding
banteg Jul 7, 2022
1622abc
test: decode events without providing abi
banteg Jul 7, 2022
1ec6214
refactor: simplify finding selector for ds-note
banteg Jul 7, 2022
0c6666e
fix: black/flake8 clash
banteg Jul 7, 2022
7952e32
feat: lazy fetching of contract types
banteg Jul 7, 2022
8fdc8de
fix: catch decoding error
banteg Jul 7, 2022
10c25df
fix: handle None contract types
antazoey Jul 7, 2022
9e7f370
chore: Merge branch 'main' into feat/decode-events
antazoey Jul 7, 2022
10d062a
chore: rm optional annotation
antazoey Jul 7, 2022
4cdc91d
refactor: move stuff to ethereum and handle anon logs more
antazoey Jul 7, 2022
d876f3d
test: add simple tests
antazoey Jul 7, 2022
1660f86
test: make fixture
antazoey Jul 7, 2022
c5ed093
test: tests for getting ds notes from receipt
antazoey Jul 8, 2022
0185ab2
chore: del unneeded file
antazoey Jul 8, 2022
eafb03f
chore: merge main
antazoey Jul 10, 2022
e033050
fix: decode lib log optimization
antazoey Jul 10, 2022
44eee65
refactor: move decode lib logs to ecosystem
antazoey Jul 10, 2022
7d12143
chore: put back fixture use
antazoey Jul 10, 2022
563b2cf
Merge branch 'main' into feat/decode-events
fubuloubu Jul 14, 2022
903ba84
chore: merge main
antazoey Jul 20, 2022
83536ff
chore: resolve merge conflicts
antazoey Jul 20, 2022
eddf2d3
fix: address regressions from merge
antazoey Jul 20, 2022
69f9f22
refator: make list of abis
antazoey Jul 20, 2022
3125cdd
chore: pr feedback
antazoey Jul 20, 2022
25d9288
fix: base method param type correct
antazoey Jul 20, 2022
c22bb8a
fix: skip unfound contracttypes
antazoey Jul 20, 2022
0b924bc
refactor: use return from pool map
antazoey Jul 20, 2022
fbc160c
refactor: make get_all method in contract cache
antazoey Jul 20, 2022
d1b8c89
fix: handle conversion and non contract types
antazoey Jul 20, 2022
afd7915
refactor: get contract type
banteg Jul 20, 2022
659191a
feat: exit early for empty code
banteg Jul 20, 2022
0901366
fix: exit early
banteg Jul 20, 2022
2b6d167
refactor: get multiple
banteg Jul 20, 2022
832616e
fix: decode in correct order
banteg Jul 20, 2022
9a20e30
feat: add event selector to ecosystem
banteg Jul 22, 2022
ab67dd9
Merge branch 'main' into feat/decode-events
banteg Jul 22, 2022
08323a6
refactor: make decode_logs method in ape_eth txn
antazoey Jul 22, 2022
57bb7b3
fix: add abstract method
antazoey Jul 22, 2022
ccccb5c
Merge branch 'main' into feat/decode-events
antazoey Jul 22, 2022
2861def
feat: rm get lib logs method
antazoey Jul 25, 2022
25c12fb
chore: resolve conflicts
antazoey Jul 25, 2022
5dc6276
fix: contract cache issue suddenly
antazoey Jul 25, 2022
c8d0e67
fix: cache before return
antazoey Jul 25, 2022
56b272f
chore: Merge branch 'main' into feat/decode-events
antazoey Jul 26, 2022
64cb17d
refactor: use * not for event abis
antazoey Jul 26, 2022
55625f2
refactor: rename name to event_name
antazoey Jul 26, 2022
d320ded
feat: support transaction_index
antazoey Jul 26, 2022
cc0a676
fix: to int
banteg Jul 26, 2022
a6b3232
feat: handle raw responses
antazoey Jul 26, 2022
ae68813
test: improve test
antazoey Jul 26, 2022
084f5af
fix: issue when not using web3 provideR
antazoey Jul 26, 2022
8f223c4
fix: revert unneeded change
antazoey Jul 26, 2022
d2ee852
chore: new found mypy issues
antazoey Jul 26, 2022
299b048
chore: rm unused import
antazoey Jul 27, 2022
e5b764b
feat: remove support for snake_case
antazoey Jul 27, 2022
083ca81
Merge branch 'main' into feat/decode-events
antazoey Jul 27, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/ape/api/networks.py
Original file line number Diff line number Diff line change
Expand Up @@ -305,13 +305,13 @@ def encode_transaction(
"""

@abstractmethod
def decode_logs(self, abi: EventABI, raw_logs: List[Dict]) -> Iterator[ContractLog]:
def decode_logs(self, logs: List[Dict], *events: EventABI) -> Iterator["ContractLog"]:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the variadic argument feels a bit weird here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fubuloubu wanted it and I was hesitant at first, but I kind of like how it feels by not having to wrap single ABIs in a list first.

But I can go either way still, let's reach a consensus

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a breaking change anyways, live life on the edge 🤘

"""
Decode any contract logs that match the given event ABI from the raw log data.

Args:
abi (EventABI): The event producing the logs.
raw_logs (List[Dict]): A list of raw log data from the chain.
logs (List[Dict]): A list of raw log data from the chain.
*events (EventABI): Event definitions to decode.

Returns:
Iterator[:class:`~ape.types.ContractLog`]
Expand Down
21 changes: 14 additions & 7 deletions src/ape/api/providers.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from pathlib import Path
from signal import SIGINT, SIGTERM, signal
from subprocess import PIPE, Popen
from typing import Any, Iterator, List, Optional
from typing import Any, Dict, Iterator, List, Optional

from eth_typing import HexStr
from eth_utils import add_0x_prefix
Expand All @@ -21,6 +21,7 @@
from web3 import Web3
from web3.exceptions import ContractLogicError as Web3ContractLogicError
from web3.exceptions import TimeExhausted
from web3.types import RPCEndpoint

from ape.api.config import PluginConfig
from ape.api.networks import LOCAL_NETWORK_NAME, NetworkAPI
Expand Down Expand Up @@ -732,7 +733,7 @@ def block_ranges(self, start=0, stop=None, page=None):
if stop is None:
stop = self.chain_manager.blocks.height
if page is None:
page = self.chain_manager.provider.block_page_size
page = self.block_page_size

for start_block in range(start, stop + 1, page):
stop_block = min(stop, start_block + page - 1)
Expand All @@ -750,20 +751,26 @@ def fetch_log_page(block_range):
# eth-tester expects a different format, let web3 handle the conversions for it
raw = "EthereumTester" not in self.client_version
logs = self._get_logs(page_filter.dict(), raw)
return self.network.ecosystem.decode_logs(log_filter.events, logs)
return self.network.ecosystem.decode_logs(logs, *log_filter.events)

with ThreadPoolExecutor(self.concurrency) as pool:
for page in pool.map(fetch_log_page, block_ranges):
yield from page

def _get_logs(self, filter_params, raw=True):
def _get_logs(self, filter_params, raw=True) -> List[Dict]:
if raw:
response = self.web3.provider.make_request("eth_getLogs", [filter_params])
response = self.web3.provider.make_request(RPCEndpoint("eth_getLogs"), [filter_params])
if "error" in response:
raise ValueError(response["error"]["message"])
error = response["error"]
if isinstance(error, dict) and "message" in error:
raise ValueError(error["message"])
else:
# Should never get here, mostly for mypy
raise ValueError(str(error))

return response["result"]
else:
return self.web3.eth.get_logs(filter_params)
return [vars(d) for d in self.web3.eth.get_logs(filter_params)]

def send_transaction(self, txn: TransactionAPI) -> ReceiptAPI:
try:
Expand Down
24 changes: 10 additions & 14 deletions src/ape/api/transactions.py
Original file line number Diff line number Diff line change
Expand Up @@ -215,14 +215,12 @@ def _confirmations_occurred(self) -> int:

return latest_block.number - self.block_number

def raise_for_status(self):
"""
Handle provider-specific errors regarding a non-successful
:class:`~api.providers.TransactionStatusEnum`.
"""

@abstractmethod
def decode_logs(
self, abi: Union[List[Union[EventABI, "ContractEvent"]], Union[EventABI, "ContractEvent"]]
self,
abi: Optional[
Union[List[Union[EventABI, "ContractEvent"]], Union[EventABI, "ContractEvent"]]
] = None,
) -> Iterator[ContractLog]:
"""
Decode the logs on the receipt.
Expand All @@ -233,14 +231,12 @@ def decode_logs(
Returns:
Iterator[:class:`~ape.types.ContractLog`]
"""
if not isinstance(abi, (list, tuple)):
abi = [abi]

for event_abi in abi:
if not isinstance(event_abi, EventABI):
event_abi = event_abi.abi

yield from self.provider.network.ecosystem.decode_logs(event_abi, self.logs)
def raise_for_status(self):
"""
Handle provider-specific errors regarding a non-successful
:class:`~api.providers.TransactionStatusEnum`.
"""

def await_confirmations(self) -> "ReceiptAPI":
"""
Expand Down
2 changes: 1 addition & 1 deletion src/ape/contracts/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,7 @@ def from_receipt(self, receipt: ReceiptAPI) -> Iterator[ContractLog]:
Iterator[:class:`~ape.contracts.base.ContractLog`]
"""
ecosystem = self.provider.network.ecosystem
yield from ecosystem.decode_logs(self.abi, receipt.logs)
yield from ecosystem.decode_logs(receipt.logs, self.abi)

def poll_logs(
self,
Expand Down
8 changes: 7 additions & 1 deletion src/ape/managers/chain.py
Original file line number Diff line number Diff line change
Expand Up @@ -580,7 +580,6 @@ def get(
return default

contract_type = self._get_contract_type_from_disk(address_key)

if not contract_type:
# Contract could be a minimal proxy
proxy_info = self._local_proxies.get(address_key) or self._get_proxy_info_from_disk(
Expand All @@ -596,6 +595,13 @@ def get(
self._local_proxies[address_key] = proxy_info
return self.get(proxy_info.target)

if not self.provider.get_code(address_key):
if default:
self._local_contracts[address_key] = default
self._cache_contract_to_disk(address_key, default)

return default

# Also gets cached to disk for faster lookup next time.
contract_type = self._get_contract_type_from_explorer(address_key)

Expand Down
29 changes: 26 additions & 3 deletions src/ape/types/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
from web3.types import FilterParams

from ape._compat import Literal
from ape.utils.misc import to_int, to_snake_case

from .signatures import MessageSignature, SignableMessage, TransactionSignature

Expand Down Expand Up @@ -78,8 +79,8 @@ def validate_addresses(cls, value):
def dict(self, client=None):
return FilterParams(
address=self.addresses,
fromBlock=hex(self.start_block),
toBlock=hex(self.stop_block),
fromBlock=hex(self.start_block), # type: ignore
toBlock=hex(self.stop_block), # type: ignore
topics=self.topic_filter, # type: ignore
)

Expand Down Expand Up @@ -147,7 +148,7 @@ class ContractLog(BaseModel):
An instance of a log from a contract.
"""

name: str
event_name: str
"""The name of the event."""

contract_address: AddressType
Expand All @@ -168,6 +169,28 @@ class ContractLog(BaseModel):
log_index: int
"""The index of the log on the transaction."""

transaction_index: Optional[int] = None
"""
The index of the transaction's position when the log was created.
Is `None` when from the pending block.
"""

class Config:
alias_generator = to_snake_case

@validator("block_number", "log_index", "transaction_index", pre=True)
def validate_hex_ints(cls, value):
if not isinstance(value, int):
return to_int(value)

return value

@validator("contract_address", pre=True)
def validate_address(cls, value):
from ape import convert

return convert(value, AddressType)

def __str__(self) -> str:
args = " ".join(f"{key}={val}" for key, val in self.event_arguments.items())
return f"{self.name} {args}"
Expand Down
4 changes: 4 additions & 0 deletions src/ape/utils/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@
raises_not_implemented,
singledispatchmethod,
stream_response,
to_int,
to_snake_case,
)
from ape.utils.os import get_all_files_in_directory, get_relative_path, use_temp_sys_path
from ape.utils.process import JoinableQueue, spawn
Expand Down Expand Up @@ -85,6 +87,8 @@
"Struct",
"StructParser",
"TraceStyles",
"to_int",
"to_snake_case",
"use_temp_sys_path",
"USER_AGENT",
"ZERO_ADDRESS",
Expand Down
5 changes: 5 additions & 0 deletions src/ape/utils/misc.py
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,10 @@ def to_int(value) -> int:
raise ValueError(f"cannot convert {repr(value)} to int")


def to_snake_case(string: str) -> str:
return "".join(["_" + i.lower() if i.isupper() else i for i in string]).lstrip("_")


__all__ = [
"cached_property",
"expand_environment_variables",
Expand All @@ -261,5 +265,6 @@ def to_int(value) -> int:
"raises_not_implemented",
"singledispatchmethod",
"stream_response",
"to_snake_case",
"USER_AGENT",
]
4 changes: 2 additions & 2 deletions src/ape/utils/trace.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@
from hexbytes import HexBytes
from rich.tree import Tree

from ape.api.networks import EcosystemAPI
from ape.exceptions import ContractError, DecodingError
from ape.utils.abi import Struct, parse_type
from ape.utils.misc import ZERO_ADDRESS

if TYPE_CHECKING:
from ape.api.networks import EcosystemAPI
from ape.api.transactions import ReceiptAPI


Expand Down Expand Up @@ -85,7 +85,7 @@ def __init__(
self.colors = color_set

@property
def _ecosystem(self) -> EcosystemAPI:
def _ecosystem(self) -> "EcosystemAPI":
return self._receipt.provider.network.ecosystem

def parse_as_tree(self, call: CallTreeNode) -> Tree:
Expand Down
27 changes: 12 additions & 15 deletions src/ape_ethereum/ecosystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@
is_array,
parse_type,
returns_array,
to_snake_case,
)
from ape.utils.misc import to_int
from ape_ethereum.transactions import (
AccessListTransaction,
BaseTransaction,
Expand Down Expand Up @@ -146,6 +146,8 @@ def encode_address(cls, address: AddressType) -> RawAddress:

def get_proxy_info(self, address: AddressType) -> Optional[ProxyInfo]:
code = self.provider.get_code(address).hex()[2:]
if not code:
return None
patterns = {
ProxyType.Minimal: r"363d3d373d3d3d363d73(.{40})5af43d82803e903d91602b57fd5bf3",
ProxyType.Vyper: r"366000600037611000600036600073(.{40})5af4602c57600080fd5b6110006000f3", # noqa: E501
Expand Down Expand Up @@ -442,13 +444,7 @@ def create_transaction(self, **kwargs) -> TransactionAPI:

return txn_class(**kwargs) # type: ignore

def decode_logs(
self, events: Union[EventABI, List[EventABI]], logs: List[Dict]
) -> Iterator["ContractLog"]:

if not isinstance(events, list):
events = [events]

def decode_logs(self, logs: List[Dict], *events: EventABI) -> Iterator["ContractLog"]:
abi_inputs = {
encode_hex(keccak(text=abi.selector)): LogInputABICollection(abi) for abi in events
}
Expand All @@ -468,13 +464,14 @@ def decode_logs(
continue

event_arguments = abi.decode(topics, log["data"])

log = {to_snake_case(k): v for k, v in log.items()}
yield ContractLog(
name=abi.event_name,
block_hash=log["block_hash"],
block_number=log["block_number"],
contract_address=self.decode_address(log["address"]),
event_arguments=event_arguments,
transaction_hash=log["transactionHash"],
block_number=to_int(log["blockNumber"]),
block_hash=log["blockHash"],
log_index=to_int(log["logIndex"]),
)
event_name=abi.event_name,
log_index=log["log_index"],
transaction_hash=log["transaction_hash"],
transaction_index=log["transaction_index"],
) # type: ignore
Loading