Skip to content

V3 state crawler #1142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Feb 11, 2025
Merged

V3 state crawler #1142

merged 22 commits into from
Feb 11, 2025

Conversation

Andrei-Dolgolev
Copy link
Contributor

@Andrei-Dolgolev Andrei-Dolgolev commented Nov 26, 2024

Added V3 support fo state tasks

QueryAPI sql Handling Improvements:

  • Added resolve_table_names and prepare_query functions to dynamically resolve table names and prepare SQL queries with placeholders replaced by actual table names (crawlers/mooncrawl/mooncrawl/actions.py). [1] [2]
  • Introduced QueryTextClauseException for handling errors when transforming queries to TextClause (crawlers/mooncrawl/mooncrawl/actions.py).
  • Updated queries_data_update_handler to use the new query preparation functions and handle exceptions accordingly (crawlers/mooncrawl/mooncrawl/api.py). [1] [2]

Blockchain Support Enhancements:

  • Added support for new AvailableBlockchainType with corresponding environment variables and configurations (crawlers/mooncrawl/mooncrawl/blockchain.py, crawlers/mooncrawl/mooncrawl/settings.py). [1] [2] [3] [4]

Data Model Updates:

  • Enhanced ViewTasks model to include optional customer_id, instance_id, and v3 fields (crawlers/mooncrawl/mooncrawl/data.py).

These changes collectively improve the flexibility and robustness of the Mooncrawl project's query handling and blockchain support.State jobs have ability run over v3 instaces.

state-crawler --web3-uri  crawl-jobs --blockchain <blockchain> --customer-db-uri

Added --customer-db-uri parameter.

new types of jobs available.

[{
    "customer_id": "uuid",
    "instance_id": int,
    "v3": true,
    "type": "function",
    "stateMutability": "view",
    "inputs": [],
    "name": "symbol",
    "constant": true,
    "payable": false,
    "outputs": [
        {
            "name": "",
            "type": "string"
        }
    ],
    "address": {
            "customer_id": "uuid",
            "instance_id": int,
            "type": "queryAPI",
            "query_url": "template_arbitrum_gateway_tokens",
            "blockchain": "<chain>",
            "params": {
                "address": "<address>"
            },
            "keys": [
                "address"
            ]
        }
}]

Determines the table names based on the blockchain and labels version.
Returns an empty dictionary if blockchain is not provided.
"""
if not request_data.blockchain:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to throw exception here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need that resolution just for return block_number and block_timestamp.
That enough for see state of database but if was send actual blockchain you can see even more related to query data.

Copy link
Contributor

@kompotkot kompotkot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lg

@Andrei-Dolgolev Andrei-Dolgolev merged commit bd568a4 into main Feb 11, 2025
0 of 2 checks passed
@Andrei-Dolgolev Andrei-Dolgolev deleted the v3-state-crawler branch February 11, 2025 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants