Skip to content

Provide the batch SQL analysis API #663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 10, 2024
Merged

Conversation

goldmedal
Copy link
Contributor

@goldmedal goldmedal commented Jul 9, 2024

Description

Implement the batch SQL analysis API

URL

GET /v2/analysis/sqls

Request Body

  • manifestStr: based64 manifset json string.
  • sqls: the list of input sql.
{
    "manifestStr": "ewogICJjYXRhbG9nIjogIndyZW5haSIsCiAgInNjaGVtYSI6ICJwdWJsaWMiLAogICJtb2RlbHMiOiBbCiAgICB7CiAgICAgICJuYW1lIjogIm9yZGVycyIsCiAgICAgICJyZWZTcWwiOiAic2VsZWN0ICogZnJvbSBvcmRlcnMiLAogICAgICAiY29sdW1ucyI6IFsKICAgICAgICB7CiAgICAgICAgICAibmFtZSI6ICJvcmRlcmtleSIsCiAgICAgICAgICAiZXhwcmVzc2lvbiI6ICJvX29yZGVya2V5IiwKICAgICAgICAgICJ0eXBlIjogImludGVnZXIiCiAgICAgICAgfSwKICAgICAgICB7CiAgICAgICAgICAibmFtZSI6ICJjdXN0a2V5IiwKICAgICAgICAgICJleHByZXNzaW9uIjogIm9fY3VzdGtleSIsCiAgICAgICAgICAidHlwZSI6ICJpbnRlZ2VyIgogICAgICAgIH0KICAgICAgXQogICAgfSwKICAgIHsKICAgICAgIm5hbWUiOiAiY3VzdG9tZXIiLAogICAgICAicmVmU3FsIjogInNlbGVjdCAqIGZyb20gY3VzdG9tZXIiLAogICAgICAiY29sdW1ucyI6IFsKICAgICAgICB7CiAgICAgICAgICAibmFtZSI6ICJjdXN0a2V5IiwKICAgICAgICAgICJleHByZXNzaW9uIjogImNfY3VzdGtleSIsCiAgICAgICAgICAidHlwZSI6ICJpbnRlZ2VyIgogICAgICAgIH0sCiAgICAgICAgewogICAgICAgICAgIm5hbWUiOiAibmFtZSIsCiAgICAgICAgICAiZXhwcmVzc2lvbiI6ICJjX25hbWUiLAogICAgICAgICAgInR5cGUiOiAidmFyY2hhciIKICAgICAgICB9CiAgICAgIF0KICAgIH0KICBdCn0=",
    "sqls": [
        "SELECT * FROM customer",
        "WITH t1 AS (SELECT * FROM customer) SELECT * FROM t1",
        "SELECT * FROM orders WHERE orderkey = 1 UNION SELECT * FROM orders where orderkey = 2"
    ]
}

The manifestStr is encoded from:

{
  "catalog": "wrenai",
  "schema": "public",
  "models": [
    {
      "name": "orders",
      "refSql": "select * from orders",
      "columns": [
        {
          "name": "orderkey",
          "expression": "o_orderkey",
          "type": "integer"
        },
        {
          "name": "custkey",
          "expression": "o_custkey",
          "type": "integer"
        }
      ]
    },
    {
      "name": "customer",
      "refSql": "select * from customer",
      "columns": [
        {
          "name": "custkey",
          "expression": "c_custkey",
          "type": "integer"
        },
        {
          "name": "name",
          "expression": "c_name",
          "type": "varchar"
        }
      ]
    }
  ]
}

Response

The result format is List<List<QueryAnalysis>>. You can check #548 for the detail of QueryAnalysis

[
    // first sql result: SELECT * FROM customer
    [
        {
            "selectItems": [
                ...
            ],
            "relation": {
                ...
            },
            "groupByKeys": [],
            "sortings": [],
            "isSubqueryOrCte": false
        }
    ],
    // second sql result: WITH t1 AS (SELECT * FROM customer) SELECT * FROM t1
    [
        // for with query
        {
            "selectItems": [
                ...
            ],
            "relation": {
                ...
            },
            "groupByKeys": [],
            "sortings": [],
            "isSubqueryOrCte": true
        },
        // for the main query
        {
            "selectItems": [
                ...
            ],
            "relation": {
                ...
            },
            "groupByKeys": [],
            "sortings": [],
            "isSubqueryOrCte": false
        }
    ],
    // third sql result: SELECT * FROM customer UNION SELECT * FROM customer
    [
        // left of union
        {
            "selectItems": [
                ...
            ],
            "relation": {
                ...
            },
            "filter": {
                ...
            },
            "groupByKeys": [],
            "sortings": [],
            "isSubqueryOrCte": false
        },
        // right of union
        {
            "selectItems": [
                ...
            ],
            "relation": {
                ...
            },
            "filter": {
                ...
            },
            "groupByKeys": [],
            "sortings": [],
            "isSubqueryOrCte": false
        }
    ]
]

@goldmedal goldmedal force-pushed the feature/batch-analysis branch from 4a98076 to 2f4195c Compare July 9, 2024 08:01
@goldmedal goldmedal marked this pull request as ready for review July 9, 2024 08:21
@goldmedal goldmedal requested a review from grieve54706 July 9, 2024 08:21
Comment on lines 227 to 231
assert len(result) == 4
assert len(result[0]) == 1
assert len(result[1]) == 1
assert len(result[2]) == 2
assert len(result[3]) == 2
Copy link
Contributor

@grieve54706 grieve54706 Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add more expected results to let other contributors know the result content format?

@grieve54706
Copy link
Contributor

@goldmedal Could you add more information about the response content in the pull request? To explain why it needs two axes.

Copy link
Contributor

@grieve54706 grieve54706 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks @goldmedal.

@grieve54706 grieve54706 merged commit 0deb0e9 into main Jul 10, 2024
4 checks passed
@grieve54706 grieve54706 deleted the feature/batch-analysis branch July 10, 2024 06:25
@goldmedal
Copy link
Contributor Author

Thanks @grieve54706

grieve54706 pushed a commit that referenced this pull request Dec 13, 2024
* remove unused api

* add Deprecated annotation for v1 api

* add analysis batch api

* add required ibis api

* enhance python test

* fix test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants