add conformance reporting

[yliuuuu]

Relevant Issues

• [Closes/Related To] N/A

Description

Update: We can now access the full report from Github Action run. In both the step summary and the produced artifact. See link.

• This PRs builds on Refactors partiql-tests-runner for multiple engines #1289. The main goal is to have incorporate multiple backend in conformance tests GitHub action.
• For every pull request, the GitHub CI now posts comments on 1) cross engine comparison base on the pull request commit; 2) cross commit comparison for every engine between pull request commit and head of target commit.

For an example comment message, see: yliuuuu#15 (comment)

Note: For the new eval engine: there is one test (path expression with ambiguous table alias (lowercase), compileOption: LEGACY) that is showing up in Success-in-previous-but-failed-in-current and failed-in-previous-but-success-in-current. I suspect that this issue is caused by test cases with duplicated name but different query, but have not yet looked into this.

Other Information

• Updated Unreleased Section in CHANGELOG: [YES/NO]

  • No.

• Any backward-incompatible changes? [YES/NO]

  • No.

• Any new external dependencies? [YES/NO]

  • No.

• Do your changes comply with the Contributing Guidelines
  and Code Style Guidelines? [YES/NO]
  Yes.

License Information

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (partiql-plugin-impl@c7466e4). Click here to learn what that means.

Additional details and impacted files

@@                  Coverage Diff                   @@
##             partiql-plugin-impl    #1333   +/-   ##
======================================================
  Coverage                       ?   49.27%           
  Complexity                     ?     1046           
======================================================
  Files                          ?      166           
  Lines                          ?    13395           
  Branches                       ?     2504           
======================================================
  Hits                           ?     6600           
  Misses                         ?     6138           
  Partials                       ?      657           

Flag	Coverage Δ
CLI	11.86% <0.00%> (?)
EXAMPLES	80.28% <0.00%> (?)
LANG	54.71% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alancai98]

Looks good. Only had a few minor comments. FYI the full list of conformance tests failing in one engine and passing in another can also be found in the CI summary https://github.com/yliuuuu/partiql-lang-kotlin/actions/runs/7483462300/attempts/1#summary-20368831620. From the docs, it appears the limit for the summary is 1MiB:

Job summaries are isolated between steps and each step is restricted to a maximum size of 1MiB.

That size limit is a lot larger than the GH comment size restriction but could be an issue if we were to add more engines or tests to the comparison report. As we get more tests passing in the eval engine, the comparison report size should decrease, so maybe it's not an issue for now.