feat(ibis): bump ibis to 10.0.0 #1060
Conversation
github-actions
bot
added
bigquery
ibis
dependencies
|
Warning Rate limit exceeded@goldmedal has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 6 minutes and 14 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (1)
WalkthroughThis pull request makes several updates across the project. In the core files, the Dockerfile now installs the MySQL client library, the default SSL mode for MySQL connections is changed from Changes
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
ibis-server/tests/routers/v2/connector/test_trino.py (1)
115-115: LGTM! Date format standardization is complete.The update to use full timestamp format completes the standardization across all database connectors. This change in ibis 10.0.0 ensures consistent date handling regardless of the underlying database.
Consider documenting this breaking change in the upgrade guide to help users update their date format expectations.
ibis-server/Dockerfile (1)
52-54: Optimize image size by combining package installations.While the MySQL client library installation is correctly placed, it should be combined with the PostgreSQL client installation to ensure proper cleanup of apt lists and minimize the image size.
Apply this diff to optimize the Dockerfile:
-# Install libmysqlclient-dev for mysql -RUN apt-get install -y default-libmysqlclient-dev - # libpq-dev is required for psycopg2 -RUN apt-get -y install libpq-dev \ +# Install database client libraries +RUN apt-get -y install \ + default-libmysqlclient-dev \ + libpq-dev \ && rm -rf /var/lib/apt/lists/*
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
ibis-server/poetry.lockis excluded by!**/*.lock
📒 Files selected for processing (14)
ibis-server/Dockerfile(1 hunks)ibis-server/app/model/__init__.py(1 hunks)ibis-server/pyproject.toml(2 hunks)ibis-server/tests/routers/v2/connector/test_bigquery.py(1 hunks)ibis-server/tests/routers/v2/connector/test_canner.py(1 hunks)ibis-server/tests/routers/v2/connector/test_clickhouse.py(2 hunks)ibis-server/tests/routers/v2/connector/test_mssql.py(1 hunks)ibis-server/tests/routers/v2/connector/test_mysql.py(4 hunks)ibis-server/tests/routers/v2/connector/test_postgres.py(3 hunks)ibis-server/tests/routers/v2/connector/test_snowflake.py(1 hunks)ibis-server/tests/routers/v2/connector/test_trino.py(1 hunks)ibis-server/tests/routers/v3/connector/bigquery/test_functions.py(1 hunks)ibis-server/tests/routers/v3/connector/bigquery/test_query.py(1 hunks)ibis-server/tests/routers/v3/connector/postgres/test_query.py(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: ci
🔇 Additional comments (18)
ibis-server/tests/routers/v3/connector/bigquery/test_functions.py (1)
97-97: LGTM: Date format standardization.The updated date format aligns with ibis 10.0.0's standardized timestamp format for date values.
ibis-server/tests/routers/v3/connector/bigquery/test_query.py (1)
87-87: LGTM: Consistent date format update.The date format change aligns with ibis 10.0.0's standardized timestamp format.
ibis-server/tests/routers/v2/connector/test_snowflake.py (1)
93-93: LGTM: Standardized date format.The updated date format is consistent with ibis 10.0.0's standardized timestamp format across all connectors.
ibis-server/tests/routers/v3/connector/postgres/test_query.py (1)
121-121: LGTM! Note the breaking change in date format.The update to use full timestamp format
"1996-01-02 00:00:00.000000"aligns with ibis 10.0.0's standardized date handling. This is a breaking change that may affect consumers expecting the previous date-only format.✅ Verification successful
🌐 Web query:
What are the date handling changes in ibis 10.0.0?💡 Result:
Ibis 10.0 introduced several significant changes to date and timestamp handling through API standardization and new features:
Date/Timestamp Construction Changes
- Component-based creation now requires explicit positional arguments without timezone support:
# Valid component usage ibis.date(2023, 1, 2) ibis.timestamp(2023, 1, 2, 3, 4, 5) # Timezone specification prohibited with components ibis.timestamp(2023, 1, 2, tz="UTC") # Raises NotImplementedError[1][4]- String parsing remains supported:
ibis.date("2023-01-02") ibis.timestamp("2023-01-02T03:04:05")API Standardization
- Temporal methods became positional-only:
as_timestamp(unit)argument position enforced[4]StringValue.as_time()added for string-to-time conversion[4]- Scalar return types changed from numpy to native Python objects:
# Before: numpy.datetime64 # Now: datetime.date expr = ibis.date(2023, 1, 2) expr.execute() # returns datetime.date(2023, 1, 2)[4]Backend Changes Affecting Dates
- Pandas backend removal requires using DuckDB/DataFusion for DataFrame operations:
# Still valid but executes through different backend df = pd.DataFrame({'date': ['2023-01-01']}) t = ibis.memtable(df)[4][7]- Schema handling standardized across backends for temporal types[4][7]
These changes improve consistency across Ibis backends while requiring explicit temporal component specification. Users converting from older versions should update timestamp construction patterns and expect native Python date objects instead of numpy types[1][4][7].
Citations:
- 1: https://github.com/ibis-project/ibis/blob/main/ibis/expr/api.py
- 2: https://www.ibm.com/docs/en/iii/10.0.0?topic=100-infosphere-identity-insight-release-notes
- 3: https://cdrdv2-public.intel.com/705023/ug-qpp-programmer-18-1-683039-705023.pdf
- 4: https://ibis-project.org/release_notes
- 5: https://ibis.org/ver7.0/ver7_0.pdf
- 6: https://www.youtube.com/watch?v=MtIJyumE5OU
- 7: https://ibis-project.org/reference/expression-tables
- 8: https://ibis.org/ver5.0/ver5_0.doc
- 9: https://www.mouser.com/pdfDocs/Kintex-UltraScale-ds892-kintex-ultrascale-data-sheet.pdf
- 10: https://github.com/ibis-project/ibis/actions/runs/7378899166/workflow
LGTM! Date format update aligns with ibis 10.0.0's new timestamp handling.
The change to using
"1996-01-02 00:00:00.000000"matches the updated string parsing behavior in ibis 10.0.0, which now standardizes full timestamp representations rather than date-only formats. This is a breaking change that consumers need to be aware of, but the test adjustment correctly reflects the new behavior.ibis-server/tests/routers/v2/connector/test_canner.py (1)
98-98: LGTM! Date format change is consistent.The update to use full timestamp format matches the standardized date handling across all connectors.
ibis-server/tests/routers/v2/connector/test_bigquery.py (1)
94-94: LGTM! Date format change is consistent.The update to use full timestamp format maintains consistency with other database connectors.
ibis-server/tests/routers/v2/connector/test_mssql.py (1)
128-128: LGTM: Date format standardization.The change from
"1996-01-02"to"1996-01-02 00:00:00.000000"aligns with the standardization of date formats across all database connectors, ensuring consistent timestamp handling after the ibis upgrade.ibis-server/tests/routers/v2/connector/test_mysql.py (4)
5-8: LGTM: Import changes for MySQL authentication.The import changes support the new authentication flow:
- Added
pymysqlfor initial connection to change authentication plugin- Updated
OperationalErrorimport to useMySQLdbwhich is used by ibis
120-138: Well-documented MySQL authentication workaround.The implementation and comments clearly explain the need to change the authentication plugin from
caching_sha2_passwordtomysql_native_passwordwhen SSL is disabled. This is necessary because:
- MySQL's default
caching_sha2_passwordrequires SSL- MysqlDB (used by ibis) only supports
caching_sha2_passwordwith SSL
441-445: LGTM: Improved SSL error message.The updated error message
"SSL connection error: SSL is required but the server doesn't support it"provides better clarity about the SSL requirement compared to the previous generic "Bad handshake" message.
163-163: LGTM: Date format standardization.The change from
"1996-01-02"to"1996-01-02 00:00:00.000000"aligns with the standardization of date formats across all database connectors, ensuring consistent timestamp handling after the ibis upgrade.ibis-server/tests/routers/v2/connector/test_clickhouse.py (2)
183-183: LGTM: Date format standardization.The change from
"1996-01-02"to"1996-01-02 00:00:00.000000"aligns with the standardization of date formats across all database connectors, ensuring consistent timestamp handling after the ibis upgrade.
522-522: Verify the change in table ordering.The change from
response.json()[1]toresponse.json()[0]suggests a change in how tables are ordered in the response after the ibis upgrade.✅ Verification successful
🏁 Script executed:
#!/bin/bash # Description: Check if the orders table is consistently the first table in the response. # Test: Search for test_metadata_list_tables implementations across all connectors. # Expect: Similar changes in array indexing. rg -A 5 'test_metadata_list_tables'Length of output: 7021
I'll inspect the surrounding lines of
ibis-server/tests/routers/v2/connector/test_clickhouse.pyaround line 522 to verify the intended ordering change. Let’s view lines 520–525 to see how the response is being indexed.
🏁 Script executed:
#!/bin/bash sed -n '520,525p' ibis-server/tests/routers/v2/connector/test_clickhouse.pyLength of output: 296
Verification Confirmed: Table Ordering Updated
The test now explicitly asserts that the table at index 0 is "test.orders" (i.e.
assert result["name"] == "test.orders"). This confirms that the ordering change following the ibis upgrade is deliberate and correctly validated in the clickhouse tests.ibis-server/tests/routers/v2/connector/test_postgres.py (3)
146-146: LGTM: Date format standardization.The change from
"1996-01-02"to"1996-01-02 00:00:00.000000"aligns with the standardization of date formats across all database connectors, ensuring consistent timestamp handling after the ibis upgrade.
299-301: LGTM: Improved error message pattern matching.The change to use a regex pattern
r'.*FATAL: password authentication failed for user "test".*'provides more flexible matching of PostgreSQL authentication error messages.
6-6: Verify psycopg version compatibility.The change from
psycopg2topsycopgindicates an upgrade to a newer PostgreSQL driver. Please verify compatibility with the latest version.✅ Verification successful
🌐 Web query:
What is the latest version of psycopg and what are the key differences from psycopg2?💡 Result:
The latest version of Psycopg is 3.2.4 (released January 15, 2025)[8], with Psycopg 3 representing a major evolution from Psycopg2 (last updated in October 2024)[6]. Here are the key differences between Psycopg3 and Psycopg2:
Architecture and Design
- Language base:
- Psycopg2: Primarily written in C[2][6].
- Psycopg3: Mostly Python with optional C acceleration[2][7].
- Dependencies:
- Psycopg3 simplifies installation with pre-compiled bundles (
psycopgpackage)[2], while Psycopg2 requires separatepsycopg2/psycopg2-binarypackages[5][6].Feature Comparison
Feature Psycopg2 Psycopg3 Python Support Up to Python 3.12[6] Up to Python 3.13[2][8] PostgreSQL Support PostgreSQL 7.4–17[2] PostgreSQL 10–17[2] Async Support Limited (via third-party libs)[2] Native async/await support[2][7] COPY Operations File-based interface[3][7] Object-oriented API for dynamic data[7] Prepared Statements Not supported[2] Automatic or opt-in usage[2][7] Type Adaptation Basic type mapping[3] Advanced support (JSONB, ranges)[2][7] Performance Improvements
Psycopg3 shows ~300% faster data retrieval speeds in benchmarks compared to Psycopg2 (500k vs 150k rows/sec)[9]. This stems from:
- Binary protocol usage
- Pipeline mode for batch operations[2]
- Optimized memory management
API Changes
Multi-statement execution:
- Psycopg2 returns only last query's results
- Psycopg3 provides access to all results via
cursor.nextset()[7]# Psycopg3 example: cur.execute("SELECT 1; SELECT 2") print(cur.fetchall()) # [(1,)] cur.nextset() print(cur.fetchall()) # [(2,)][8]Context managers:
- Closing connection context manager now terminates transactions and connections[3][7]
Compatibility Notes
While maintaining DBAPI 2.0 compliance:
- Requires libpq v17 for full feature set (PostgreSQL ≥17)[1]
- Default date handling avoids automatic infinity conversions[7]
For new projects, Psycopg3 is recommended due to its modern feature set and performance gains[2][9]. Existing systems using Psycopg2 remain supported but won't receive new features[6].
Citations:
- 1: https://www.postgresql.org/about/news/psycopg-32-released-2887/
- 2: https://www.psycopg.org/features/
- 3: https://dokk.org/documentation/psycopg/3.0.18/basic/from_pg2/
- 4: https://www.psycopg.org/docs/news.html
- 5: https://pysql.tecladocode.com/section05/lectures/04_psycopg2_vs_psycopg2-binary/
- 6: https://pypi.org/project/psycopg2/
- 7: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html
- 8: https://pypi.org/project/psycopg/
- 9: https://www.timescale.com/blog/psycopg2-vs-psycopg3-performance-benchmark
- 10: https://github.com/psycopg/psycopg/actions
Psycopg Upgrade Confirmed
The upgrade from
psycopg2topsycopgis valid. The latest version (3.2.4) brings major enhancements—such as native async support, improved performance, and modern API changes—that justify its use in new projects. Just ensure that any features affected by the API changes (e.g., multi-statement execution and context management) are fully tested in your integration.ibis-server/pyproject.toml (2)
36-36: Consider making mysqlclient a required dependency.Since MySQL support is included in the ibis-framework extras and the Dockerfile installs the MySQL client library by default, consider making
mysqlclienta required dependency instead of optional.Let's verify if mysqlclient is required for MySQL support in ibis-framework 10.0.0:
✅ Verification successful
🌐 Web query:
Is mysqlclient required for MySQL support in ibis-framework 10.0.0?💡 Result:
To use MySQL with ibis-framework 10.0.0, mysqlclient is required. This change was introduced when Ibis switched from using
pymysqlto theMySQLdbdriver (provided by mysqlclient) for MySQL connectivity[7].Key details:
- The mysqlclient package provides the
MySQLdbinterface needed by Ibis's MySQL backend[6][7].- Installation requires either:
or explicit installation viapip install 'ibis-framework[mysql]' # Installs mysqlclient as a dependencypip install mysqlclient[1][6].- Some systems may require additional MySQL client libraries (e.g., via
brew install mysql-clienton macOS or system packages likelibmysqlclient-devon Ubuntu)[3][6].This requirement applies specifically to ibis-framework versions ≥10.x that implemented this backend switch[7]. Earlier versions used different drivers without requiring mysqlclient.
Citations:
- 1: https://ibis-project.org/backends/mysql
- 2: https://ibis-project.org/install
- 3: https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-hdp/topics/hue-install-mysql-client-mysql-db.html
- 4: https://dev.mysql.com/blog-archive/are-you-ready-for-mysql-10/
- 5: https://github.com/ibis-project/ibis/blob/main/pyproject.toml
- 6: https://pypi.org/project/mysqlclient/
- 7: https://ibis-project.org/release_notes
- 8: https://secure.software/pypi/packages/ibis-framework/dependencies/10.0.0.dev256
- 9: https://github.com/All-Hands-AI/OpenHands/blob/main/poetry.lock
- 10: https://pypi.org/project/ibis-framework/
Make
mysqlclienta required dependency. The web result confirms that in ibis-framework 10.0.0,mysqlclientis essential for MySQL connectivity. Given that MySQL support is provided via the extras (and the Dockerfile already installs the MySQL client library), it's appropriate to remove the optional flag from the dependency inibis-server/pyproject.toml(line 36).
- File Impacted:
ibis-server/pyproject.toml(line 36)
13-21: Verify breaking changes in ibis-framework 10.0.0.The major version bump from 9.5.0 to 10.0.0 could introduce breaking changes.
Let's check the changelog and release notes:
✅ Verification successful
🌐 Web query:
What are the breaking changes in ibis-framework 10.0.0?💡 Result:
The release of Ibis 10.0.0 introduces several breaking changes across APIs, backends, and configurations:
API Standardization
- Positional-only arguments enforced for:
- Interval/timestamp units (
as_interval,as_timestamp)- Table operations (
drop_null,fill_null,window_by,alias,sample,aggregate)- Set operations (
union,intersect, etc.)- Type casting (
cast,try_cast)- Window functions (
nth,ntile, preceding/following ranges)- Join conditions (
howparameter keyword-only)- Required keyword arguments added for:
- Struct type definitions (
struct(..., type=...))- Aggregate function filters (
where=...)- Geospatial distance checks (
d_within(distance=...))Backend-Specific Changes
DuckDB:
- CSV type specifications require DuckDB-native type names (e.g.,
"double"instead of"float64") [1]- Removed
read_in_memory()– useibis.memtable()instead [1]- Bumped minimum version to v0.10 with storage compatibility requirements [1]
MySQL:
- Switched driver from PyMySQL to MySQLdb (may require client library updates) [1]
Removed Backends:
- Pandas and Dask execution backends removed – use DuckDB/Polars/DataFusion for pandas DataFrames [1]
Major Feature Removals
- Deprecated methods deleted:
Table.relabel()StringValue.initcap()(use.capitalize())- Top-level geo functions (moved to column methods)
.greatest()/.least()column methods (useibis.greatest()/ibis.least())- Configuration system:
- Removed legacy
option_contextmanager [1]- In-memory joins with pandas/pyarrow objects now require explicit
.memtable()conversion [1]Behavioral Changes
- Scalar values returned as native Python types instead of numpy objects [1]
- String padding operations now follow Python semantics (leave long strings untouched) [1]
- Schema/database terminology standardized:
- schema: Column name → type mapping
- database: Table collection
- catalog: Database collection
These changes aim to standardize APIs across backends and simplify Ibis' architecture while maintaining compatibility with pandas DataFrames as input/output formats through other supported engines like DuckDB[1][7]. Users should audit their code for deprecated method usage and update type specifications/argument positions where needed[1][5].
Citations:
- 1: https://ibis-project.org/release_notes
- 2: https://github.com/ibis-project/ibis-substrait/blob/main/CHANGELOG.md
- 3: https://ibispaint.com/newFeature.jsp?lang=en
- 4: https://github.com/ibis-project/ibis/blob/main/pyproject.toml
- 5: https://ibis-project.org
- 6: https://www.kaggle.com/code/rkuo2000/fine-tuning-tinyllama/log
- 7: https://pypi.org/project/ibis-framework/
- 8: https://www.ibm.com/docs/en/iii/10.0.0?topic=100-infosphere-identity-insight-release-notes
- 9: perf: Ibis slows down considerably for wider tables ibis-project/ibis#9111
- 10: https://www.wheelodex.org/projects/ibis-framework/
Bump to ibis-framework 10.0.0 – Confirmed Breaking Changes
The web query confirms that upgrading to ibis-framework 10.0.0 introduces several breaking changes. Notable updates include:
API Standardization:
- Enforced positional-only arguments for various functions (e.g., interval/timestamp conversions, table operations, type casting, window functions, join parameters).
- Required keyword arguments for struct type definitions, aggregate filters, and geospatial distance checks.
Backend-Specific Changes:
- In DuckDB, CSV type specifications now require DuckDB-native type names and the removal of
read_in_memory().- For MySQL, a switch from PyMySQL to MySQLdb may require client library updates.
Removed Backends & Deprecated Methods:
- Removal of Pandas and Dask execution backends in favor of using alternatives such as DuckDB or Polars.
- Deprecated methods like
Table.relabel(),StringValue.initcap(), and others have been removed in favor of updated APIs.Behavioral Adjustments:
- Scalar values now return as native Python types instead of numpy objects.
- Changes in string padding semantics and standardization of schema/database terminology.
These changes are well-documented in the release notes and require a review of any dependent code to ensure compatibility.
github-actions
bot
added
the
documentation
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
ibis-server/docs/development.md (3)
98-98: Fix grammar in the introduction sentence.There's a grammatical error in this sentence.
-If you want run tests related to MySQL Server +If you want to run tests related to MySQL Server🧰 Tools
🪛 LanguageTool
[grammar] ~98-~98: Did you mean “to run”?
Context: ...No driver for MySQL Server If you want run tests related to MySQL Server or connec...(MISSING_TO_BEFORE_A_VERB)
99-99: Fix grammar and improve clarity in the Linux instructions.The sentence has incorrect preposition usage and could be clearer.
-- For linux, you can install `libmysqlclient-dev`. By the way, there are some different names for different linux versions. You should take care about it. +- For Linux, you can install `default-libmysqlclient-dev`. Note that package names may vary across different Linux distributions.🧰 Tools
🪛 LanguageTool
[grammar] ~99-~99: The correct preposition in this context would be “of”.
Context: ...nt linux versions. You should take care about it. - For Mac, you can install `mysql-c...(TAKE_CARE_ABOUT)
101-101: Fix typo in the Windows instructions.There's a typo in "dowanload".
-- For Windows, you can dowanload [the libraries](https://dev.mysql.com/downloads/c-api) +- For Windows, you can download [the libraries](https://dev.mysql.com/downloads/c-api)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
ibis-server/docs/development.md(1 hunks)ibis-server/tests/routers/v2/connector/test_clickhouse.py(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- ibis-server/tests/routers/v2/connector/test_clickhouse.py
🧰 Additional context used
🪛 LanguageTool
ibis-server/docs/development.md
[grammar] ~98-~98: Did you mean “to run”?
Context: ...No driver for MySQL Server If you want run tests related to MySQL Server or connec...
(MISSING_TO_BEFORE_A_VERB)
[grammar] ~99-~99: The correct preposition in this context would be “of”.
Context: ...nt linux versions. You should take care about it. - For Mac, you can install `mysql-c...
(TAKE_CARE_ABOUT)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: ci
| ssl_mode: SecretStr | None = Field( | ||
| alias="sslMode", | ||
| default="ENABLED", | ||
| description="Use ssl connection or not. The default value is `ENABLED` because MySQL uses `caching_sha2_password` by default and the driver MySQLdb support caching_sha2_password with ssl only.", | ||
| ) |
There was a problem hiding this comment.
The ssl_model is enabled by default because ibis ports to MySQLdb that only support caching_sha2_password with SSL.
c.c @ongdisheng who implements this.
|
There are some issues with the BigQuery secret. Let's ignore the CI fails of BigQuery testing. |
|
LTGM ! |
Changed
MySQL
mysql-client-devby ourselves.caching_sha2_password, which only supports SSL connections. The SSL disabled mode (SSLMode.DISABLED) supportsmysql_native_passwordonly.MSSQL
Summary by CodeRabbit
Summary by CodeRabbit
New Features
Bug Fixes