Release 2.18.0

This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.18.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* timescale#7104: Hypercore table access method.
* timescale#6901: Add hypertable support for transition tables.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7433: Add support for merging chunks.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7436: Add index creation on `order by` columns.
* timescale#7443: Add hypercore function and view aliases.
* timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565: Add hint when hypertable creation fails.
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* timescale#7486: Prevent building against PostgreSQL versions with broken ABI.
* timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517: Fix the performance regression on the `cagg_migrate` procedure.
* timescale#7527: Restart scheduler on error.
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566: Improve transaction check in CAGG refresh.
* timescale#7584: Fix NaN-handling for vectorized aggregation.
* timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.

Author: pallavisontakke

.unreleased/compressed-sort-transform

@@ -4,6 +4,87 @@
 `psql` with the `-X` flag to prevent any `.psqlrc` commands from
 accidentally triggering the load of a previous DB version.**
 
+
+## 2.18.0 (2025-01-23)
+
+This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.
+
+**Highlighted features in TimescaleDB v2.18.0**
+
+* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
+* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
+* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
+* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.
+
+**Dropping support for Bitnami images**
+
+After the recent change in Bitnami’s [LTS support policy](https://github.com/bitnami/containers/issues/75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)
+
+**Deprecation Notice**
+
+We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:
+
+| Deprecated | Replacement | Type |
+| --- | --- | --- |
+| decompress_chunk | convert_to_rowstore | Procedure |
+| compress_chunk | convert_to_columnstore | Procedure | 
+| add_compression_policy | add_columnstore_policy | Function | 
+| remove_compression_policy | remove_columnstore_policy | Function | 
+| hypertable_compression_stats | hypertable_columnstore_stats | Function | 
+| chunk_compression_stats | chunk_columnstore_stats | Function | 
+| hypertable_compression_settings | hypertable_columnstore_settings | View | 
+| chunk_compression_settings | chunk_columnstore_settings | View | 
+| compression_settings | columnstore_settings | View | 
+| timescaledb.compress | timescaledb.enable_columnstore | Parameter | 
+| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | 
+| timescaledb.compress_orderby  | timescaledb.orderby | Parameter | 
+
+**Features**
+* #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
+* #7104: Hypercore table access method.
+* #6901: Add hypertable support for transition tables.
+* #7482: Optimize recompression of partially compressed chunks.
+* #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
+* #7433: Add support for merging chunks.
+* #7271: Push down `order by` in real-time continuous aggregate queries.
+* #7455: Support `drop not null` on compressed hypertables.
+* #7295: Support `alter table set access method` on hypertable.
+* #7411: Change parameter name to enable hypercore table access method.
+* #7436: Add index creation on `order by` columns.
+* #7443: Add hypercore function and view aliases.
+* #7521: Add optional `force` argument to `refresh_continuous_aggregate`.
+* #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
+* #7565: Add hint when hypertable creation fails.
+* #7390: Disable custom `hashagg` planner code.
+* #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
+* #7486: Prevent building against PostgreSQL versions with broken ABI.
+* #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
+* #7413: Add GUC for segmentwise recompression.
+
+**Bugfixes**
+* #7378: Remove obsolete job referencing `policy_job_error_retention`.
+* #7409: Update `bgw_job` table when altering procedure.
+* #7410: Fix the `aggregated compressed column not found` error on aggregation query.
+* #7426: Fix `datetime` parsing error in chunk constraint creation.
+* #7432: Verify that the heap tuple is valid before using.
+* #7434: Fix the segfault when internally setting the replica identity for a given chunk.
+* #7488: Emit error for transition table trigger on chunks.
+* #7514: Fix the error: `invalid child of chunk append`.
+* #7517: Fix the performance regression on the `cagg_migrate` procedure.
+* #7527: Restart scheduler on error.
+* #7557: Fix null handling for in-memory tuple filtering.
+* #7566: Improve transaction check in CAGG refresh.
+* #7584: Fix NaN-handling for vectorized aggregation.
+* #7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.
+
+**Thanks**
+* @bharrisau for reporting the segfault when creating chunks.
+* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
+* @k-rus for suggesting that we add a hint when hypertable creation fails.
+* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
+* @uasiddiqi for reporting the `aggregated compressed column not found` error.
+
+
 ## 2.17.2 (2024-11-06)
 
 This release contains bug fixes since the 2.17.1 release. We recommend that you

.unreleased/nan-vectorized-filters

@@ -45,7 +45,8 @@ set(MOD_FILES
     updates/2.16.0--2.16.1.sql
     updates/2.16.1--2.17.0.sql
     updates/2.17.0--2.17.1.sql
-    updates/2.17.1--2.17.2.sql)
+    updates/2.17.1--2.17.2.sql
+    updates/2.17.2--2.18.0.sql)
 
 # The downgrade file to generate a downgrade script for the current version, as
 # specified in version.config
@@ -90,7 +91,8 @@ set(OLD_REV_FILES
     2.16.1--2.16.0.sql
     2.17.0--2.16.1.sql
     2.17.1--2.17.0.sql
-    2.17.2--2.17.1.sql)
+    2.17.2--2.17.1.sql
+    2.18.0--2.17.2.sql)
 
 set(MODULE_PATHNAME "$libdir/timescaledb-${PROJECT_VERSION_MOD}")
 set(LOADER_PATHNAME "$libdir/timescaledb")

.unreleased/pr_6901

@@ -2,9 +2,9 @@
 -- Please see the included NOTICE for copyright information and
 -- LICENSE-APACHE for a copy of the license.
 
-CREATE OR REPLACE FUNCTION @extschema@.ts_hypercore_handler(internal) RETURNS table_am_handler
+CREATE OR REPLACE FUNCTION ts_hypercore_handler(internal) RETURNS table_am_handler
 AS '@MODULE_PATHNAME@', 'ts_hypercore_handler' LANGUAGE C;
 
-CREATE OR REPLACE FUNCTION @extschema@.ts_hypercore_proxy_handler(internal) RETURNS index_am_handler
+CREATE OR REPLACE FUNCTION ts_hypercore_proxy_handler(internal) RETURNS index_am_handler
 AS '@MODULE_PATHNAME@', 'ts_hypercore_proxy_handler' LANGUAGE C;
 

.unreleased/pr_7104

@@ -2,10 +2,10 @@
 -- Please see the included NOTICE for copyright information and
 -- LICENSE-APACHE for a copy of the license.
 
-CREATE ACCESS METHOD hypercore TYPE TABLE HANDLER @extschema@.ts_hypercore_handler;
+CREATE ACCESS METHOD hypercore TYPE TABLE HANDLER ts_hypercore_handler;
 COMMENT ON ACCESS METHOD hypercore IS 'Storage engine using hybrid row/columnar compression';
 
-CREATE ACCESS METHOD hypercore_proxy TYPE INDEX HANDLER @extschema@.ts_hypercore_proxy_handler;
+CREATE ACCESS METHOD hypercore_proxy TYPE INDEX HANDLER ts_hypercore_proxy_handler;
 COMMENT ON ACCESS METHOD hypercore_proxy IS 'Hypercore proxy index access method';
 
 -- An index AM needs at least one operator class for the column type

.unreleased/pr_7271

@@ -0,0 +1,172 @@
+-- remove obsolete job
+DELETE FROM _timescaledb_config.bgw_job WHERE id = 2;
+
+-- Hypercore updates
+CREATE FUNCTION _timescaledb_debug.is_compressed_tid(tid) RETURNS BOOL
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder' LANGUAGE C STRICT;
+
+DROP FUNCTION IF EXISTS @[email protected]_chunk(uncompressed_chunk REGCLASS,	if_not_compressed BOOLEAN, recompress BOOLEAN);
+
+CREATE FUNCTION @[email protected]_chunk(
+    uncompressed_chunk REGCLASS,
+    if_not_compressed BOOLEAN = true,
+    recompress BOOLEAN = false,
+    hypercore_use_access_method BOOL = NULL
+) RETURNS REGCLASS AS '@MODULE_PATHNAME@', 'ts_update_placeholder' LANGUAGE C VOLATILE;
+
+DROP FUNCTION IF EXISTS @[email protected]_compression_policy(hypertable REGCLASS, compress_after "any", if_not_exists BOOL, schedule_interval INTERVAL, initial_start TIMESTAMPTZ, timezone TEXT, compress_created_before INTERVAL);
+
+CREATE FUNCTION @[email protected]_compression_policy(
+    hypertable REGCLASS,
+    compress_after "any" = NULL,
+    if_not_exists BOOL = false,
+    schedule_interval INTERVAL = NULL,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+    compress_created_before INTERVAL = NULL,
+    hypercore_use_access_method BOOL = NULL
+)
+RETURNS INTEGER
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+DROP FUNCTION IF EXISTS timescaledb_experimental.add_policies(relation REGCLASS, if_not_exists BOOL, refresh_start_offset "any", refresh_end_offset "any", compress_after "any", drop_after "any");
+
+CREATE FUNCTION timescaledb_experimental.add_policies(
+    relation REGCLASS,
+    if_not_exists BOOL = false,
+    refresh_start_offset "any" = NULL,
+    refresh_end_offset "any" = NULL,
+    compress_after "any" = NULL,
+    drop_after "any" = NULL,
+    hypercore_use_access_method BOOL = NULL)
+RETURNS BOOL
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+DROP PROCEDURE IF EXISTS _timescaledb_functions.policy_compression_execute(job_id INTEGER, htid INTEGER, lag ANYELEMENT, maxchunks INTEGER, verbose_log BOOLEAN, recompress_enabled  BOOLEAN, use_creation_time BOOLEAN);
+
+DROP PROCEDURE IF EXISTS _timescaledb_functions.policy_compression(job_id INTEGER, config JSONB);
+
+CREATE PROCEDURE @[email protected]_to_columnstore(
+    chunk REGCLASS,
+    if_not_columnstore BOOLEAN = true,
+    recompress BOOLEAN = false,
+    hypercore_use_access_method BOOL = NULL)
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C;
+
+CREATE PROCEDURE @[email protected]_to_rowstore(
+    chunk REGCLASS,
+    if_columnstore BOOLEAN = true)
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C;
+
+CREATE PROCEDURE @[email protected]_columnstore_policy(
+    hypertable REGCLASS,
+    after "any" = NULL,
+    if_not_exists BOOL = false,
+    schedule_interval INTERVAL = NULL,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+    created_before INTERVAL = NULL,
+    hypercore_use_access_method BOOL = NULL
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE PROCEDURE @[email protected]_columnstore_policy(
+       hypertable REGCLASS,
+       if_exists BOOL = false
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE FUNCTION @[email protected]_columnstore_stats (hypertable REGCLASS)
+    RETURNS TABLE (
+        chunk_schema name,
+        chunk_name name,
+        compression_status text,
+        before_compression_table_bytes bigint,
+        before_compression_index_bytes bigint,
+        before_compression_toast_bytes bigint,
+        before_compression_total_bytes bigint,
+        after_compression_table_bytes bigint,
+        after_compression_index_bytes bigint,
+        after_compression_toast_bytes bigint,
+        after_compression_total_bytes bigint,
+        node_name name)
+    LANGUAGE SQL
+    STABLE STRICT
+    AS 'SELECT * FROM @[email protected]_compression_stats($1)'
+    SET search_path TO pg_catalog, pg_temp;
+
+CREATE FUNCTION @[email protected]_columnstore_stats (hypertable REGCLASS)
+    RETURNS TABLE (
+        total_chunks bigint,
+        number_compressed_chunks bigint,
+        before_compression_table_bytes bigint,
+        before_compression_index_bytes bigint,
+        before_compression_toast_bytes bigint,
+        before_compression_total_bytes bigint,
+        after_compression_table_bytes bigint,
+        after_compression_index_bytes bigint,
+        after_compression_toast_bytes bigint,
+        after_compression_total_bytes bigint,
+        node_name name)
+    LANGUAGE SQL
+    STABLE STRICT
+    AS 'SELECT * FROM @[email protected]_compression_stats($1)'
+    SET search_path TO pg_catalog, pg_temp;
+
+-- Recreate `refresh_continuous_aggregate` procedure to add `force` argument
+DROP PROCEDURE IF EXISTS @[email protected]_continuous_aggregate (continuous_aggregate REGCLASS, window_start "any", window_end "any");
+
+CREATE PROCEDURE @[email protected]_continuous_aggregate(
+    continuous_aggregate     REGCLASS,
+    window_start             "any",
+    window_end               "any",
+    force                    BOOLEAN = FALSE
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+-- Add `include_tiered_data` argument to `add_continuous_aggregate_policy`
+DROP FUNCTION @[email protected]_continuous_aggregate_policy(
+    continuous_aggregate REGCLASS, start_offset "any",
+    end_offset "any", schedule_interval INTERVAL,
+    if_not_exists BOOL,
+    initial_start TIMESTAMPTZ,
+    timezone TEXT
+);
+CREATE FUNCTION @[email protected]_continuous_aggregate_policy(
+    continuous_aggregate REGCLASS, start_offset "any",
+    end_offset "any", schedule_interval INTERVAL,
+    if_not_exists BOOL = false,
+    initial_start TIMESTAMPTZ = NULL,
+    timezone TEXT = NULL,
+	include_tiered_data BOOL = NULL
+)
+RETURNS INTEGER
+AS '@MODULE_PATHNAME@', 'ts_update_placeholder'
+LANGUAGE C VOLATILE;
+
+-- Merge chunks
+CREATE PROCEDURE @[email protected]_chunks(
+    chunk1 REGCLASS, chunk2 REGCLASS
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE PROCEDURE @[email protected]_chunks(
+    chunks REGCLASS[]
+) LANGUAGE C AS '@MODULE_PATHNAME@', 'ts_update_placeholder';
+
+CREATE FUNCTION ts_hypercore_handler(internal) RETURNS table_am_handler
+AS '@MODULE_PATHNAME@', 'ts_hypercore_handler' LANGUAGE C;
+
+CREATE FUNCTION ts_hypercore_proxy_handler(internal) RETURNS index_am_handler
+AS '@MODULE_PATHNAME@', 'ts_hypercore_proxy_handler' LANGUAGE C;
+
+CREATE ACCESS METHOD hypercore TYPE TABLE HANDLER ts_hypercore_handler;
+COMMENT ON ACCESS METHOD hypercore IS 'Storage engine using hybrid row/columnar compression';
+
+CREATE ACCESS METHOD hypercore_proxy TYPE INDEX HANDLER ts_hypercore_proxy_handler;
+COMMENT ON ACCESS METHOD hypercore_proxy IS 'Hypercore proxy index access method';
+
+CREATE OPERATOR CLASS int4_ops
+DEFAULT FOR TYPE int4 USING hypercore_proxy AS
+       OPERATOR 1 = (int4, int4),
+       FUNCTION 1 hashint4(int4);