Release notes 3.2.3

Release date: February 10, 2024

New Features

• [Preview] Supports hybrid row-column storage for tables. It allows better performance for high-concurrency, low-latency point lookups against Primary Key tables and partial data updates. Currently, this feature does not support modification via ALTER TABLE, changing Sort Key, and partial updates in column mode.
• Supports backing up and restoring asynchronous materialized views.
• Broker Load supports loading JSON-type data.
• Supports query rewrite using asynchronous materialized views created upon views. Queries against a view can be rewritten based on materialized views that are created upon that view.
• Supports CREATE OR REPLACE PIPE. #37658

Behavior Changes

• Added the session variable enable_strict_order_by. When this variable is set to the default value TRUE, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, select distinct t1.* from tbl1 t1 order by t1.k1;. The logic is the same as that in v2.3 and earlier. When this variable is set to FALSE, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. #37910
• Added the session variable enable_materialized_view_for_insert, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is false. #37505
• When a single query is executed within the Pipeline framework, its memory limit is now constrained by the variable query_mem_limit instead of exec_mem_limit. Setting the value of query_mem_limit to 0 indicates no limit. #34120

Parameter Changes

• Added the FE configuration item http_worker_threads_num, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is 0. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. #37530
• Added the BE configuration item lake_pk_compaction_max_input_rowsets, which controls the maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data StarRocks cluster. This helps optimize resource consumption for compaction tasks. #39611
• Added the session variable connector_sink_compression_codec, which specifies the compression algorithm used for writing data into Hive tables or Iceberg tables, or exporting data with Files(). Valid algorithms include GZIP, BROTLI, ZSTD, and LZ4. #37912
• Added the FE configuration item routine_load_unstable_threshold_second. #36222
• Added the BE configuration item pindex_major_compaction_limit_per_disk to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is 1. #36681
• Added the BE configuration item enable_lazy_delta_column_compaction. The default value is true, indicating that StarRocks does not perform frequent compaction operations on delta columns. #36654
• Added the FE configuration item default_mv_refresh_immediate, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is true. #37093
• Changed the default value of the FE configuration item default_mv_refresh_partition_numto 1. This indicates that when multiple partitions need to be updated during a materialized view refresh, the task will be split in batches, refreshing only one partition at a time. This helps reduce resource consumption during each refresh. #36560

Improvements

• Added date formats yyyy-MM-ddTHH:mm and yyyy-MM-dd HH:mm to support TIMESTAMP partition fields in Apache Iceberg tables. #39986
• Added Data Cache-related metrics to the monitoring API. #40375
• Optimized BE log printing to prevent too many irrelevant logs. #22820 #36187
• Added the field storage_medium to the view information_schema.be_tablets. #37070
• Supports SET_VAR in multiple sub-queries. #36871
• A new field LatestSourcePosition is added to the return result of SHOW ROUTINE LOAD to record the position of the latest message in each partition of the Kafka topic, helping check the latencies of data loading. #38298
• When the string on the right side of the LIKE operator within the WHERE clause does not include % or _, the LIKE operator is converted into the = operator. #37515
• The default retention period of trash files is changed to 1 day from the original 3 days. #37113
• Supports collecting statistics from Iceberg tables with Partition Transform. #39907
• The scheduling policy for Routine Load is optimized, so that slow tasks do not block the execution of the other normal tasks. #37638

Bug Fixes

Fixed the following issues:

• The execution of ANALYZE TABLE gets stuck occasionally. #36836
• The memory consumption by PageCache exceeds the threshold specified by the BE dynamic parameter storage_page_cache_limit in certain circumstances. #37740
• Hive metadata in Hive catalogs is not automatically refreshed when new fields are added to Hive tables. #37549
• In some cases, bitmap_to_string may return incorrect results due to data type overflow. #37405
• When SELECT ... FROM ... INTO OUTFILE is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. #38045
• In some cases, querying semi-structured data in tables may cause BEs to crash. #40208

Release notes 2.5.19

Release date: February 8, 2024

New features

• Added a pattern-matching function: regexp_extract_all.
• Added Bitmap value processing functions: serialize, deserialize, and serializeToString. #40162

Improvements

• Supports automatic activation of inactive materialized views when refreshing these materialized views. #38521
• Optimized BE log printing to prevent too many irrelevant logs. #22820 #36187
• Supports using Hive UDFs to process and load Bitmap data into StarRocks and export Bitmap data from StarRocks to Hive. #40165 #40168
• Added date formats yyyy-MM-ddTHH:mm and yyyy-MM-dd HH:mm to support TIMESTAMP partition fields in Apache Iceberg tables. #39986

Bug Fixes

Fixed the following issues:

• Running a Spark Load job that has no PROPERTIES specified causes null pointer exceptions (NPEs). #38765
• INSERT INTO SELECT occasionally encounters the error "timeout by txn manager". #36688
• The memory consumption of PageCache exceeds the threshold specified by the BE dynamic parameter storage_page_cache_limit in certain circumstances. #37740
• After a table is dropped and then re-created with the same table name, refreshing asynchronous materialized views created on that table fails. #38008 #38982
• Writing data to S3 buckets using SELECT INTO occasionally encounters the error "The tablet write operation update metadata take a long time". #38443
• Some operations during data loading may encounter "reached timeout". #36746
• The DECIMAL type returned by SHOW CREATE TABLE is inconsistent with that specified in CREATE TABLE. #39297
• If partition columns in external tables contain null values, queries against those tables will cause BEs to crash. #38888
• When deleting data from a Duplicate Key table, if the condition in the WHERE clause of the DELETE statement has a leading space, the deleted data can still be queried using SELECT. #39797
• Loading array<string> data from ORC files into StarRocks (array<json>) may cause BEs to crash. #39233
• Querying Hive catalogs may be stuck and even expire. #39863
• Partitions cannot be dynamically created if hour-level partitions are specified in the PARTITION BY clause. #40256
• The error message "failed to call frontend service" is returned during loading from Apache Flink. #40710

Release notes 3.1.8

Release date: February 6, 2024

New Features

• StarRocks Community provides the StarRocks Cross-cluster Data Migration Tool, which supports migrating data from a shared-nothing cluster to either another shared-nothing cluster or a shared-data cluster.
• Supports creating synchronous materialized views with the WHERE clause specified.
• Added metrics that show memory usage of the data cache to MemTracker. #39600
• Added an array function, array_unique_agg.

Parameter Change

• Added a BE configuration item, lake_pk_compaction_max_input_rowsets, which controls the maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data StarRocks cluster. This helps optimize resource consumption for compaction tasks. #39611

Improvements

• Supports ORDER BY and INDEX clauses in CTAS statements. #38886
• Supports equality deletes on ORC-formatted Iceberg v2 tables. #37419
• Supports setting the datacache.partition_duration property for cloud-native tables created with the list partitioning strategy. This property controls the validity period of the data cache and can be dynamically configured. #35681 #38509
• Optimized the BE configuration item update_compaction_per_tablet_min_interval_seconds. This parameter is originally used only to control the frequency of compaction tasks on Primary Key tables. After the optimization, it can also be used to control the frequency of major compaction tasks on Primary Key table indexes. #39640
• Parquet Reader supports converting INT32-type data in Parquet-formatted data to DATETIME-type data and storing the resulting data to StarRocks. #39808

Bug Fixes

Fixed the following issues:

• Using NaN (Not a Number) columns as ORDER BY columns may cause BEs to crash. #30759
• Failure to update primary key indexes may cause the error "get_applied_rowsets failed". #27488
• The resources occupied by compaction_state_cache are not recycled after compaction task failures. #38499
• If partition columns in external tables contain null values, queries against those tables will cause BEs to crash. #38888
• After a table is dropped and then re-created with the same table name, refreshing asynchronous materialized views created on that table fails. #38008
• Refreshing asynchronous materialized views created on empty Iceberg tables fail. #24068

[Stable] 2.5.18

RRelease date: Jan 10, 2024

New Features

• Users can set or modify session variables when they CREATE or ALTER asynchronous materialized views. #37401

Improvements

• When using JDK, the default GC algorithm is G1. #37498
• The result returned by the SHOW ROUTINE LOAD statement now includes the timestamps of consumption messages from each partition. #36222

Behavior Change

• Added the session variable enable_materialized_view_for_insert, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is false. #37505
• Added the session variable enable_strict_order_by. When this variable is set to the default value TRUE, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, select distinct t1.* from tbl1 t1 order by t1.k1;. The logic is the same as that in v2.3 and earlier. When this variable is set to FALSE, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. #37910

Parameter Change

• Added session variables transaction_read_only and tx_read_only to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. #37249
• Added the FE configuration item routine_load_unstable_threshold_second. #36222
• Added the FE configuration item http_worker_threads_num, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is 0. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. #37530
• Added the BE configuration item pindex_major_compaction_limit_per_disk to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is 1. #37695

Bug Fixes

Fixed the following issues:

• Using NaN (Not a Number) columns as ORDER BY columns may cause BEs to crash. #30759
• Failure to update primary key indexes may cause the error "get_applied_rowsets failed". #27488
• Hive metadata in Hive catalogs is not automatically refreshed when new fields are added to Hive tables. #37668
• When SELECT ... FROM ... INTO OUTFILE is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. #38045
• In some cases, bitmap_to_string may return incorrect result due to data type overflow. #37405

3.1.7

Release date: January 12, 2024

New Features

• Added a new function, unnest_bitmap. #38136
• Supports conditional updates for Broker Load. #37400

Behavior Change

• Added the session variable enable_materialized_view_for_insert, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is false. #37505
• The FE dynamic parameter enable_new_publish_mechanism is changed to a static parameter. You must restart the FE after you modify the parameter settings. #35338
• Added the session variable enable_strict_order_by. When this variable is set to the default value TRUE, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, select distinct t1.* from tbl1 t1 order by t1.k1;. The logic is the same as that in v2.3 and earlier. When this variable is set to FALSE, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. #37910

Parameter Change

• Added the FE configuration item routine_load_unstable_threshold_second. #36222
• Added the FE configuration item http_worker_threads_num, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is 0. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. #37530
• Added the BE configuration item pindex_major_compaction_limit_per_disk to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is 1. #36681
• Added session variables transaction_read_only and tx_read_only to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. #37249
• Added the FE configuration item default_mv_refresh_immediate, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is true. #37093
• Added a new BE configuration item lake_enable_vertical_compaction_fill_data_cache, which specifies whether to allow compaction tasks to cache data on local disks in a shared-data cluster. The default value is false. #37296

Improvements

• INSERT INTO FILE() SELECT FROM supports reading BINARY-type data from tables and exporting the data to Parquet-formatted files in remote storage. #36797
• Asynchronous materialized views support dynamically setting the datacache.partition_duration property, which controls the validity period of the hot data in the data cache. #35681
• Wen using JDK, the default GC algorithm is G1. #37386
• The date_trunc, adddate, and time_slice functions support setting the interval parameter to values that are accurate to the millisecond and microsecond. #36386
• When the string on the right side of the LIKE operator within the WHERE clause does not include % or _, the LIKE operator is converted into the = operator. #37515
• A new field LatestSourcePosition is added to the return result of SHOW ROUTINE LOAD to record the position of the latest message in each partition of the Kafka topic, helping check the latencies of data loading. #38298
• Added a new resource group property, spill_mem_limit_threshold, to control the memory usage threshold (percentage) at which a resource group triggers the spilling of intermediate results when the system variable spill_mode is set to auto. The valid range is (0, 1). The default value is 1, indicating the threshold does not take effect. #37707
• The result returned by the SHOW ROUTINE LOAD statement now includes the timestamps of consumption messages from each partition. #36222
• The scheduling policy for Routine Load is optimized, so that slow tasks do not block the execution of the other normal tasks. #37638

Bug Fixes

Fixed the following issues:

• The execution of ANALYZE TABLE gets stuck occasionally. #36836
• The memory consumption by PageCache exceeds the threshold specified by the BE dynamic parameter storage_page_cache_limit in certain circumstances. #37740
• Hive metadata in Hive catalogs is not automatically refreshed when new fields are added to Hive tables. #37668
• In some cases, bitmap_to_string may return incorrect results due to data type overflow. #37405
• Executing the DELETE statement on an empty table returns "ERROR 1064 (HY000): Index: 0, Size: 0". #37461
• When the FE dynamic parameter enable_sync_publish is set to TRUE, queries on data that is written after the BEs crash and then restart may fail. #37398
• The value of the TABLE_CATALOG field in views of the StarRocks Information Schema is null. #37570
• When SELECT ... FROM ... INTO OUTFILE is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. #38045

3.2.2

Release date: December 30, 2023

Bug Fixes

Fixed the following issue:

• When StarRocks is upgraded from v3.1.2 or earlier to v3.2, FEs may fail to restart. #38172

3.0.9

Release date: January 2, 2024

New features

• Added the percentile_disc function. #36352
• Added a new metric max_tablet_rowset_num for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". #36539

Improvements

• A new value option GROUP_CONCAT_LEGACY is added to the session variable sql_mode to provide compatibility with the implementation logic of the group_concat function in versions earlier than v2.5. #36150
• When using JDK, the default GC algorithm is changed to G1. #37386
• The be_tablets view in the information_schema database provides a new field INDEX_DISK, which records the disk usage (measured in bytes) of persistent indexes #35615
• Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. #35917
• Supports updates onto the specified partitions of an automatically partitioned table. If the specified partitions do not exist, an error is returned. #34777
• The Primary Key table size returned by the SHOW DATA statement includes the sizes of .cols files (these are files related to partial column updates and generated columns) and persistent index files. #34898
• Optimized the performance of persistent index update when compaction is performed on all rowsets of a Primary Key table, which reduces disk read I/O. #36819
• When the string on the right side of the LIKE operator within the WHERE clause does not include % or _, the LIKE operator is converted into the = operator. #37515
• Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. #36534
• The result returned by the SHOW ROUTINE LOAD statement now includes the timestamps of consumption messages from each partition. #36222
• Optimized the performance of some Bitmap-related operations, including:
  • Optimized nested loop joins. #340804 #35003
  • Optimized the bitmap_xor function. #34069
  • Supports Copy on Write to optimize Bitmap performance and reduce memory consumption. #34047

Compatibility Changes

Behavior Change

• Added the session variable enable_materialized_view_for_insert, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is false. #37505
• Changed the FE configuration item enable_new_publish_mechanism to a static parameter from a dynamic one. You must restart the FE after you modify the parameter settings. #35338
• The default retention period of trash files is changed to 1 day from the original 3 days. #37113

Parameters

Session variables

• Added session variable cbo_decimal_cast_string_strict, which controls how the CBO converts data from the DECIMAL type to the STRING type. If this variable is set to true, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to false, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is true. #34208
• Added session variables transaction_read_only and tx_read_only to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. #37249

FE configurations

• Added the FE configuration item routine_load_unstable_threshold_second. #36222
• Added the FE configuration item http_worker_threads_num, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is 0. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. #37530
• Added the FE configuration item default_mv_refresh_immediate, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is true. #37093

BE configurations

• Added the BE configuration item enable_stream_load_verbose_log. The default value is false. With this parameter set to true, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. #36113
• Added the BE configuration item pindex_major_compaction_limit_per_disk to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is 2. #36681
• Added BE configuration items to specify the timeout duration for connecting to object storage:
  • object_storage_connect_timeout_ms: Timeout duration to establish socket connections with object storage. The default value is -1, which means to use the default timeout duration of the SDK configurations.
  • object_storage_request_timeout_ms: Timeout duration to establish HTTP connections with object storage. The default value is -1, which means to use the default timeout duration of the SDK configurations.

Bug Fixes

Fixed the following issues:

• In some cases, BEs may crash when a Catalog is used to read ORC external tables. #27971
• The BEs crash if users create persistent indexes in the event of data corruption. #30841
• BEs occasionally crash after a Bitmap index is added. #26463
• Failures in replaying replica operations may cause FEs to crash. #32295
• Setting the FE parameter recover_with_empty_tablet to true may cause FEs to crash. #33071
• Queries fail during hash joins, causing BEs to crash. #32219
• In a StarRocks shared-nothing cluster, queries against Iceberg or Hive tables may cause BEs to crash. #34682
• The error "get_applied_rowsets failed, tablet updates is in error state: tablet:18849 actual row size changed after compaction" is returned for queries. #33246
• Running show proc '/statistic' may cause a deadlock. #34237
• The FE performance plunges after the FE configuration item enable_collect_query_detail_info is set to true. #35945
• Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. #34352
• After StarRocks is upgraded from v2.4 or earlier to a later version, compaction scores may rise unexpectedly. #34618
• If INFORMATION_SCHEMA is queried by using the database driver MariaDB ODBC, the CATALOG_NAME column returned in the schemata view holds only null values. #34627
• FEs crash due to the abnormal data loaded and cannot restart. #34590
• If schema changes are being executed while a Stream Load job is in the PREPARD state, a portion of the source data to be loaded by the job is lost. #34381
• Including two or more slashes (/) at the end of the HDFS storage path causes the backup and restore of the data from HDFS to fail. #34601
• The partition_live_number property added by using the ALTER TABLE statement does not take effect. #34842
• The array_distinct function occasionally causes the BEs to crash. [#36377](https://github.com/S...

3.2.1

Release date: December 21, 2023

New Features

Data Lake Analytics

• Supports reading Hive Catalog tables and file external tables in Avro, SequenceFile, and RCFile formats through Java Native Interface (JNI).

Materialized View

• Added a view object_dependencies to the database sys. It contains the lineage information of asynchronous materialized views. #35060
• Supports creating synchronous materialized views with the WHERE clause.
• Supports partition-level incremental refresh for asynchronous materialized views created upon Iceberg catalogs.
• [Preview] Supports creating asynchronous materialized views based on tables in a Paimon catalog with partition-level refresh.

Query and SQL functions

• Supports the prepared statement. It allows better performance for processing high-concurrency point lookup queries. It also prevents SQL injection effectively.
• Supports the following Bitmap functions: subdivide_bitmap, bitmap_from_binary, and bitmap_to_binary.
• Supports the Array function array_unique_agg.

Monitoring and alerts

• Added a new metric max_tablet_rowset_num for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". #36539

Parameter changes

• A new BE configuration item enable_stream_load_verbose_log is added. The default value is false. With this parameter set to true, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. #36113

Improvements

• Upgraded the default GC algorithm in JDK8 to G1. #37268
• A new value option GROUP_CONCAT_LEGACY is added to the session variable sql_mode to provide compatibility with the implementation logic of the group_concat function in versions earlier than v2.5. #36150
• The authentication information aws.s3.access_key and aws.s3.access_secret for AWS S3 in Broker Load jobs are hidden in audit logs. #36571
• The be_tablets view in the information_schema database provides a new field INDEX_DISK, which records the disk usage (measured in bytes) of persistent indexes. #35615
• The result returned by the SHOW ROUTINE LOAD statement provides a new field OtherMsg, which shows information about the last failed task. #35806

Bug Fixes

Fixed the following issues:

• The BEs crash if users create persistent indexes in the event of data corruption.#30841
• The array_distinct function occasionally causes the BEs to crash. #36377
• After the DISTINCT window operator pushdown feature is enabled, errors are reported if SELECT DISTINCT operations are performed on the complex expressions of the columns computed by window functions. #36357
• Some S3-compatible object storage returns duplicate files, causing the BEs to crash. #36103

2.5.17

Release date: December 19, 2023

New Features

• Added a new metric max_tablet_rowset_num for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". #36539
• Added the subdivide_bitmap function. #35817

Improvements

• The result returned by the SHOW ROUTINE LOAD statement provides a new field OtherMsg, which shows information about the last failed task. #35806
• The default retention period of trash files is changed to 1 day from the original 3 days. #37113
• Optimized the performance of persistent index update when compaction is performed on all rowsets of a Primary Key table, which reduces disk read I/O. #36819
• Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. #36534
• Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. #35917
• Added the bitmap_from_binary function to Spark Load to support loading Binary data. #36050
• The bRPC expiration time is shortened from 1 hour to the duration specified by the session variable query_timeout. This prevents query failures caused by RPC request expiration. #36778

Compatibility Changes

Parameters

• A new BE configuration item enable_stream_load_verbose_log is added. The default value is false. With this parameter set to true, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. #36113
• The BE static parameter update_compaction_per_tablet_min_interval_seconds becomes mutable. #36819

Bug Fixes

Fixed the following issues:

• Queries fail during hash joins, causing BEs to crash. #32219
• The FE performance plunges after the FE configuration item enable_collect_query_detail_info is set to true. #35945
• Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. #34352
• The starrocks_be process may exit unexpectedly when ./agentctl.sh stop be is used to stop a BE. #35108
• The array_distinct function occasionally causes the BEs to crash. #36377
• Deadlocks may occur when users refresh materialized views. #35736
• In some scenarios, dynamic partitioning may encounter an error, which causes FE start failures. #36846

3.1.6

Release date: December 18, 2023

New Features

• Added the now(p) function to return the current date and time with the specified fractional seconds precision (accurate to the microsecond). If p is not specified, this function returns only date and time accurate to the second. #36676
• Added a new metric max_tablet_rowset_num for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". #36539
  Supports obtaining heap profiles by using a command line tool, making troubleshooting easier.#35322
• Supports creating asynchronous materialized views with common table expressions (CTEs). #36142
• Added the following bitmap functions: subdivide_bitmap, bitmap_from_binary, and bitmap_to_binary. #35817 #35621

Parameter Changes

• The FE dynamic parameter enable_new_publish_mechanism is changed to a static parameter. You must restart the FE after you modify the parameter settings. #35338
• The default retention period of trash files is changed to 1 day from the original 3 days. #37113
• A new FE configuration item routine_load_unstable_threshold_second is added. #36222
• A new BE configuration item enable_stream_load_verbose_log is added. The default value is false. With this parameter set to true, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. #36113
• A new BE configuration item enable_lazy_delta_column_compaction is added. The default value is true, indicating that StarRocks does not perform frequent compaction operations on delta columns. #36654

Improvements

• A new value option GROUP_CONCAT_LEGACY is added to the session variable sql_mode to provide compatibility with the implementation logic of the group_concat function in versions earlier than v2.5. #36150
• The Primary Key table size returned by the SHOW DATA statement includes the sizes of .cols files (these are files related to partial column updates and generated columns) and persistent index files. #34898
• Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. #35917
• Plugin loading failures will no longer cause an error or cause an FE start failure. Instead, the FE can properly start, and the error status of the plug-in can be queried using SHOW PLUGINS. #36566
• Dynamic partitioning supports random distribution. #35513
• The result returned by the SHOW ROUTINE LOAD statement now includes the timestamps of consumption messages from each partition. #36222
• The result returned by the SHOW ROUTINE LOAD statement provides a new field OtherMsg, which shows information about the last failed task. #35806
• The authentication information aws.s3.access_key and aws.s3.access_secret for AWS S3 in Broker Load jobs are hidden in audit logs. #36571
• The be_tablets view in the information_schema database provides a new field INDEX_DISK, which records the disk usage (measured in bytes) of persistent indexes #35615

Bug Fixes

Fixed the following issues:

• The BEs crash if users create persistent indexes in the event of data corruption. #30841
• If users create an asynchronous materialized view that contains nested queries, the error "resolve partition column failed" is reported. #26078
• If users create an asynchronous materialized view on a base table whose data is corrupted, the error "Unexpected exception: null" is reported. #30038
• If users run a query that contains a window function, the SQL error "[1064] [42000]: Row count of const column reach limit: 4294967296" is reported. #33561
• The FE performance plunges after the FE configuration item enable_collect_query_detail_info is set to true. #35945
• In the StarRocks shared-data mode, the error "Reduce your request rate" may be reported when users attempt to delete files from object storage. #35566
• Deadlocks may occur when users refresh materialized views. #35736
• After the DISTINCT window operator pushdown feature is enabled, errors are reported if SELECT DISTINCT operations are performed on the complex expressions of the columns computed by window functions. #36357
• The BEs crash if the source data file is in ORC format and contains nested arrays. #36127
  Some S3-compatible object storage returns duplicate files, causing the BEs to crash. #36103