Timezone Conversion Issue with INT96 Timestamps During Parquet Stream Load in Doris

[sjahongir]

Issue Description:

I'm encountering an issue when using Doris' Stream Load feature to load Parquet files that contain INT96-formatted timestamps. The timestamps appear to be automatically converted to UTC+8 during the load process, which is not the expected behavior. These timestamps should be absolute date, but Doris seems to be applying an incorrect timezone offset, leading to discrepancies in the data.

Steps to Reproduce:
Create a Parquet file with INT96 timestamps using a tool like Parquet.Net.
Use Doris Stream Load feature to load the Parquet file.
Observe that the timestamps are converted to UTC+8 during the load process.
Expected Behavior:
The INT96 timestamps should be absolute date, or Doris should allow configuration of the timezone for timestamp fields during the load. Alternatively, the timestamps should be loaded without any automatic timezone conversion.

Actual Behavior:
The timestamps in the INT96 format are incorrectly converted to UTC+8 during the load process when using the Stream Load feature. This results in inaccurate data in the loaded Parquet files.

Metadata of the Parquet File:
The metadata for the Parquet file is as follows:

{
  "Version": 1,
  "NumRows": 3,
  "NumRowGroups": 1,
  "NumFields": 2,
  "CreatedBy": "Parquet.Net version 5.0.2 (build b7322effa264da943718efdbca76d182b450b949)",
  "Schema": {
    "Path": "root",
    "Children": [
      {
        "Path": "vchreferencenumber",
        "Type": "BYTE_ARRAY",
        "LogicalType": {
          "Name": "STRING"
        },
        "RepetitionType": "OPTIONAL",
        "ConvertedType": "UTF8",
        "Children": []
      },
      {
        "Path": "dtcreatedate",
        "Type": "INT96",
        "LogicalType": null,
        "RepetitionType": "OPTIONAL",
        "ConvertedType": "",
        "Children": []
      }
    ]
  },
  "RowGroups": [
    {
      "NumRows": 3,
      "TotalByteSize": 221
    }
  ]
}

[zclllyybb]

We need more information to reproduce this problem:

1. the value of session variable time_zone
2. the parameters used in stream load
3. some of the import data to let us know the data format.

[horus-leonardo]

Am having the same problem.

A Parquet date that is set as midnight appears as 08 in the morning.

Session timezone:

Variable_name   |Value            |Default_Value|Changed|
----------------+-----------------+-------------+-------+
time_zone       |America/Sao_Paulo|Etc/UTC      |1      |
system_time_zone|Etc/UTC          |Etc/UTC      |0      |

Parquet inspection:

➜  ~ parquet-tools show bug.parquet
+-------------+-------------+------------+
|       DATA1 |       DATA2 |      DATA3 |
|-------------+-------------+------------|
| 1.73575e+18 | 1.73569e+18 | 1.7357e+18 |
+-------------+-------------+------------+
➜  ~ parquet-tools inspect bug.parquet

############ file meta data ############
created_by: Parquet.Net version 4.24.0 (build 4875cc5828cf1613aece87f3c4ca5c035b39dd30)
num_columns: 3
num_rows: 1
num_row_groups: 1
format_version: 1.0
serialized_size: 419


############ Columns ############
DATA1
DATA2
DATA3

############ Column(DATA1) ############
name: DATA1
path: DATA1
max_definition_level: 1
max_repetition_level: 0
physical_type: INT96
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: -2%)

############ Column(DATA2) ############
name: DATA2
path: DATA2
max_definition_level: 1
max_repetition_level: 0
physical_type: INT96
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: -2%)

############ Column(DATA3) ############
name: DATA3
path: DATA3
max_definition_level: 1
max_repetition_level: 0
physical_type: INT96
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: -2%)

➜  ~ parquet-tools csv bug.parquet
DATA1,DATA2,DATA3
2025-01-01 17:18:19,2025-01-01,2025-01-01 02:02:02

Load:

➜  ~ curl -X PUT \                
  -H "Expect: 100-continue" \
  -H "label: loadtest_$(date +%s)" \
  -H "format: parquet" \
  -H "Content-Type: application/octet-stream" \
  -H "Authorization: Basic ..." \
  --data-binary @./bug.parquet \
  -v \
  --location-trusted \
  "http://ip:8030/api/schema/table/_stream_load"

Select from table:

DATA1              |DATA2              |DATA3     |
-------------------+-------------------+----------+
2025-01-02 01:18:19|2025-01-01 08:00:00|2025-01-01|

Table definition:

CREATE TABLE `test` (
  `DATA1` datetime NOT NULL,
  `DATA2` datetime NULL,
  `DATA3` date NULL
) ENGINE=OLAP
DUPLICATE KEY(`DATA1`)
AUTO PARTITION BY RANGE (date_trunc(`DATA1`, 'month'))
(PARTITION p20250101000000 VALUES [('2025-01-01 00:00:00'), ('2025-02-01 00:00:00')))
DISTRIBUTED BY RANDOM BUCKETS 8
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"min_load_replica_num" = "-1",
"is_being_synced" = "false",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V1",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false",
"group_commit_interval_ms" = "10000",
"group_commit_data_bytes" = "134217728"
);

[horus-leonardo]

Passing "timezone=UTC" in the header fixed the issue.