🐛 During sync from source to S3 destination int, long, float, double data types are mapped to double type in Avro and Parquet files.

[sashaNeshcheret]

The issue was found during creating MLP documentation for s3 destination. It is complex issue that consists of source and destination parts.
On source side we have several primitives json schema types, one of them is Number. So all numeric source data types (int, long, float, etc) are mapped to number json schema type.
On destination side simple number json schema types is mapped to double type.
For handling it on source side we need use complex json schema types, that consist for instance type(Number) and sub type(int).
On destination side - based on complex json schema types, we could choose relevant data type.

Environment

• Airbyte version: 0.35.55-alpha
• Source Connector and version: any source
• Destination Connector and version: destination-s3:3:0.2.13
• Severity: Medium
• Step where error happened: Sync job

Current Behavior

During sync from source to S3 destination int, long, float, double data types are mapped to double type in Avro and Parquet files.

Expected Behavior

During sync from source to S3 destination int, long, float, double data types are mapped to proper datatypes (int -> int, etc.)

[grishick]

@sashaNeshcheret which sources did you use when you ran into this problem? Specifically, which sources did you notice were able to produce long, float, and double types?

[sashaNeshcheret]

@grishick i used java types such as long, float, and double data types to generalize issue. Seems the behavior is relevant for all sources connectors, i checked for postgres(int, bigint, float, double) and mysql(int, bigint, double) connectors.

[sashaNeshcheret]

I created scoping doc for current issue. @VitaliiMaltsev, @alexandr-shegeda @grishick please take a look at it.

[grishick]

@edgao @tuliren would the same approach that we discussed here work for this issue?
Specifically, I suggest the following:

1. Step 1 we make int, long, float, and double encoded as JSON schema type number and add airbyte_type integer/float/double
2. Step 2: gradually update destinations to prioritize airbyte_type over JSON Schema type.
3. Step 3: after all destinations are updated to handle airbyte_type correctly for numeric values, change sources to encode numeric values as JSON schema type string

[edgao]

yep, sounds right to me. (nitpick: I think step 1 should just be long+double, rather than int/long/float/double)