[destination-databricks] sync more than 100+ tables and 10Tb+ data, meet OutOfMemoryError

[amelia-ay]

Connector Name

destination-databricks

Connector Version

3.2.5

What step the error happened?

During the sync

Relevant information

When we use helm-airbyte to synchronize data from MySQL binlog to Databricks, an error of java.lang.OutOfMemoryError occurred. The.env environment has been configured with JAVA_OPTS = "-server -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=75.0 -XX:NativeMemoryTracking=detail -XX:+UseZGC -XX:+ZGenerational -XX:+UseStringDeduplication", and HPA: true has also been enabled in values.yaml. However, it is noted that MAX_SYNC_WORKERS=10 has not taken effect. After referring to https://docs.airbyte.com/operator-guides/configuring-connector-resources and configuring resource_requirements, the obtained log parameter is ***(paste below). It seems that the classified resources have not been adjusted. Is there any way to optimize this part?

Relevant log output

Attempting to start pod = destination-databricks-local-write-29044-0-edyas for * with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@efe65bd[cpuRequest=2.0,cpuLimit=8.0,memoryRequest=4Gi,memoryLimit=16Gi,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@2baa6518[cpuRequest=0.05,cpuLimit=0.2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@3670f0c2[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@59e1e073[cpuRequest=0.5,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@4f838e99[cpuRequest=0.01,cpuLimit=0.5,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null

Contribute

• Yes, I want to contribute

### Tasks