🎉 Snowflake destination: reduce memory footprint

[tuliren]

• Estimate record message size by string length, instead of byte length. This significantly reduce the amount of memory usage.
• To further reduce redundant string serialization, the size estimation only sample the string size very N records, instead of doing it for every record.
• See rationales in [EPIC] scale warehouse destination connectors to handle arbitrary number of streams #10260.

[subodh1810]

Am not sure I understand how this will reduce heap consumption. The only difference I see is that with previous logic we were calculating the size for each record using s.getBytes(StandardCharsets.UTF_8).length; but with new logic we do it via Jsons.serialize(data).length() * 4L but for every 20th record. How does this help with lesser heap consumption?

[tuliren]

@subodh1810, calculating the bytes of the serialized json strings creates lots of byte array objects. So switching away from generating the byte array is the fix. Before this change, the connector will run out of memory when the max heap size is 500 MB. After this change, it works even with just 300 MB heap size.

See here for raw data.

[tuliren]

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1860490882
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1860490882

[tuliren]

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1860626168
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1860626168