Reduce High snowflake loading costs for many destinations per single snowflake DW.

[viggin543]

Problem

High snowflake loading costs
for a scenario of multiple destinations to the same cluster.

Jitsu constantly executes metadata queries. for example:
desc table PUBLIC.XXXX_EVENTS
SELECT count(*) from INFORMATION_SCHEMA.COLUMNS where TABLE_SCHEMA = ? and TABLE_NAME = ?

since they are executed very often, snowflake charges for additional "cloud services" fees. Which gets to ~ 30% of the monthly bill!

Solution

cache table existence in redis in order to avoid running queries on snowflake INFORMATION_SCHEMA

---

its gets to more than 400$ / month on our end, with the smallest snowflake cluster and jitsu that loads data every 5 hours !!!

[absorbb]

• Removed redundant query in table existence check step to reduce usage costs
• Added snowflake.always_cache_schema: true option to enable table schema caching in batch mode. That may reduce usage costs a bit more