Add checkpointing to JDBC Source databases

[alafanechere]

Summary

Created from Slack: https://airbytehq.slack.com/archives/C01MFR03D5W/p1640005429033600?thread_ts=1640005429.033600&cid=C01MFR03D5W

when starting full sync on a huge table i can see that there is a long query in the DB(something like this):
SELECT <all columns> FROM <table>

But the table is 4TB, this way it will take days to finish the first query (i will probably fail before).. any way to configure the first sync to be divided into smaller chanks?

Connector publication

Connector	Old	New	PR
source-postgres	0.4.40	0.4.41	#14903
source-postgres-strict-encrypt	^	^	^
source-mssql	0.4.16	0.4.17	#16261
source-mssql-strict-encrypt	^	^	^
source-mysql	0.6.7	0.6.8	#16259
source-mysql-strict-encrypt	^	^	^
source-oracle	0.3.20	0.3.21	#16238
source-oracle-strict-encrypt	^	^	^
source-redshift	0.3.13	0.3.14	#16258
source-snowflake	0.1.19	0.1.20	^
source-clickhouse	0.1.12	0.1.13	#16256
source-clickhouse-strict-encrypt	0.1.9	^	^
source-cockroachdb	0.1.16	0.1.17	^
source-cockroachdb-strict-encrypt	^	^	^
source-db2	0.1.14	0.1.15	^
source-db2-strict-encrypt	^	^	^
source-tidb	0.2.0	0.2.1	^

^ means it is the same as above.

[leoalmeidasant]

Hi @alafanechere, do you find any solution to this issue? I've been passing through this same problem for about three weeks and I don't know how to solve.

[grishick]

The issue will be addressed when a source and a destination in a given connection are making periodic checkpoints.

[grishick]

@tuliren looks like you have already done this for Postgres and now it should be possible to do this for MySQL and other JDBC database sources. If the implementation will require source-specific changes, then please file separate issues for each source.

[grishick]

Work left:

• create a PR and link to this issue
• test all JDBC sources
• publish all JDBC sources before publishing SQL Sources should include previous sync's cursor value in SELECT statement as not to skip rows added mid-sync #14732

[evantahler]

Is this issue a duplicate of #14770? if so, feel free to close one of them!

[tuliren]

@evantahler, no, they are not duplications. #14770 is for destination. This one is for source.