Source Oracle - Memory Leak

[philippeboyd]

Connector Name

source-oracle

Connector Version

0.4.0

What step the error happened?

During the sync

Revelant information

There seems to be a big memory leak with the source-oracle connector. Vertical scaling doesn't help.

I sync about ~1.5 GB of data from difference Oracle sources and the source-oracle-read container can take from 9GB up to 18Gb of RAM sometime.

There's no way that syncing 1.5GB of data takes between 10 and 18GB of RAM in a java process.

Might be linked to the other memory leak issue I opened #27844

[marcosmarxm]

@prateekmukhedkar this is something of knowledge of the connectors team?

[philippeboyd]

@marcosmarxm @prateekmukhedkar any news on this? We're seeing ram explode for syncing 1GB worth of data

[prateekmukhedkar]

The amount of memory used will depend on the schema of the tables you have selected for replication. We have also realized that reading all the data from a table in one single read transaction is not the most efficient use of resources, and instead we will be reading data in smaller chunks. We have not scoped out these improvements for source-oracle at present.

At this point I'll suggest creating a view, limiting data from last X days and/or including certain columns from your source table. In the replication tab, configure a sync for the view you created. Please keep us updated on how things proceed.