Skip to content

[DynamoDB]: java.lang.OutOfMemoryError: Java heap space #56350

New issue
New issue
Open
Open
[DynamoDB]: java.lang.OutOfMemoryError: Java heap space#56350
Labels
area/connectorsConnector related issuescommunityconnectors/source/dynamodbteam/db-dw-sourcesBacklog for Database and Data Warehouse Sources teamtype/bugSomething isn't working
@moseleyi

Description

@moseleyi
moseleyi
opened on Mar 23, 2025

Connector Name

destination-bigquery

Connector Version

0.2.0

What step the error happened?

Configuring a new connector

Relevant information

I'm getting this error when I try to set up a connection between DynamoDB and BigQuery:

Internal message: java.lang.OutOfMemoryError: Java heap space
Failure type: system_error

Relevant log output

2025-03-23 13:02:33 info APPLY Stage: BUILD — (workloadId = 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover) — (dataplaneId = gcp-us-expanded-queue-1)
2025-03-23 13:02:33 info APPLY Stage: CLAIM — (workloadId = 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover) — (dataplaneId = gcp-us-expanded-queue-1)
2025-03-23 13:02:33 info Claimed: true for 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover via API for 47d56868-41ea-4676-8b36-6b0bcb6f5833
2025-03-23 13:02:33 info APPLY Stage: LOAD_SHED — (workloadId = 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover) — (dataplaneId = gcp-us-expanded-queue-1)
2025-03-23 13:02:33 info APPLY Stage: CHECK_STATUS — (workloadId = 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover) — (dataplaneId = gcp-us-expanded-queue-1)
2025-03-23 13:02:34 info No pod found running for workload 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover
2025-03-23 13:02:34 info APPLY Stage: MUTEX — (workloadId = 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover) — (dataplaneId = gcp-us-expanded-queue-1)
2025-03-23 13:02:34 info No mutex key specified for workload: 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover. Continuing...
2025-03-23 13:02:34 info APPLY Stage: LAUNCH — (workloadId = 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover) — (dataplaneId = gcp-us-expanded-queue-1)
2025-03-23 13:02:34 info [initContainer] image: airbyte/workload-init-container:build-58bc865f0d-21444-master resources: ResourceRequirements(claims=[], limits={memory=1000Mi, cpu=4}, requests={memory=1000Mi, cpu=4}, additionalProperties={})
2025-03-23 13:02:46 info Attempting to update workload: 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover to LAUNCHED.
2025-03-23 13:02:46 info Pipeline completed for workload: 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover.
2025-03-23 13:02:52 info 
2025-03-23 13:02:52 info ----- START DISCOVER -----
2025-03-23 13:02:52 info 
2025-03-23 13:03:46 info Connector exited, processing output
2025-03-23 13:03:46 info Output file jobOutput.json found
2025-03-23 13:03:46 info Connector exited with exit code 1
2025-03-23 13:03:46 info Reading messages from protocol version 0.2.0
2025-03-23 13:03:46 info INFO i.a.i.s.d.DynamodbSource(main):49 starting Source: class io.airbyte.integrations.source.dynamodb.DynamodbSource
2025-03-23 13:03:46 info INFO i.a.c.i.b.IntegrationCliParser(parseOptions):126 integration args: {discover=null, config=/config/connectionConfiguration.json}
2025-03-23 13:03:46 info INFO i.a.c.i.b.IntegrationRunner(runInternal):132 Running integration: io.airbyte.integrations.source.dynamodb.DynamodbSource
2025-03-23 13:03:46 info INFO i.a.c.i.b.IntegrationRunner(runInternal):133 Command: DISCOVER
2025-03-23 13:03:46 info INFO i.a.c.i.b.IntegrationRunner(runInternal):134 Integration config: IntegrationConfig{command=DISCOVER, configPath='/config/connectionConfiguration.json', catalogPath='null', statePath='null'}
2025-03-23 13:03:46 warn WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2025-03-23 13:03:46 warn WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2025-03-23 13:03:46 info INFO i.a.i.s.d.DynamodbUtils(createDynamoDbClient):40 Creating credentials using access key and secret key
2025-03-23 13:03:46 error ERROR i.a.c.i.b.AirbyteExceptionHandler(uncaughtException):64 Something went wrong in the connector. See the logs for more details. java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays$ArrayList.iterator(Arrays.java:4325) ~[?:?]
	at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1075) ~[?:?]
	at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1074) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:229) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:125) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312030.unmarshall(Unknown Source) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallMap$1(JsonProtocolUnmarshaller.java:152) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac03e0420.accept(Unknown Source) ~[?:?]
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:986) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallMap(JsonProtocolUnmarshaller.java:150) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312450.unmarshall(Unknown Source) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:240) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:125) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312030.unmarshall(Unknown Source) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallMap$1(JsonProtocolUnmarshaller.java:152) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac03e0420.accept(Unknown Source) ~[?:?]
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:986) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallMap(JsonProtocolUnmarshaller.java:150) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312450.unmarshall(Unknown Source) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:240) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:125) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312030.unmarshall(Unknown Source) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallMap$1(JsonProtocolUnmarshaller.java:152) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac03e0420.accept(Unknown Source) ~[?:?]
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:986) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallMap(JsonProtocolUnmarshaller.java:150) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312450.unmarshall(Unknown Source) ~[?:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallList$2(JsonProtocolUnmarshaller.java:167) ~[aws-json-protocol-2.18.1.jar:?]
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac03cdf00.apply(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]

Stack Trace: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays$ArrayList.iterator(Arrays.java:4325)
	at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1075)
	at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1074)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:229)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:125)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312030.unmarshall(Unknown Source)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallMap$1(JsonProtocolUnmarshaller.java:152)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac03e0420.accept(Unknown Source)
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:986)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallMap(JsonProtocolUnmarshaller.java:150)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312450.unmarshall(Unknown Source)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:240)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:125)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312030.unmarshall(Unknown Source)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallMap$1(JsonProtocolUnmarshaller.java:152)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac03e0420.accept(Unknown Source)
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:986)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallMap(JsonProtocolUnmarshaller.java:150)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312450.unmarshall(Unknown Source)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:240)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallStructured(JsonProtocolUnmarshaller.java:125)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312030.unmarshall(Unknown Source)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallMap$1(JsonProtocolUnmarshaller.java:152)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac03e0420.accept(Unknown Source)
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:986)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.unmarshallMap(JsonProtocolUnmarshaller.java:150)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac0312450.unmarshall(Unknown Source)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller.lambda$unmarshallList$2(JsonProtocolUnmarshaller.java:167)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonProtocolUnmarshaller$$Lambda/0x00007beac03cdf00.apply(Unknown Source)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)

2025-03-23 13:03:46 info Checking for optional control message...
2025-03-23 13:03:46 info Writing output of 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover to the doc store
2025-03-23 13:03:46 info Marking workload 50401137-8871-4c5a-abb7-1f5fda35545a_a455f0a5-8d57-4437-9992-7c4926c813ef_0_discover as successful
2025-03-23 13:03:46 info 
2025-03-23 13:03:46 info Deliberately exiting process with code 0.
2025-03-23 13:03:46 info ----- END DISCOVER -----
2025-03-23 13:03:46 info

Contribute

  • Yes, I want to contribute

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/connectorsConnector related issuescommunityconnectors/source/dynamodbteam/db-dw-sourcesBacklog for Database and Data Warehouse Sources teamtype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions