🎉 New source: MongoDb ported to java #5530

irynakruk · 2021-08-19T14:05:05Z

What

Port MongoDB Source to Java

How

Implemented with mongodb native java driver.
Added abstraction AbstractDbSource upon AbstractRelationalDbSource, so now all non jdbc and non sql sources can depend on it.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

etsybaev

Please also make sure you have added all docs as per the checklist.
Connector's README.md
docs/SUMMARY.md
docs/integrations//.md including changelog. See changelog example
docs/integrations/README.md
airbyte-integrations/builds.md

airbyte-integrations/connectors/source-mongodb-new/CHANGELOG.md

airbyte-integrations/connectors/source-mongodb-new/README.md

irynakruk · 2021-09-01T11:37:48Z

/test connector=connectors/source-mongodb-new

🕑 connectors/source-mongodb-new https://github.com/airbytehq/airbyte/actions/runs/1189997410
✅ connectors/source-mongodb-new https://github.com/airbytehq/airbyte/actions/runs/1189997410

airbyte-db/lib/src/main/java/io/airbyte/db/SqlDatabase.java

DoNotPanicUA · 2021-09-02T12:39:51Z

airbyte-integrations/connectors/source-mongodb-new/build.gradle

+    implementation project(':airbyte-protocol:models')
+    implementation files(project(':airbyte-integrations:bases:base-java').airbyteDocker.outputs)
+    implementation project(':airbyte-integrations:connectors:source-relational-db')
+    implementation group: 'org.mongodb', name: 'mongo-java-driver', version: '3.12.8'


Do we really need the driver here? Is there something outside airbyte-db and MongodbDatabase?

Yes, there is MongoCollection, BsonType and Document classes from mongo

tuliren

Looks great!

One minor issue about the name. source-mongodb-new seems informal. Can it be renamed to something like source-mongodb-v2?

@sherifnada, FYI.

...o/airbyte/integrations/io/airbyte/integration_tests/sources/MongoDbSourceAcceptanceTest.java

sherifnada · 2021-09-03T01:31:37Z

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

@@ -458,3 +458,9 @@
  dockerRepository: airbyte/source-amazon-ads
  dockerImageTag: 0.1.0
  documentationUrl: https://docs.airbyte.io/integrations/sources/amazon-ads
+- sourceDefinitionId: b2e713cd-cc36-4c0a-b5bd-b47cb8a0561e


lets remove the old mongo db implementation, having two implementations is confusing to the user

I removed old implementation from source definitions.

sherifnada · 2021-09-03T01:32:17Z

airbyte-integrations/builds.md

@@ -49,6 +49,7 @@
 | Microsoft Teams   | [![source-microsoft-teams](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-microsoft-teams%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-microsoft-teams) |
 | Mixpanel   | [![source-mixpanel](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-mixpanel%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-mixpanel) |
 | Mongo DB   | [![source-mongodb](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-mongodb%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-mongodb) |
+| Mongo DB new  | [![source-mongodb-new](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-mongodb-new%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-mongodb-new) |


Externally, this should just be called Mongo DB and should replace the existing mongo implementation.

Removed old | Mongo DB | and renamed a new one to | Mongo DB | accordingly

sherifnada · 2021-09-03T01:33:13Z

airbyte-integrations/connectors/source-mongodb-new/src/main/resources/spec.json

+        "examples": ["admin"],
+        "order": 5
+      },
+      "replica_set": {


can this connect to both a replica set and a single node mongo db?

Can you test it using the mongodb atlas credentials in lastpass?

Sure, working on it

@sherifnada I've updated spec.json and connection param to connect to Mongodb Atlas as it uses mongodb+srv protocol.
So now it is possible to connect to Standalone MongoDb instance, Replica Set and MongoDb Atlas Cluster as well.
I've tested it using the mongodb atlas credentials, works good.

sherifnada · 2021-09-03T01:34:11Z

docs/integrations/README.md

@@ -65,6 +65,7 @@ Airbyte uses a grading system for connectors to help users understand what to ex
 |[Microsoft Teams](./sources/microsoft-teams.md)| Certified |
 |[Mixpanel](./sources/mixpanel.md)| Beta |
 |[Mongo DB](./sources/mongodb.md)| Alpha |
+|[Mongo DB new](./sources/mongodb-new.md)| Alpha |


there should be only one mongo db, also it should be in beta

… connector info in all docs

…nto irynakruk/3428-mongodb-source

irynakruk · 2021-09-08T14:44:58Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1213783567
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1213783567

sherifnada

nice!

irynakruk · 2021-09-09T00:23:26Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/1215407854
✅ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/1215407854

irynakruk · 2021-09-09T01:14:52Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1215523637
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1215523637

irynakruk · 2021-09-09T01:49:00Z

/publish connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1215591690
✅ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1215591690

Iryna Kruk added 4 commits July 15, 2021 14:18

New abstraction for NoSql database sources

091c10c

New MongoDbSource: partial impl

02a928b

merged master

066a123

Added MongoDataType

c6ca34f

github-actions bot added the area/connectors Connector related issues label Aug 19, 2021

irynakruk changed the title ~~Irynakruk/3428 mongodb source~~ New source: MongoDb ported to java Aug 31, 2021

irynakruk changed the title ~~New source: MongoDb ported to java~~ 🎉 New source: MongoDb ported to java Aug 31, 2021

Improved MongoDatabase and fixed read method

5ff230e

irynakruk requested a review from etsybaev August 31, 2021 11:42

irynakruk linked an issue Aug 31, 2021 that may be closed by this pull request

Port MongoDB Source to Java #3428

Closed

etsybaev approved these changes Aug 31, 2021

View reviewed changes

airbyte-integrations/connectors/source-mongodb-new/CHANGELOG.md Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-mongodb-new/README.md Outdated Show resolved Hide resolved

irynakruk added 2 commits August 31, 2021 12:14

merged master;

f9eb86f

code review changes;

2db879d

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 1, 2021

irynakruk marked this pull request as ready for review September 1, 2021 11:15

merge clean up;

2f5fbdc

jrhizor temporarily deployed to more-secrets September 1, 2021 11:40 Inactive

DoNotPanicUA self-requested a review September 2, 2021 12:25

irynakruk requested a review from tuliren September 2, 2021 12:41

DoNotPanicUA reviewed Sep 2, 2021

View reviewed changes

DoNotPanicUA approved these changes Sep 2, 2021

View reviewed changes

Renamed NoSqlDatabase to AbstractDatabase

7437f48

tuliren approved these changes Sep 2, 2021

View reviewed changes

...o/airbyte/integrations/io/airbyte/integration_tests/sources/MongoDbSourceAcceptanceTest.java Outdated Show resolved Hide resolved

sherifnada reviewed Sep 3, 2021

View reviewed changes

irynakruk added 5 commits September 3, 2021 11:09

merged master;

703bc7d

merged master;

2a8eb53

formatter changes;

ad7f500

code review changes: changed mongodb-new to mongodb-v2; left only new…

d58350a

… connector info in all docs

code review changes: changed mongodb-new to mongodb-v2; left only new…

eb9465a

… connector info in all docs

Merge remote-tracking branch 'origin/irynakruk/3428-mongodb-source' i…

bbac807

…nto irynakruk/3428-mongodb-source

github-actions bot added the area/platform issues related to the platform label Sep 3, 2021

irynakruk added 2 commits September 6, 2021 16:17

updated spec.json and toDatabaseConfig() method

e1fdc7f

updated doc accordingly to spec.json changes

500b1a9

irynakruk requested a review from sherifnada September 8, 2021 01:02

jrhizor temporarily deployed to more-secrets September 8, 2021 14:47 Inactive

sherifnada approved these changes Sep 8, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets September 9, 2021 00:26 Inactive

jrhizor temporarily deployed to more-secrets September 9, 2021 00:57 Inactive

jrhizor temporarily deployed to more-secrets September 9, 2021 01:17 Inactive

jrhizor temporarily deployed to more-secrets September 9, 2021 01:51 Inactive

irynakruk merged commit ffecc1c into master Sep 9, 2021

irynakruk deleted the irynakruk/3428-mongodb-source branch September 9, 2021 12:14

jrhizor mentioned this pull request Sep 16, 2021

Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha #6125

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New source: MongoDb ported to java #5530

🎉 New source: MongoDb ported to java #5530

irynakruk commented Aug 19, 2021

etsybaev left a comment

irynakruk commented Sep 1, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA Sep 2, 2021

irynakruk Sep 2, 2021

tuliren left a comment

sherifnada Sep 3, 2021

irynakruk Sep 3, 2021

sherifnada Sep 3, 2021

irynakruk Sep 3, 2021

sherifnada Sep 3, 2021

irynakruk Sep 3, 2021

irynakruk Sep 6, 2021

sherifnada Sep 3, 2021

irynakruk Sep 3, 2021

irynakruk commented Sep 8, 2021 •

edited by github-actions bot

Loading

sherifnada left a comment

irynakruk commented Sep 9, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 9, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 9, 2021 •

edited by github-actions bot

Loading

🎉 New source: MongoDb ported to java #5530

🎉 New source: MongoDb ported to java #5530

Conversation

irynakruk commented Aug 19, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

etsybaev left a comment

Choose a reason for hiding this comment

irynakruk commented Sep 1, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tuliren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

irynakruk commented Sep 8, 2021 • edited by github-actions bot Loading

sherifnada left a comment

Choose a reason for hiding this comment

irynakruk commented Sep 9, 2021 • edited by github-actions bot Loading

irynakruk commented Sep 9, 2021 • edited by github-actions bot Loading

irynakruk commented Sep 9, 2021 • edited by github-actions bot Loading

irynakruk commented Sep 1, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 8, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 9, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 9, 2021 •

edited by github-actions bot

Loading

irynakruk commented Sep 9, 2021 •

edited by github-actions bot

Loading