Skip to content

Add Dataline types to interface for check connection and discovery workers #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 20, 2020

Conversation

cgardens
Copy link
Contributor

@cgardens cgardens commented Aug 19, 2020

What

  • Enforce contract of what test connection and discover schema output.
  • Map from singer types to our types for test connection and discover schema.

@@ -0,0 +1,114 @@
{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yolo. i tried to create a jscon schema configuration for catalog.json to make all the type parsing easier. 🤞 that it's good enough. with the lack of consistency in singer, i'm a big worried, but for now it's helpful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based off the example on the singer webpage.


import io.dataline.config.StandardConnectionStatus;

public interface CheckConnectionWorker extends Worker<StandardConnectionStatus> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trying to make it very clear what the contract is for each job type. CheckConnection outputs StandardConnectionStatus. Discover outputs StandardDiscoveryOutput. Ideally we'd be able to get the input types (e.g. connection configuration). will look at that subsequently.

}
}

private static StandardDiscoveryOutput toDiscoveryOutput(SingerCatalog catalog) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

singer catalog to our schema data model.

@cgardens cgardens marked this pull request as ready for review August 19, 2020 23:27
@cgardens cgardens merged commit ce5db31 into master Aug 20, 2020
@cgardens cgardens deleted the cgardens/singer_type_conversion branch August 20, 2020 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants