Skip to content

Attempt running singer taps from a docker container #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from

Conversation

sherifnada
Copy link
Contributor

@sherifnada sherifnada commented Aug 12, 2020

Problem

Singer binaries (and their environments) used in java tests should be the exact same as what we ship in our docker container. While it's easy to isolate Singer binaries (they're stored in a single folder), it's not easy to isolate their dependencies. For example, tap-postgres requires pg_config and gcc to be installed on the host machine/container. By default, alpine-linux does not ship with these dependencies, so we'd need a custom instruction in the Dockerfile to install them. But without building the container then running tests inside it (or encountering a bug while running it), we don't know if the local env in which tests succeeded accurately mirrors the docker container we produce.

Potential solutions

  1. Not an MVP problem. Let's manually rig up our local envs to match the docker image as closely as possible, run into these bugs and add them into our Dockerfile until we figure out a better way.
  2. Create a docker container containing all singer libs and their dependencies, and make our intellij/build tests run inside that Docker container.

This draft PR is me exploring solution no.2. Resolution of this problem is currently blocking writing any real tests for singer workers, so I'll need to figure this out to be able to meaningfully complete workers.

EDIT: I think i will go for solution 1 since I realized the docker container runs tests as part of its build process, so we'll catch any issues I'm worried about here. Ignore the docker file for now, will generate singer binaries as part of the build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant