use postgres / jooq (without code generation) / test containers #57

jrhizor · 2020-08-14T15:35:46Z

switches from SQLite to Postgres
allows dependency injection of db connection pool instead of using a singleton connection pool
provides a way to use the same version of postgres for testing

You should look at ServerUuid for an example of how to use JOOQ for querying. I didn't do codegen so it isn't using the ORM functionality, but Result<Record> provides a much nicer interface than something like ResultSet.

dataline-db/src/main/resources/schema.sql

michel-tricot · 2020-08-14T16:53:38Z

.gitignore

@@ -2,3 +2,4 @@
 .idea
 build
 .DS_Store
+dataline-db/pg_data/*


what generates that directory?

It's what I'm using as the default mount for the dev docker db. I'm just setting it to an explicit location so it's easy to delete if we want to iterate on db schemas or start from scratch.

michel-tricot · 2020-08-14T16:54:50Z

.env

@@ -1,2 +1,7 @@
 VERSION=0.1.0
 ENV=docker
+POSTGRES_VERSION=13-alpine


we should hardcode this one in compose. We should keep the env for the env varialbes that are read by the code

This should be read by test containers to pull the correct version of the image, so it's consistent with docker compose. I'll add that.

Actually it's hard to set variables used for test in intellij, github, and gradle versions of tests. Maybe that should just be hardcoded in both places.

.env

michel-tricot · 2020-08-14T17:00:03Z

dataline-db/src/main/java/io/dataline/db/DatabaseHelper.java

-    return connectionPool;
+  public static BasicDataSource getConnectionPoolFromEnv() {
+    return getConnectionPool(
+        System.getenv("POSTGRES_USER"),


Can we create a class in commons that serves as a repository for these env variable and provides accessors?

I'll do this separately since I already have some ENV helpers in the scheduling PR.

michel-tricot · 2020-08-14T17:01:06Z

dataline-db/src/main/resources/schema.sql

-CREATE TABLE DATALINE_METADATA (id varchar(255), value varchar(255), PRIMARY KEY (id));
-INSERT INTO DATALINE_METADATA VALUES  ('server-uuid', ((lower(hex(randomblob(4))) || '-' || lower(hex(randomblob(2))) || '-4' || substr(lower(hex(randomblob(2))),2) || '-' || substr('89ab',abs(random()) % 4 + 1, 1) || substr(lower(hex(randomblob(2))),2) || '-' || lower(hex(randomblob(6))))));
-CREATE TABLE JOBS (id INTEGER PRIMARY KEY AUTOINCREMENT, scope varchar(255), created_at INTEGER, started_at INTEGER, updated_at INTEGER, status INTEGER, config BLOB, output, BLOB, stdout_path varchar(255), stderr_path varchar(255));
+-- extensions


michel-tricot · 2020-08-14T17:02:52Z

dataline-db/src/test/java/io/dataline/db/TestServerUuid.java

+  private static final JdbcDatabaseContainer POSTGRES_CONTAINER;
+  private static final BasicDataSource CONNECTION_POOL;
+
+  static {


NIT: can you use @BeforeAll instead?

michel-tricot · 2020-08-14T17:04:12Z

dataline-db/src/test/java/io/dataline/db/TestServerUuid.java

+
+  static {
+    // todo: load image from env and don't hardcode absolute path
+    POSTGRES_CONTAINER = new PostgreSQLContainer("postgres:13-alpine").withInitScript("schema.sql");


really nice!

Are there good practices to ensure that tests run in isolation? It seems that since you instantiate the db once, that content in tables will be shared accross the tests.

What we did in the past was create a single db container and then run tests one at a time, clearing the db in between runs. I'll make a better db test class after merging this PR.

michel-tricot · 2020-08-14T17:07:27Z

docker-compose.dev.yaml

+      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+      - POSTGRES_DB=${POSTGRES_DB}
+      - POSTGRES_CONNECT_STR=${POSTGRES_CONNECT_STR}
+      - WAIT_HOSTS=db:5432


I assume the depends_on isn't enough? Maybe we can make it work with some sort of a tcp healthcheck defined in the db.

This is basically just a tcp healtcheck. Searching around, it seems like doing something like using one of the wait/wait-for/wait-for-it scripts is the "accepted" method of doing the check. You can do things like setting the value of WAIT_HOSTS to db:5432,db2:5433 if you have multiple TCP dependencies, for example.

And to answer the other question, yes, depends_on isn't enough. It only waits for the container to start, but the PG server doesn't run until the init script finishes.

jrhizor added 12 commits August 13, 2020 14:22

add database image to docker compose

81b50bf

add database setup

ab3ebf9

allow connetions from a connect string in env variables

a510b1b

add a test for the docker case

ca18852

add pg init script in docker-compose

384dad0

fix env and add server dependency on db

9396e96

convert schema to postgres

bd0e73a

use postgres driver

c10a19d

fix build

190a353

fix edge cases (dependencies between containers, data dirs, etc)

f3e542a

use docker wait

d1adf29

remove debugging logging

e2ec601

jrhizor requested a review from cgardens August 14, 2020 16:28

add grant

65d703c

cgardens approved these changes Aug 14, 2020

View reviewed changes

dataline-db/src/main/resources/schema.sql Outdated Show resolved Hide resolved

michel-tricot reviewed Aug 14, 2020

View reviewed changes

working test and working docker-compose

dd671d3

jrhizor marked this pull request as ready for review August 14, 2020 17:42

jrhizor added 4 commits August 14, 2020 10:48

use beforeall

fa8114a

use key instead of id for k-v metadata

445d9c2

clean up env names

6f65a6e

fix typo

7ec0f7a

michel-tricot approved these changes Aug 14, 2020

View reviewed changes

jrhizor merged commit ef0db38 into master Aug 14, 2020

jrhizor deleted the jrhizor/postgres branch August 14, 2020 18:02

hugoJuhel mentioned this pull request Mar 16, 2022

🎉 Destination mssql : adding ssh tunnel for normalisation #11204

Merged

21 tasks

ptiurin mentioned this pull request Sep 15, 2022

🎉 Base Normalization: Firebolt support #16774

Closed

14 tasks

rskrishnar mentioned this pull request Mar 17, 2025

fix: remove workflows using tj-actions/changed-files action #55792

Closed

use postgres / jooq (without code generation) / test containers #57

use postgres / jooq (without code generation) / test containers #57

Uh oh!

Conversation

jrhizor commented Aug 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jrhizor commented Aug 14, 2020 •

edited

Loading