You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-[Developing a connector against a pinned CDK version](#developing-a-connector-against-a-pinned-cdk-version)
19
+
-[Changelog](#changelog)
20
+
-[Java CDK](#java-cdk)
21
21
22
22
## Intro to the Java CDK
23
23
@@ -31,15 +31,23 @@ The java CDK is comprised of separate modules, among which:
31
31
32
32
Each CDK submodule may contain these elements:
33
33
34
-
-`src/main` - (Required.) The classes that will ship with the connector, providing capabilities to the connectors.
35
-
-`src/test` - (Required.) These are unit tests that run as part of every build of the CDK. They help ensure that CDK `main` code is in a healthy state.
36
-
-`src/testFixtures` - (Optional.) These shared classes are exported for connectors for use in the connectors' own test implementations. Connectors will have access to these classes within their unit and integration tests, but the classes will not be shipped with connectors when they are published.
34
+
-`src/main` - (Required.) The classes that will ship with the connector, providing capabilities to
35
+
the connectors.
36
+
-`src/test` - (Required.) These are unit tests that run as part of every build of the CDK. They
37
+
help ensure that CDK `main` code is in a healthy state.
38
+
-`src/testFixtures` - (Optional.) These shared classes are exported for connectors for use in the
39
+
connectors' own test implementations. Connectors will have access to these classes within their
40
+
unit and integration tests, but the classes will not be shipped with connectors when they are
41
+
published.
37
42
38
43
### How is the CDK published?
39
44
40
-
The CDK is published as a set of jar files sharing a version number. Every submodule generates one runtime jar for the main classes. If the submodule contains test fixtures, a second jar will be published with the test fixtures classes.
45
+
The CDK is published as a set of jar files sharing a version number. Every submodule generates one
46
+
runtime jar for the main classes. If the submodule contains test fixtures, a second jar will be
47
+
published with the test fixtures classes.
41
48
42
-
Note: Connectors do not have to manage which jars they should depend on, as this is handled automatically by the `airbyte-java-connector` plugin. See example below.
49
+
Note: Connectors do not have to manage which jars they should depend on, as this is handled
50
+
automatically by the `airbyte-java-connector` plugin. See example below.
43
51
44
52
## Using the Java CDK
45
53
@@ -55,17 +63,20 @@ To build and test the Java CDK, execute the following:
55
63
56
64
You will need to bump this version manually whenever you are making changes to code inside the CDK.
57
65
58
-
While under development, the next version number for the CDK is tracked in the file: `airbyte-cdk/java/airbyte-cdk/core/src/main/resources/version.properties`.
66
+
While under development, the next version number for the CDK is tracked in the file:
If the CDK is not being modified, this file will contain the most recently published version number.
61
70
62
71
### Publishing the CDK
63
72
64
73
_⚠️ These steps should only be performed after all testing and approvals are in place on the PR. ⚠️_
65
74
66
-
The CDK can be published with a GitHub Workflow and a slash command which can be run by Airbyte personnel.
75
+
The CDK can be published with a GitHub Workflow and a slash command which can be run by Airbyte
76
+
personnel.
67
77
68
-
To invoke via slash command (recommended), use the following syntax in a comment on the PR that contains your changes:
78
+
To invoke via slash command (recommended), use the following syntax in a comment on the PR that
79
+
contains your changes:
69
80
70
81
```bash
71
82
/publish-java-cdk # Run with the defaults (dry-run=false, force=false)
@@ -77,12 +88,18 @@ Note:
77
88
78
89
- Remember to **document your changes** in the Changelog section below.
79
90
- After you publish the CDK, remember to toggle `useLocalCdk` back to `false` in all connectors.
80
-
- Unless you specify `force=true`, the pipeline should fail if the version you are trying to publish already exists.
81
-
- By running the publish with `dry-run=true`, you can confirm the process is working as expected, without actually publishing the changes.
82
-
- In dry-run mode, you can also view and download the jars that are generated. To do so, navigate to the job status in GitHub Actions and navigate to the 'artifacts' section.
83
-
- You can also invoke manually in the GitHub Web UI. To do so: go to `Actions` tab, select the `Publish Java CDK` workflow, and click `Run workflow`.
84
-
- You can view and administer published CDK versions here: https://admin.cloudrepo.io/repository/airbyte-public-jars/io/airbyte/cdk
85
-
- The public endpoint for published CDK versions is here: https://airbyte.mycloudrepo.io/public/repositories/airbyte-public-jars/io/airbyte/cdk/
91
+
- Unless you specify `force=true`, the pipeline should fail if the version you are trying to publish
92
+
already exists.
93
+
- By running the publish with `dry-run=true`, you can confirm the process is working as expected,
94
+
without actually publishing the changes.
95
+
- In dry-run mode, you can also view and download the jars that are generated. To do so, navigate to
96
+
the job status in GitHub Actions and navigate to the 'artifacts' section.
97
+
- You can also invoke manually in the GitHub Web UI. To do so: go to `Actions` tab, select the
98
+
`Publish Java CDK` workflow, and click `Run workflow`.
99
+
- You can view and administer published CDK versions here:
Replace `0.1.0` with the CDK version you are working with. If you're actively developing the CDK and want to use the latest version locally, use the `useLocalCdk` flag to use the live CDK code during builds and tests.
124
+
Replace `0.1.0` with the CDK version you are working with. If you're actively developing the CDK and
125
+
want to use the latest version locally, use the `useLocalCdk` flag to use the live CDK code during
126
+
builds and tests.
108
127
109
128
### Developing a connector alongside the CDK
110
129
111
-
You can iterate on changes in the CDK local and test them in the connector without needing to publish the CDK changes publicly.
130
+
You can iterate on changes in the CDK local and test them in the connector without needing to
131
+
publish the CDK changes publicly.
112
132
113
133
When modifying the CDK and a connector in the same PR or branch, please use the following steps:
114
134
115
-
1. Set the version of the CDK in `version.properties` to the next appropriate version number and add a description in the `Changelog` at the bottom of this readme file.
135
+
1. Set the version of the CDK in `version.properties` to the next appropriate version number and add
136
+
a description in the `Changelog` at the bottom of this readme file.
116
137
2. Modify your connector's build.gradle file as follows:
117
-
1. Set `useLocalCdk` to `true` in the connector you are working on. This will ensure the connector always uses the local CDK definitions instead of the published version.
138
+
1. Set `useLocalCdk` to `true` in the connector you are working on. This will ensure the
139
+
connector always uses the local CDK definitions instead of the published version.
118
140
2. Set `cdkVersionRequired` to use the new _to-be-published_ CDK version.
119
141
120
-
After the above, you can build and test your connector as usual. Gradle will automatically use the local CDK code files while you are working on the connector.
142
+
After the above, you can build and test your connector as usual. Gradle will automatically use the
143
+
local CDK code files while you are working on the connector.
121
144
122
145
### Publishing the CDK and switching to a pinned CDK reference
123
146
@@ -128,30 +151,38 @@ Once you are done developing and testing your CDK changes:
128
151
129
152
### Troubleshooting CDK Dependency Caches
130
153
131
-
Note: after switching between a local and a pinned CDK reference, you may need to refresh dependency caches in Gradle and/or your IDE.
154
+
Note: after switching between a local and a pinned CDK reference, you may need to refresh dependency
155
+
caches in Gradle and/or your IDE.
132
156
133
-
In Gradle, you can use the CLI arg `--refresh-dependencies` the next time you build or test your connector, which will ensure that the correct version of the CDK is used after toggling the `useLocalCdk` value.
157
+
In Gradle, you can use the CLI arg `--refresh-dependencies` the next time you build or test your
158
+
connector, which will ensure that the correct version of the CDK is used after toggling the
159
+
`useLocalCdk` value.
134
160
135
161
### Developing a connector against a pinned CDK version
136
162
137
-
You can always pin your connector to a prior stable version of the CDK, which may not match what is the latest version in the `airbyte` repo. For instance, your connector can be pinned to `0.1.1` while the latest version may be `0.2.0`.
163
+
You can always pin your connector to a prior stable version of the CDK, which may not match what is
164
+
the latest version in the `airbyte` repo. For instance, your connector can be pinned to `0.1.1`
165
+
while the latest version may be `0.2.0`.
138
166
139
-
Maven and Gradle will automatically reference the correct (pinned) version of the CDK for your connector, and you can use your local IDE to browse the prior version of the codebase that corresponds to that version.
167
+
Maven and Gradle will automatically reference the correct (pinned) version of the CDK for your
168
+
connector, and you can use your local IDE to browse the prior version of the codebase that
| 0.30.6 | 2024-04-19 |[\#37442](https://github.com/airbytehq/airbyte/pull/37442)| Destinations: Rename File format related classes to be agnostic of S3 |
156
187
| 0.30.3 | 2024-04-12 |[\#37106](https://github.com/airbytehq/airbyte/pull/37106)| Destinations: Simplify constructors in `AsyncStreamConsumer`|
0 commit comments