🎉 New Source: RKI (Robert Koch-Institut) Covid Public API #11732

Zawar92 · 2022-04-05T16:46:51Z

What

Implemented (Robert Koch-Institut - von Marlon Lückert) Covid-19 connector.
The base endpoint in Germany. All other as extension of Germany endpoint.

How

Use the open-source [https://github.com/marlon360/rki-covid-api].
Can also be found at [https://api.corona-zahlen.org/docs/#%E2%98%95%EF%B8%8F-donation]

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

[ X] Community member? Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
[X ] Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
[ X] Documentation updated
- [ X] Connector's README.md
- [ X] Connector's bootstrap.md. See description and examples
- docs/SUMMARY.md
- docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
- docs/integrations/README.md
- airbyte-integrations/builds.md
[ X] PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Results (2.28s):
48 passed

Integration

Put your integration tests output here.

Results (0.01s):

Acceptance

Results (11.49s):
11 passed

…urce with check and discover method added germany.json.

… parameters, updated streams, added cursor field for incremental streams.

…ec, source, schemas updated, added class GermanHistoryIncidence added.

…streams for deaths/:days, recovered/:days, frozen-incidence/:days, hospitalization/:days. Updated source.py methods. Updated sepc.json properties key. Updated configured_catalogue.json with required streams. Updated config.json

… cases and german history incidence.

…ence, germanhistoryhospitalization, germanhistoryrecovered. Fixing other test cases.

…rementalMixin will be implemented to all incremental classes.

marcosmarxm

Thanks for the contribution @Zawar92, I left some comments let me know what do you think of.

marcosmarxm · 2022-04-07T22:41:11Z

airbyte-integrations/connectors/source-rki-covid/source_rki_covid/source.py

+        self, stream_state: Mapping[str, Any] = None, stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None
+    ) -> str:
+        if self.config.get('cases_in_days'):
+            return "germany/history/cases/"+str(self.config.get('cases_in_days'))


You're creating a dependency here for optional parameters and stream. This is not good. Maybe change the cases_in_day to start_date and use to all streams and create a function to convert date to integer days.

added start_date and added a function that converts date to integer days.

marcosmarxm · 2022-04-07T22:42:43Z

airbyte-integrations/connectors/source-rki-covid/source_rki_covid/spec.json

+      "cases_in_days": {
+        "type": "integer",
+        "description": "Number of cases days in the past from today."
+      },
+      "incidence_in_days": {
+        "type": "integer",
+        "description": "Number of incidence days in the past from today."
+      },
+      "deaths_in_days": {
+        "type": "integer",
+        "description": "Number of deaths days in the past from today."
+      },
+      "recovered_in_days": {
+        "type": "integer",
+        "description": "Number of recovered days in the past from today."
+      },
+      "frozen_incidence_in_days": {
+        "type": "integer",
+        "description": "Number of frozen-incidence days in the past from today."
+      },
+      "hospitalization_in_days": {
+        "type": "integer",
+        "description": "Number of hospitalization days in the past from today."


Change those parameters to start_date and let users select the stream and receive data based on that.

changed the parameters to start_date so that user can select the stream and receive data based on that.

marcosmarxm · 2022-04-07T22:44:36Z

airbyte-integrations/connectors/source-rki-covid/source_rki_covid/spec.json

+  "documentationUrl": "https://docsurl.com",
+  "connectionSpecification": {
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "title": "Rki Covid Spec",


Today the correct name of this connector is Rki Covid Germany. Why not create a parameter to select the country Germany and US, if in the future there is a new one would be easy to implement too.

Well the reason is RKI means Robert Koch-Institut COVID-19 API - von Marlon Lückert. Its an institute of Germany, that only deals with data for Germany. So adding a parameter is not a problem.
In future if they choose to give data for other countries then it would be easily implemented

main.py

CLAassistant · 2022-04-08T21:28:42Z

All committers have signed the CLA.

Zawar92 · 2022-04-26T09:52:48Z

Thanks for the comments @marcosmarxm. Can I get an updated on my PR.
It would be helpful.

marcosmarxm · 2022-05-26T19:06:04Z

/test connector=connectors/source-rki-covid

🕑 connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2392594604
❌ connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2392594604
🐛 https://gradle.com/s/dditguhnmnfy2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_airbyte_trace_message_on_failure[inputs0]
======================== 1 failed, 22 passed in 53.62s =========================

marcosmarxm · 2022-05-26T19:42:06Z

/test connector=connectors/source-rki-covid

🕑 connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2392761622
❌ connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2392761622
🐛 https://gradle.com/s/mi33dtr4q7xjs

Build Failed

Test summary info:

Could not find result summary

marcosmarxm · 2022-05-26T20:01:51Z

/test connector=connectors/source-rki-covid

🕑 connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2392845646
❌ connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2392845646
🐛 https://gradle.com/s/recbgp24bah3o

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
======================== 1 failed, 22 passed in 35.45s =========================

marcosmarxm · 2022-05-26T20:21:31Z

/test connector=connectors/source-rki-covid

🕑 connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2392936388
✅ connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2392936388
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                           Stmts   Miss  Cover
--------------------------------------------------
source_rki_covid/__init__.py       2      0   100%
source_rki_covid/source.py       243     62    74%
--------------------------------------------------
TOTAL                            245     62    75%

Build Passed

Test summary info:

All Passed

marcosmarxm · 2022-05-27T19:26:27Z

/publish connector=connectors/source-rki-covid

🕑 connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2398381594
❌ Failed to publish connectors/source-rki-covid
❌ Couldn't auto-bump version for connectors/source-rki-covid

marcosmarxm · 2022-05-27T19:47:07Z

/publish connector=connectors/source-rki-covid

🕑 connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2398463382
🚀 Successfully published connectors/source-rki-covid
🚀 Auto-bumped version for connectors/source-rki-covid
✅ connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2398463382

…byte into marcos/test-pr-11732

marcosmarxm · 2022-05-30T12:32:55Z

/publish connector=connectors/source-rki-covid

🕑 connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2409067435
🚀 Successfully published connectors/source-rki-covid
🚀 Auto-bumped version for connectors/source-rki-covid
✅ connectors/source-rki-covid https://github.com/airbytehq/airbyte/actions/runs/2409067435

…11732) * Added source for RKI-covid-germany, updated spec.json, implemented source with check and discover method added germany.json. * implemented incremental method for germany history cases with date as parameters, updated streams, added cursor field for incremental streams. * main file added. * added Incidence source with date as parameter. Incremental stream. spec, source, schemas updated, added class GermanHistoryIncidence added. * Added a full-refresh stream for germany/age-group .Added incremental streams for deaths/:days, recovered/:days, frozen-incidence/:days, hospitalization/:days. Updated source.py methods. Updated sepc.json properties key. Updated configured_catalogue.json with required streams. Updated config.json * writting test unit test cases for incremental streams: german history cases and german history incidence. * Incremental streams for germanhistorydeaths, germanhistoryfrozenIncidence, germanhistoryhospitalization, germanhistoryrecovered. Fixing other test cases. * Added test stream for age group and germany. * changes in Readme and source.py. * IncrementalMixin added to class GermanyHistoryCases. AFter review IncrementalMixin will be implemented to all incremental classes. * Implemented Incremental mixin or GermanHistory Cases connector. * corrected changes. * adding integration test * comment acceptence test. * On path with master * updated the changes for PR request. * changed file source-rki-covid/integration_tests/configured_catalog.json * corrected flake and blackformatting. Build gradel. * Worked on the suggestions. * source_rki_covid/schemas/germany_age_groups.json * uodated abnormal_state.json * updated the schemas for german age groups and history hospitalization. * correct dockerfile and update airbyte_cdk version * run format * update python version * correct dockerfile build * add source in seed * update uuid for rki-covid source * change docker * add bash * auto-bump connector version * run seed file * correct doc * auto-bump connector version Co-authored-by: Marcos Marx Millnitz <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]>

Zawar92 added 18 commits March 7, 2022 17:17

Added source for RKI-covid-germany, updated spec.json, implemented so…

c8b42e0

…urce with check and discover method added germany.json.

implemented incremental method for germany history cases with date as…

2c93e79

… parameters, updated streams, added cursor field for incremental streams.

main file added.

55e513f

added Incidence source with date as parameter. Incremental stream. sp…

5041be2

…ec, source, schemas updated, added class GermanHistoryIncidence added.

writting test unit test cases for incremental streams: german history…

d40604a

… cases and german history incidence.

Incremental streams for germanhistorydeaths, germanhistoryfrozenIncid…

c383fb7

…ence, germanhistoryhospitalization, germanhistoryrecovered. Fixing other test cases.

Added test stream for age group and germany.

b2f8fee

changes in Readme and source.py.

51cc121

IncrementalMixin added to class GermanyHistoryCases. AFter review Inc…

d829806

…rementalMixin will be implemented to all incremental classes.

Implemented Incremental mixin or GermanHistory Cases connector.

7836aa0

corrected changes.

6299be9

updated with master.

bfe7929

adding integration test

33f710b

fetch stream master

99bcadb

comment acceptence test.

62e7df9

On path with master

1bd5a35

On path with master

b5391db

github-actions bot added the area/connectors Connector related issues label Apr 5, 2022

octavia-squidington-iii added the community label Apr 5, 2022

marcosmarxm requested changes Apr 7, 2022

View reviewed changes

marcosmarxm self-assigned this Apr 7, 2022

updated the changes for PR request.

3eaa129

updated with master.

4f97496

Zawar92 requested a review from marcosmarxm April 13, 2022 09:30

marcosmarxm changed the title ~~Zawar92/add source rki covid~~ 🎉 New Source: RKI (Robert Koch-Institut) Covid API Apr 19, 2022

marcosmarxm changed the title ~~🎉 New Source: RKI (Robert Koch-Institut) Covid API~~ 🎉 New Source: RKI (Robert Koch-Institut) Covid Public API Apr 19, 2022

changed file source-rki-covid/integration_tests/configured_catalog.json

c37d843

marcosmarxm added 2 commits May 26, 2022 16:10

run format

562906e

update python version

83827e6

correct dockerfile build

8c44676

merge master

bf78600

marcosmarxm added 4 commits May 26, 2022 17:35

add source in seed

5c7f8d4

update uuid for rki-covid source

9ea2a53

Merge branch 'master' into marcos/test-pr-11732

fdc266f

change docker

f3b23ab

add bash

857dc42

marcosmarxm and others added 5 commits May 27, 2022 16:47

Merge branch 'master' into marcos/test-pr-11732

6b3c29a

auto-bump connector version

5e36611

run seed file

75e5c33

Merge branch 'zawar92/add_source_rki_covid' of github.com:Zawar92/air…

8f0a47f

…byte into marcos/test-pr-11732

correct doc

ef8f98a

github-actions bot added the area/documentation Improvements or additions to documentation label May 30, 2022

auto-bump connector version

da459e4

marcosmarxm approved these changes May 30, 2022

View reviewed changes

marcosmarxm merged commit d6d52c5 into airbytehq:master May 30, 2022

octavia-squidington-iii mentioned this pull request May 31, 2022

Bump Airbyte version from 0.39.5-alpha to 0.39.6-alpha #13350

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: RKI (Robert Koch-Institut) Covid Public API #11732

🎉 New Source: RKI (Robert Koch-Institut) Covid Public API #11732

Zawar92 commented Apr 5, 2022

marcosmarxm left a comment

marcosmarxm Apr 7, 2022

Zawar92 Apr 8, 2022

marcosmarxm Apr 7, 2022

Zawar92 Apr 8, 2022

marcosmarxm Apr 7, 2022

Zawar92 Apr 8, 2022

CLAassistant commented Apr 8, 2022 •

edited

Loading

Zawar92 commented Apr 26, 2022

marcosmarxm commented May 26, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 26, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 26, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 26, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 27, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 27, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 30, 2022 •

edited by github-actions bot

Loading

🎉 New Source: RKI (Robert Koch-Institut) Covid Public API #11732

🎉 New Source: RKI (Robert Koch-Institut) Covid Public API #11732

Conversation

Zawar92 commented Apr 5, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm Apr 7, 2022

Choose a reason for hiding this comment

Zawar92 Apr 8, 2022

Choose a reason for hiding this comment

marcosmarxm Apr 7, 2022

Choose a reason for hiding this comment

Zawar92 Apr 8, 2022

Choose a reason for hiding this comment

marcosmarxm Apr 7, 2022

Choose a reason for hiding this comment

Zawar92 Apr 8, 2022

Choose a reason for hiding this comment

CLAassistant commented Apr 8, 2022 • edited Loading

Zawar92 commented Apr 26, 2022

marcosmarxm commented May 26, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented May 26, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented May 26, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented May 26, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented May 27, 2022 • edited by github-actions bot Loading

marcosmarxm commented May 27, 2022 • edited by github-actions bot Loading

marcosmarxm commented May 30, 2022 • edited by github-actions bot Loading

CLAassistant commented Apr 8, 2022 •

edited

Loading

marcosmarxm commented May 26, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 26, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 26, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 26, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 27, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 27, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 30, 2022 •

edited by github-actions bot

Loading