🚀 Source Klaviyo: New Stream addition along with update to existing #11685

ahmed-buksh · 2022-04-04T07:37:28Z

What

Enhance Klaviyo source to provide flows information along with event updated with relational information.

How

Flows schema is added
Event schema is updated with new fields like flow_id, campaign_id, and flow_message_id

🚨 User Impact 🚨

No breaking changes

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…on to event

marcosmarxm · 2022-04-04T22:19:56Z

@ahmed-buksh please ping me this PR is ready to review.

marcosmarxm · 2022-04-14T16:23:33Z

@ahmed-buksh are you planning to keep working on this? Today is a draft, please let me know!

ahmed-buksh · 2022-04-15T06:05:48Z

@marcosmarxm sorry for the delay, I switched to some other work. I am just ensuring a few tests are working fine so it's in progress for now.

ahmed-buksh · 2022-04-15T11:08:55Z

@marcosmarxm I made a couple of changes that were breaking a test but I generally see 2 tests failing on Klaivyo's existing code base as well. Should we halt until we have to fix them as well or we can move forward with this?

marcosmarxm · 2022-04-19T00:38:02Z

/test connector=connectors/source-klaviyo

🕑 connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2186848392
❌ connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2186848392
🐛 https://gradle.com/s/pfg6t5kieju7o

harshithmullapudi · 2022-04-27T08:47:58Z

/test connector=connectors/source-klaviyo

🕑 connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2231927358
❌ connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2231927358
🐛 https://gradle.com/s/s4p7m6c3taytg

CLAassistant · 2022-05-05T13:14:25Z

All committers have signed the CLA.

harshithmullapudi · 2022-05-09T10:33:38Z

@ahmed-buksh can you check the failed tests and share the screenshot of tests passing in local

ahmed-buksh · 2022-05-09T14:10:03Z

@harshithmullapudi I just fixed the tests and they are all passed.

harshithmullapudi · 2022-05-16T10:19:29Z

/test connector=connectors/source-klaviyo

🕑 connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2331243705
❌ connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2331243705
🐛 https://gradle.com/s/haw7gffyiauxg

harshithmullapudi · 2022-05-16T10:42:04Z

Hey @ahmed-buksh can you check these

Merge the latest master
Run ./gradlew format
Check for the sourceAcceptanceTest

…hancement/source-klaviyo

ahmed-buksh · 2022-05-25T12:56:54Z

@harshithmullapudi I executed ./gradlew :airbyte-integrations:connectors:source-klaviyo:format from root directory but klaviyo has no support for format
Other than that I pulled master and re-ran tests and it worked fine.
As per previous log, there seems to be a Flake8 issue which I just fixed and pushed.

ahmed-buksh · 2022-05-25T13:03:42Z

/test connector=connectors/source-klaviyo

harshithmullapudi · 2022-05-31T04:56:42Z

/test connector=connectors/source-klaviyo

🕑 connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2412743066
❌ connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2412743066
🐛 https://gradle.com/s/b3pm4fljddwp2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
======================== 1 failed, 23 passed in 17.42s =========================

harshithmullapudi · 2022-05-31T05:48:31Z

/test connector=connectors/source-klaviyo

🕑 connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2412922642
✅ connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2412922642
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                         Stmts   Miss  Cover
------------------------------------------------
source_klaviyo/__init__.py       2      0   100%
source_klaviyo/streams.py      136     21    85%
source_klaviyo/schemas.py      102     26    75%
source_klaviyo/source.py        28     12    57%
------------------------------------------------
TOTAL                          268     59    78%

Build Passed

Test summary info:

All Passed

harshithmullapudi · 2022-05-31T06:17:49Z

/publish connector=connectors/source-klaviyo auto-bump-version=false

🕑 connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2413035840
🚀 Successfully published connectors/source-klaviyo
✅ connectors/source-klaviyo https://github.com/airbytehq/airbyte/actions/runs/2413035840

…irbytehq#11685) * 🚀 flow stream added along with flow, campaign and flow message addition to event * ⚡ liniting fix * 🔨 annotations updated along with update method for klaviyo * 💥 docker version updated and log added * 🔨 fixed acceptance test for klaviyo flow stream * 🔨 unused import removed * fix: flows stream has no records thus tests are failing * chore: update seed file Co-authored-by: Harshith Mullapudi <[email protected]>

🚀 flow stream added along with flow, campaign and flow message additi…

b148298

…on to event

github-actions bot added the area/connectors Connector related issues label Apr 4, 2022

octavia-squidington-iii added the community label Apr 4, 2022

⚡ liniting fix

a7a81c5

marcosmarxm added the blocked label Apr 14, 2022

ahmed-buksh marked this pull request as ready for review April 15, 2022 06:05

ahmed-buksh changed the title ~~🚀 Enhancement to Klaviyo Source~~ 🚀 [WIP] Enhancement to Klaviyo Source Apr 15, 2022

ahmed-buksh added 2 commits April 15, 2022 15:54

🔨 annotations updated along with update method for klaviyo

95349b3

💥 docker version updated and log added

fd4aa7a

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 15, 2022

ahmed-buksh changed the title ~~🚀 [WIP] Enhancement to Klaviyo Source~~ 🚀 Enhancement to Klaviyo Source Apr 15, 2022

ahmed-buksh changed the title ~~🚀 Enhancement to Klaviyo Source~~ 🚀 Source Klaviyo: New Stream addition along with update to existing Apr 15, 2022

harshithmullapudi self-requested a review April 25, 2022 15:01

harshithmullapudi self-assigned this Apr 25, 2022

🔨 fixed acceptance test for klaviyo flow stream

0e07e40

harshithmullapudi removed the blocked label May 16, 2022

Merge branch 'master' of https://github.com/airbytehq/airbyte into en…

7e867f7

…hancement/source-klaviyo

🔨 unused import removed

c31c722

harshithmullapudi mentioned this pull request May 31, 2022

Source Klaviyo: need to create data for flows stream #13321

Closed

fix: flows stream has no records thus tests are failing

e255c08

chore: update seed file

34af4e6

harshithmullapudi force-pushed the enhancement/source-klaviyo branch from 9e5fdb9 to 34af4e6 Compare May 31, 2022 06:42

harshithmullapudi merged commit c55f185 into airbytehq:master May 31, 2022

octavia-squidington-iii mentioned this pull request May 31, 2022

Bump Airbyte version from 0.39.5-alpha to 0.39.6-alpha #13350

Merged

🚀 Source Klaviyo: New Stream addition along with update to existing #11685

🚀 Source Klaviyo: New Stream addition along with update to existing #11685

Uh oh!

Conversation

ahmed-buksh commented Apr 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

Uh oh!

marcosmarxm commented Apr 4, 2022

Uh oh!

marcosmarxm commented Apr 14, 2022

Uh oh!

ahmed-buksh commented Apr 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahmed-buksh commented Apr 15, 2022

Uh oh!

marcosmarxm commented Apr 19, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harshithmullapudi commented Apr 27, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented May 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harshithmullapudi commented May 9, 2022

Uh oh!

ahmed-buksh commented May 9, 2022

Uh oh!

harshithmullapudi commented May 16, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harshithmullapudi commented May 16, 2022

Uh oh!

ahmed-buksh commented May 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahmed-buksh commented May 25, 2022

Uh oh!

harshithmullapudi commented May 31, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

harshithmullapudi commented May 31, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

harshithmullapudi commented May 31, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ahmed-buksh commented Apr 4, 2022 •

edited

Loading

ahmed-buksh commented Apr 15, 2022 •

edited

Loading

marcosmarxm commented Apr 19, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented Apr 27, 2022 •

edited by github-actions bot

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

harshithmullapudi commented May 16, 2022 •

edited by github-actions bot

Loading

ahmed-buksh commented May 25, 2022 •

edited

Loading

harshithmullapudi commented May 31, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented May 31, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented May 31, 2022 •

edited by github-actions bot

Loading