🎉Source Sendgrid: increase unit test coverage at least 90% (#16332)

* Added unit tests

* Updated test name

* Updated setup

* Fix requirements

* Updated release stage

* Updated expected_records

Author: lazebnyi

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

@@ -904,7 +904,7 @@
   documentationUrl: https://docs.airbyte.io/integrations/sources/sendgrid
   icon: sendgrid.svg
   sourceType: api
-  releaseStage: alpha
+  releaseStage: beta
 - name: Shopify
   sourceDefinitionId: 9da77001-af33-4bcd-be46-6252bf9342b9
   dockerRepository: airbyte/source-shopify

airbyte-integrations/connectors/source-sendgrid/README.md

@@ -1,6 +1,6 @@
 # Sendgrid Source 
 
-This is the repository for the Sendgrid source connector, written in Python. 
+This is the repository for the Sendgrid source connector, written in Python.
 For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/sendgrid).
 
 ## Local development
@@ -21,6 +21,7 @@ development environment of choice. To activate it from the terminal, run:
 ```
 source .venv/bin/activate
 pip install -r requirements.txt
+pip install '.[tests]'
 ```
 If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
 
@@ -30,21 +31,22 @@ If this is mumbo jumbo to you, don't worry about it, just put your deps in `setu
 should work as you expect.
 
 #### Building via Gradle
-From the Airbyte repository root, run:
+You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.
+
+To build using Gradle, from the Airbyte repository root, run:
 ```
 ./gradlew :airbyte-integrations:connectors:source-sendgrid:build
 ```
 
 #### Create credentials
 **If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/sendgrid)
 to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_sendgrid/spec.json` file.
-Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
+Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
 See `integration_tests/sample_config.json` for a sample config file.
 
 **If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source sendgrid test creds`
 and place them into `secrets/config.json`.
 
-
 ### Locally running the connector
 ```
 python main.py spec
@@ -53,12 +55,6 @@ python main.py discover --config secrets/config.json
 python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
 ```
 
-### Unit Tests
-To run unit tests locally, from the connector directory run:
-```
-python -m pytest unit_tests
-```
-
 ### Locally running the connector docker image
 
 #### Build
@@ -82,22 +78,55 @@ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-sendgrid:dev check --c
 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-sendgrid:dev discover --config /secrets/config.json
 docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-sendgrid:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
 ```
+## Testing
+Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
+First install test dependencies into your virtual environment:
+```
+pip install .[tests]
+```
+### Unit Tests
+To run unit tests locally, from the connector directory run:
+```
+python -m pytest unit_tests
+```
 
 ### Integration Tests
-1. From the airbyte project root, run `./gradlew :airbyte-integrations:connectors:source-sendgrid:integrationTest` to run the standard integration test suite.
-1. To run additional integration tests, place your integration tests in a new directory `integration_tests` and run them with `python -m pytest -s integration_tests`.
-   Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
+There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
+#### Custom Integration tests
+Place custom tests inside `integration_tests/` folder, then, from the connector root, run
+```
+python -m pytest integration_tests
+```
+#### Acceptance Tests
+Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+To run your integration tests with acceptance tests, from the connector root, run
+```
+python -m pytest integration_tests -p integration_tests.acceptance
+```
+To run your integration tests with docker
+
+### Using gradle to run tests
+All commands should be run from airbyte project root.
+To run unit tests:
+```
+./gradlew :airbyte-integrations:connectors:source-sendgrid:unitTest
+```
+To run acceptance and custom integration tests:
+```
+./gradlew :airbyte-integrations:connectors:source-sendgrid:integrationTest
+```
 
 ## Dependency Management
 All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
+We split dependencies between two groups, dependencies that are:
+* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
+* required for the testing need to go to `TEST_REQUIREMENTS` list
 
 ### Publishing a new version of the connector
 You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
-1. Make sure your changes are passing unit and integration tests
-1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use SemVer).
-1. Create a Pull Request
-1. Pat yourself on the back for being an awesome contributor
-1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master
-
-### Changelog
-See the [docs](https://docs.airbyte.io/integrations/sources/sendgrid#changelog) for the changelog.
+1. Make sure your changes are passing unit and integration tests.
+1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
+1. Create a Pull Request.
+1. Pat yourself on the back for being an awesome contributor.
+1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

airbyte-integrations/connectors/source-sendgrid/integration_tests/expected_records.txt

@@ -1,25 +1,3 @@
-{"stream": "campaigns", "data": {"created_at": "2021-09-08T09:07:48Z", "id": "3c5a9fa6-1084-11ec-ac32-4228d699bad5", "name": "Untitled Single Send", "status": "triggered", "updated_at": "2021-09-08T09:11:08Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-09-08T09:04:36Z", "id": "c9f286fb-1083-11ec-ae03-ca0fc7f28419", "name": "Copy of Untitled Single Send", "status": "triggered", "updated_at": "2021-09-08T09:09:08Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-09-08T08:53:59Z", "id": "4e5be6a3-1082-11ec-8512-9afd40c324e6", "name": "Untitled Single Send", "status": "triggered", "updated_at": "2021-09-08T08:57:08Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-09-08T08:51:59Z", "id": "06ee105f-1082-11ec-8245-86a627812e3d", "name": "Untitled Single Send", "status": "triggered", "updated_at": "2021-09-08T08:55:08Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:13:02Z", "id": "d497b877-6486-11eb-be53-b2a243c7228c", "name": "Campaign 18", "status": "draft", "updated_at": "2021-02-01T12:13:02Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:13:00Z", "id": "d36a06e7-6486-11eb-bb4f-823d082c01b8", "name": "Campaign 17", "status": "draft", "updated_at": "2021-02-01T12:13:00Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:58Z", "id": "d258b5a0-6486-11eb-8b51-8aa6caa37fdd", "name": "Campaign 16", "status": "draft", "updated_at": "2021-02-01T12:12:58Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:56Z", "id": "d12b87ec-6486-11eb-be53-b2a243c7228c", "name": "Campaign 15", "status": "draft", "updated_at": "2021-02-01T12:12:56Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:54Z", "id": "d0072250-6486-11eb-bd19-4680e22af4b6", "name": "Campaign 14", "status": "draft", "updated_at": "2021-02-01T12:12:54Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:53Z", "id": "cf204596-6486-11eb-bd19-4680e22af4b6", "name": "Campaign 13", "status": "draft", "updated_at": "2021-02-01T12:12:53Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:51Z", "id": "cdfe8cde-6486-11eb-be53-b2a243c7228c", "name": "Campaign 12", "status": "draft", "updated_at": "2021-02-01T12:12:51Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:49Z", "id": "ccecefcb-6486-11eb-bd77-2a301ccc59da", "name": "Campaign 11", "status": "draft", "updated_at": "2021-02-01T12:12:49Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:46Z", "id": "caeabedf-6486-11eb-bd19-4680e22af4b6", "name": "Campaign 10", "status": "draft", "updated_at": "2021-02-01T12:12:46Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:43Z", "id": "c93431e7-6486-11eb-8b51-8aa6caa37fdd", "name": "Campaign 9", "status": "draft", "updated_at": "2021-02-01T12:12:43Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:41Z", "id": "c7eb9306-6486-11eb-bd19-4680e22af4b6", "name": "Campaign 8", "status": "draft", "updated_at": "2021-02-01T12:12:41Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:38Z", "id": "c646b948-6486-11eb-bd77-2a301ccc59da", "name": "Campaign 7", "status": "draft", "updated_at": "2021-02-01T12:12:38Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:35Z", "id": "c47ccad9-6486-11eb-8b51-8aa6caa37fdd", "name": "Campaign 6", "status": "draft", "updated_at": "2021-02-01T12:12:35Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:33Z", "id": "c36c66f3-6486-11eb-bd19-4680e22af4b6", "name": "Campaign 5", "status": "draft", "updated_at": "2021-02-01T12:12:33Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:31Z", "id": "c24fdc68-6486-11eb-bd77-2a301ccc59da", "name": "Campaign 4", "status": "draft", "updated_at": "2021-02-01T12:12:31Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:25Z", "id": "be9d147f-6486-11eb-8b51-8aa6caa37fdd", "name": "Third Campaign", "status": "draft", "updated_at": "2021-02-01T12:12:25Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:12:18Z", "id": "ba43f256-6486-11eb-bb4f-823d082c01b8", "name": "Second Campaign", "status": "draft", "updated_at": "2021-02-01T12:12:18Z", "is_abtest": false}, "emitted_at": 1631093369000}
-{"stream": "campaigns", "data": {"created_at": "2021-02-01T12:10:59Z", "id": "8b17a7b7-6486-11eb-bd77-2a301ccc59da", "name": "First Campaign", "status": "draft", "updated_at": "2021-02-01T12:10:59Z", "is_abtest": false}, "emitted_at": 1631093369000}
 {"stream": "lists", "data": {"name": "Test List: 19", "id": "0236d6d2-75d2-42c5-962d-603e0deaf8d1", "contact_count": 20, "_metadata": {"self": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2-75d2-42c5-962d-603e0deaf8d1"}}, "emitted_at": 1631093370000}
 {"stream": "lists", "data": {"name": "List for CI tests, number 30", "id": "041ee031-005e-41e7-ad3b-a427f90f54af", "contact_count": 0, "_metadata": {"self": "https://api.sendgrid.com/v3/marketing/lists/041ee031-005e-41e7-ad3b-a427f90f54af"}}, "emitted_at": 1631093370000}
 {"stream": "lists", "data": {"name": "my_list", "id": "07315be4-500c-4f30-8217-22cb8e39dd37", "contact_count": 0, "_metadata": {"self": "https://api.sendgrid.com/v3/marketing/lists/07315be4-500c-4f30-8217-22cb8e39dd37"}}, "emitted_at": 1631093370000}

airbyte-integrations/connectors/source-sendgrid/requirements.txt

@@ -1,2 +1,2 @@
-# This file is autogenerated -- only edit if you know what you are doing. Use setup.py for declaring dependencies.
+-e ../../bases/source-acceptance-test
 -e .

airbyte-integrations/connectors/source-sendgrid/setup.py

@@ -5,12 +5,27 @@
 
 from setuptools import find_packages, setup
 
+MAIN_REQUIREMENTS = [
+    "airbyte-cdk",
+    "backoff",
+    "requests",
+]
+
+TEST_REQUIREMENTS = [
+    "pytest~=6.1",
+    "source-acceptance-test",
+    "requests-mock",
+]
+
 setup(
     name="source_sendgrid",
     description="Source implementation for Sendgrid.",
     author="Airbyte",
     author_email="[email protected]",
     packages=find_packages(),
-    install_requires=["airbyte-cdk~=0.1", "backoff", "requests", "pytest==6.1.2", "pytest-mock"],
-    package_data={"": ["*.json", "schemas/*.json"]},
+    install_requires=MAIN_REQUIREMENTS,
+    package_data={"": ["*.json", "schemas/*.json", "schemas/shared/*.json"]},
+    extras_require={
+        "tests": TEST_REQUIREMENTS,
+    },
 )

airbyte-integrations/connectors/source-sendgrid/unit_tests/unit_test.py

@@ -3,14 +3,29 @@
 #
 
 import unittest
-from unittest.mock import MagicMock
+from unittest.mock import MagicMock, patch
 
 import pendulum
 import pytest
 import requests
 from airbyte_cdk.logger import AirbyteLogger
+from airbyte_cdk.models import SyncMode
 from source_sendgrid.source import SourceSendgrid
-from source_sendgrid.streams import Messages, SendgridStream
+from source_sendgrid.streams import (
+    Blocks,
+    Campaigns,
+    Contacts,
+    GlobalSuppressions,
+    Lists,
+    Messages,
+    Segments,
+    SendgridStream,
+    SendgridStreamIncrementalMixin,
+    SendgridStreamOffsetPagination,
+    SuppressionGroupMembers,
+    SuppressionGroups,
+    Templates,
+)
 
 FAKE_NOW = pendulum.DateTime(2022, 1, 1, tzinfo=pendulum.timezone("utc"))
 
@@ -52,3 +67,60 @@ def test_messages_stream_request_params(mock_pendulum_now):
         request_params
         == "query=last_event_time%20BETWEEN%20TIMESTAMP%20%222019-05-20T13%3A30%3A00Z%22%20AND%20TIMESTAMP%20%222022-01-01T00%3A00%3A00Z%22&limit=1000"
     )
+
+
+def test_streams():
+    streams = SourceSendgrid().streams(config={"apikey": "wrong.api.key123", "start_time": FAKE_NOW})
+
+    assert len(streams) == 15
+
+
+@patch.multiple(SendgridStreamOffsetPagination, __abstractmethods__=set())
+def test_pagination(mocker):
+    stream = SendgridStreamOffsetPagination()
+    state = {}
+    response = requests.Response()
+    mocker.patch.object(response, "json", return_value={None: 1})
+    mocker.patch.object(response, "request", return_value=MagicMock())
+    next_page_token = stream.next_page_token(response)
+    request_params = stream.request_params(stream_state=state, next_page_token=next_page_token)
+    assert request_params == {"limit": 50}
+
+
+@patch.multiple(SendgridStreamIncrementalMixin, __abstractmethods__=set())
+def test_stream_state():
+    stream = SendgridStreamIncrementalMixin(start_time=FAKE_NOW)
+    state = {}
+    request_params = stream.request_params(stream_state=state)
+    assert request_params == {"end_time": pendulum.now().int_timestamp, "start_time": FAKE_NOW}
+
+
+@pytest.mark.parametrize(
+    "stream_class, url , expected",
+    (
+        [Templates, "https://api.sendgrid.com/v3/templates", []],
+        [Lists, "https://api.sendgrid.com/v3/marketing/lists", []],
+        [Campaigns, "https://api.sendgrid.com/v3/marketing/campaigns", []],
+        [Contacts, "https://api.sendgrid.com/v3/marketing/contacts", []],
+        [Segments, "https://api.sendgrid.com/v3/marketing/segments", []],
+        [Blocks, "https://api.sendgrid.com/v3/suppression/blocks", ["name", "id", "contact_count", "_metadata"]],
+        [SuppressionGroupMembers, "https://api.sendgrid.com/v3/asm/suppressions", ["name", "id", "contact_count", "_metadata"]],
+        [SuppressionGroups, "https://api.sendgrid.com/v3/asm/groups", ["name", "id", "contact_count", "_metadata"]],
+        [GlobalSuppressions, "https://api.sendgrid.com/v3/suppression/unsubscribes", ["name", "id", "contact_count", "_metadata"]],
+    ),
+)
+def test_read_records(
+    stream_class,
+    url,
+    expected,
+    requests_mock,
+):
+    try:
+        stream = stream_class(start_time=FAKE_NOW)
+    except TypeError:
+        stream = stream_class()
+    requests_mock.get("https://api.sendgrid.com/v3/marketing", json={})
+    requests_mock.get(url, json={"name": "test", "id": "id", "contact_count": 20, "_metadata": {"self": "self"}})
+    records = list(stream.read_records(sync_mode=SyncMode))
+
+    assert records == expected