Source twitter: Migrate to manifest-only (#44777)

Author: ChristoGrab

airbyte-integrations/connectors/source-twitter/README.md

@@ -1,89 +1,63 @@
 # Twitter source connector
 
+This directory contains the manifest-only connector for `source-twitter`.
+This _manifest-only_ connector is not a Python package on its own, as it runs inside of the base `source-declarative-manifest` image.
 
-This is the repository for the Twitter source connector, written in Python.
-For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/twitter).
+For information about how to configure and use this connector within Airbyte, see [the connector's full documentation](https://docs.airbyte.com/integrations/sources/twitter).
 
 ## Local development
 
-### Prerequisites
-* Python (~=3.9)
-* Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
+We recommend using the Connector Builder to edit this connector.
+Using either Airbyte Cloud or your local Airbyte OSS instance, navigate to the **Builder** tab and select **Import a YAML**.
+Then select the connector's `manifest.yaml` file to load the connector into the Builder. You're now ready to make changes to the connector!
 
+If you prefer to develop locally, you can follow the instructions below.
 
-### Installing the connector
-From this connector directory, run:
-```bash
-poetry install --with dev
-```
-
-
-### Create credentials
-**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/twitter)
-to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_twitter/spec.yaml` file.
-Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
-See `sample_files/sample_config.json` for a sample config file.
-
-
-### Locally running the connector
-```
-poetry run source-twitter spec
-poetry run source-twitter check --config secrets/config.json
-poetry run source-twitter discover --config secrets/config.json
-poetry run source-twitter read --config secrets/config.json --catalog sample_files/configured_catalog.json
-```
+### Building the docker image
 
-### Running unit tests
-To run unit tests locally, from the connector directory run:
-```
-poetry run pytest unit_tests
-```
+You can build any manifest-only connector with `airbyte-ci`:
 
-### Building the docker image
 1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
 2. Run the following command to build the docker image:
+
 ```bash
 airbyte-ci connectors --name=source-twitter build
 ```
 
 An image will be available on your host with the tag `airbyte/source-twitter:dev`.
 
+### Creating credentials
+
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/twitter)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `spec` object in the connector's `manifest.yaml` file.
+Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
 
 ### Running as a docker container
-Then run any of the connector commands as follows:
-```
+
+Then run any of the standard source connector commands:
+
+```bash
 docker run --rm airbyte/source-twitter:dev spec
 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-twitter:dev check --config /secrets/config.json
 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-twitter:dev discover --config /secrets/config.json
 docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-twitter:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
 ```
 
-### Running our CI test suite
-You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
-```bash
-airbyte-ci connectors --name=source-twitter test
-```
+### Running the CI test suite
 
-### Customizing acceptance Tests
-Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
-If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
 
-### Dependency Management
-All of your dependencies should be managed via Poetry. 
-To add a new dependency, run:
 ```bash
-poetry add <package-name>
+airbyte-ci connectors --name=source-twitter test
 ```
 
-Please commit the changes to `pyproject.toml` and `poetry.lock` files.
-
 ## Publishing a new version of the connector
-You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
-1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-twitter test`
-2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): 
+
+If you want to contribute changes to `source-twitter`, here's how you can do that:
+1. Make your changes locally, or load the connector's manifest into Connector Builder and make changes there.
+2. Make sure your changes are passing our test suite with `airbyte-ci connectors --name=source-twitter test`
+3. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
     - bump the `dockerImageTag` value in in `metadata.yaml`
-    - bump the `version` value in `pyproject.toml`
-3. Make sure the `metadata.yaml` content is up to date.
 4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/twitter.md`).
 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
 6. Pat yourself on the back for being an awesome contributor.

airbyte-integrations/connectors/source-twitter/__init__.py

@@ -5,7 +5,7 @@ test_strictness_level: "high"
 acceptance_tests:
   spec:
     tests:
-      - spec_path: "source_twitter/spec.yaml"
+      - spec_path: "manifest.yaml"
   connection:
     tests:
       - config_path: "secrets/config.json"
@@ -19,7 +19,10 @@ acceptance_tests:
     tests:
       - config_path: "secrets/config.json"
         expect_records:
-          bypass_reason: "Only one stream tweets available. Retrieving tweets cannot be more than 7 days in the past. So need skip for expected records, but not skip for basic read."
+          bypass_reason:
+            "Only one stream tweets available. Retrieving tweets cannot
+            be more than 7 days in the past. So need skip for expected records, but
+            not skip for basic read."
   incremental:
     tests:
       - config_path: "secrets/config.json"

airbyte-integrations/connectors/source-twitter/acceptance-test-config.yml

@@ -0,0 +1,184 @@
+version: 4.5.4
+
+type: DeclarativeSource
+
+check:
+  type: CheckStream
+  stream_names:
+    - tweets
+
+definitions:
+  streams:
+    tweets:
+      type: DeclarativeStream
+      name: tweets
+      primary_key:
+        - id
+      retriever:
+        type: SimpleRetriever
+        requester:
+          $ref: "#/definitions/base_requester"
+          path: /search/recent
+          http_method: GET
+          request_parameters:
+            query: "{{ config['query'] }}"
+            tweet.fields: author_id,conversation_id,created_at,in_reply_to_user_id,lang
+        record_selector:
+          type: RecordSelector
+          extractor:
+            type: DpathExtractor
+            field_path:
+              - data
+        paginator:
+          type: DefaultPaginator
+          page_token_option:
+            type: RequestOption
+            inject_into: request_parameter
+            field_name: next_token
+          page_size_option:
+            type: RequestOption
+            field_name: max_results
+            inject_into: request_parameter
+          pagination_strategy:
+            type: CursorPagination
+            page_size: 100
+            cursor_value: "{{ response.meta.next_token }}"
+            stop_condition: "{{ 'next_token' not in response['meta'] }}"
+      incremental_sync:
+        type: DatetimeBasedCursor
+        cursor_field: created_at
+        cursor_datetime_formats:
+          - "%Y-%m-%dT%H:%M:%S.000Z"
+        datetime_format: "%Y-%m-%dT%H:%M:%S.000Z"
+        start_datetime:
+          type: MinMaxDatetime
+          datetime: >-
+            {{ config.start_date or (now_utc() -
+            duration('P6DT23H')).strftime('%Y-%m-%dT%H:%M:%SZ') }}
+          datetime_format: "%Y-%m-%dT%H:%M:%SZ"
+        start_time_option:
+          type: RequestOption
+          field_name: start_time
+          inject_into: request_parameter
+        end_time_option:
+          type: RequestOption
+          field_name: end_time
+          inject_into: request_parameter
+        end_datetime:
+          type: MinMaxDatetime
+          datetime: >-
+            {{ config.end_date or (now_utc() -
+            duration('PT10S')).strftime('%Y-%m-%dT%H:%M:%SZ') }}
+          datetime_format: "%Y-%m-%dT%H:%M:%SZ"
+        step: P7D
+        cursor_granularity: PT0S
+      schema_loader:
+        type: InlineSchemaLoader
+        schema:
+          $ref: "#/schemas/tweets"
+  base_requester:
+    type: HttpRequester
+    url_base: https://api.twitter.com/2/tweets
+    authenticator:
+      type: BearerAuthenticator
+      header: apikey
+      api_token: "{{ config.api_key }}"
+
+streams:
+  - $ref: "#/definitions/streams/tweets"
+
+spec:
+  type: Spec
+  connection_specification:
+    type: object
+    $schema: http://json-schema.org/draft-07/schema#
+    required:
+      - api_key
+      - query
+    properties:
+      api_key:
+        type: string
+        description: >-
+          App only Bearer Token. See the <a
+          href="https://developer.twitter.com/en/docs/authentication/oauth-2-0/bearer-tokens">docs</a>
+          for more information on how to obtain this token.
+        title: Access Token
+        airbyte_secret: true
+        order: 0
+      query:
+        type: string
+        description: >-
+          Query for matching Tweets. You can learn how to build this query by
+          reading <a
+          href="https://developer.twitter.com/en/docs/twitter-api/tweets/search/integrate/build-a-query">
+          build a query guide </a>.
+        title: Search Query
+        order: 1
+      start_date:
+        type: string
+        description: >-
+          The start date for retrieving tweets cannot be more than 7 days in the
+          past.
+        title: Start Date
+        format: date-time
+        order: 2
+      end_date:
+        type: string
+        description: >-
+          The end date for retrieving tweets must be a minimum of 10 seconds
+          prior to the request time.
+        title: End Date
+        format: date-time
+        order: 3
+    additionalProperties: true
+
+metadata:
+  autoImportSchema:
+    tweets: false
+  yamlComponents:
+    global:
+      - authenticator
+  testedStreams: {}
+  assist: {}
+
+schemas:
+  tweets:
+    type: object
+    $schema: http://json-schema.org/draft-07/schema#
+    properties:
+      author_id:
+        type:
+          - "null"
+          - string
+      conversation_id:
+        type:
+          - "null"
+          - string
+      created_at:
+        type:
+          - "null"
+          - string
+        format: date-time
+      edit_history_tweet_ids:
+        type:
+          - "null"
+          - array
+        items:
+          type: string
+      id:
+        type:
+          - "null"
+          - string
+      in_reply_to_user_id:
+        type:
+          - "null"
+          - string
+      lang:
+        type:
+          - "null"
+          - string
+      text:
+        type:
+          - "null"
+          - string
+    additionalProperties: true

airbyte-integrations/connectors/source-twitter/main.py

@@ -8,7 +8,7 @@ data:
   connectorSubtype: api
   connectorType: source
   definitionId: d7fd4f40-5e5a-4b8b-918f-a73077f8c131
-  dockerImageTag: 0.1.15
+  dockerImageTag: 0.2.0
   dockerRepository: airbyte/source-twitter
   documentationUrl: https://docs.airbyte.com/integrations/sources/twitter
   githubIssueLabel: source-twitter
@@ -17,7 +17,7 @@ data:
   name: Twitter
   remoteRegistries:
     pypi:
-      enabled: true
+      enabled: false
       packageName: airbyte-source-twitter
   registryOverrides:
     cloud:
@@ -27,8 +27,8 @@ data:
   releaseStage: beta
   supportLevel: community
   tags:
-    - language:python
     - cdk:low-code
+    - language:manifest-only
   connectorTestSuitesOptions:
     - suite: liveTests
       testConnections:
@@ -42,5 +42,5 @@ data:
             type: GSM
             alias: airbyte-connector-testing-secret-store
   connectorBuildOptions:
-    baseImage: docker.io/airbyte/python-connector-base:2.0.0@sha256:c44839ba84406116e8ba68722a0f30e8f6e7056c726f447681bb9e9ece8bd916
+    baseImage: docker.io/airbyte/source-declarative-manifest:4.5.4@sha256:b07a521add11f987c63c0db68c1b57e90bec0c985f1cb6f3c5a1940cde628a70
 metadataSpecVersion: "1.0"