Paginator does not work as expected on low-code CDK

[jhammarstedt]

Environment

• Airbyte version: example is 0.22.0-alpha

• OS Version / Instance: example macOS,

• Step where error happened: Setup new connection

Current Behavior

When trying to build Postmarkapp with the low code CDK they have the pagination in the params:

Discussed this with @marcosmarxm and we tried running this:

version: "0.1.0"

definitions:
  selector:
    extractor:
      field_pointer: []
  requester:
    url_base: "https://api.postmarkapp.com/"
    http_method: "GET"
    request_options_provider:
      request_body_json:
          Accept: "application/json"
    authenticator:
      type: "ApiKeyAuthenticator"
      header: "X-Postmark-Server-Token"
      api_token: "{{ config['X-Postmark-Server-Token'] }}"
  requester_account:
    url_base: "https://api.postmarkapp.com/"
    http_method: "GET"
    request_options_provider:
      request_body_json:
          Accept: "application/json"
    authenticator:
      type: "ApiKeyAuthenticator"
      header: "X-Postmark-Account-Token"
      api_token: "{{ config['X-Postmark-Account-Token'] }}"
  retriever_account:
    record_selector:
      $ref: "*ref(definitions.selector)"
    paginator:
      type: DefaultPaginator
      $options:
        url_base: "*ref(definitions.requester.url_base)"
      pagination_strategy:
        type: "OffsetIncrement"
        page_size: 500
      page_token_option:
        inject_into: "request_parameter"
        field_name: "offset"
      page_size_option:
        inject_into: "request_parameter"
        field_name: "count"
    requester:
      $ref: "*ref(definitions.requester_account)"
  retriever:
    record_selector:
      $ref: "*ref(definitions.selector)"
    paginator:
      type: DefaultPaginator
      $options:
        url_base: "*ref(definitions.requester.url_base)"
      pagination_strategy:
        type: "OffsetIncrement"
        page_size: 500
      page_token_option:
        inject_into: "request_parameter"
        field_name: "offset"
      page_size_option:
        inject_into: "request_parameter"
        field_name: "count"
    requester:
      $ref: "*ref(definitions.requester)"
  base_stream:
    retriever:
      $ref: "*ref(definitions.retriever)"
  base_stream_account:
    retriever:
      $ref: "*ref(definitions.retriever_account)"
  deliverystats:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "deliverystats"
      primary_key: "Type"
      path: "/deliverystats"
  message-streams:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "message-streams"
      primary_key: "ID"
      path: "/message-streams"
  bounces:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "bounces"
      primary_key: "ID"
      path: "/bounces"
  server:
    $ref: "*ref(definitions.base_stream_account)"
    $options:
      name: "server"
      primary_key: "ID"
      path: "/server"
  servers:
    $ref: "*ref(definitions.base_stream_account)"
    $options:
      name: "servers"
      primary_key: "ID"
      path: "/servers"
  messages:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "messages"
      primary_key: "MessageID"
      path: "/messages/outbound"

streams:
  - "*ref(definitions.deliverystats)"
  - "*ref(definitions.message-streams)"
  - "*ref(definitions.server)"
  #- "*ref(definitions.messages)"
  #- "*ref(definitions.outbound-stats)"
  #- "*ref(definitions.bounces)"
  - "*ref(definitions.servers)"


check:
  stream_names:
    - "servers"

Gave
{"type": "DEBUG", "message": "Receiving response", "data": {"headers": "{'cache-control': 'private', 'content-type': 'application/json; charset=utf-8', 'x-postmark-account': '252847', 'x-postmark-server': '-', 'x-pm-apierrorcode': '600', 'server': 'unicorns-double-rainbow', 'x-powered-by': 'ASP.NET', 'date': 'Wed, 02 Nov 2022 19:49:45 GMT', 'content-length': '81', 'strict-transport-security': 'max-age=16000000;'}", "status": "422", "body": "{\"ErrorCode\":600,\"Message\":\"Parameter 'count' is required but has been left out\"}"}}

We implemented a workaround which can be seen in #18820 but optimally it would have been nice to have the pagination work as expected.

Expected Behavior

The paginator should just insert into the request params and pass through the request

Logs

The full output when running main with --debug can be found here

Steps to Reproduce

1. Create a postmarkapp account
2. Set the connector similar to 🎉New Source: Postmarkapp [low-code cdk] #18820 but change the postmarkapp.yaml to the file above
3. Run main.py check --config secrets/config.json --debug