Skip to content

Commit cc6c6a4

Browse files
LouisAuneauLouisAuneau
authored
Source PostHog: Fix events stream pagination (#44016)
1 parent 6704217 commit cc6c6a4

File tree

5 files changed

+52
-77
lines changed

5 files changed

+52
-77
lines changed

‎airbyte-integrations/connectors/source-posthog/README.md

+18-42
Original file line numberDiff line numberDiff line change
@@ -7,32 +7,17 @@ For information about how to use this connector within Airbyte, see [the documen
77

88
### Prerequisites
99

10-
**To iterate on this connector, make sure to complete this prerequisites section.**
10+
* Python (`^3.9`)
11+
* Poetry (`^1.7`) - installation instructions [here](https://python-poetry.org/docs/#installation)
1112

12-
#### Build & Activate Virtual Environment and install dependencies
13+
### Installing the connector
1314

14-
From this connector directory, create a virtual environment:
15-
16-
```
17-
python -m venv .venv
18-
```
19-
20-
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
21-
development environment of choice. To activate it from the terminal, run:
22-
23-
```
24-
source .venv/bin/activate
25-
pip install -r requirements.txt
15+
From this connector directory, run:
16+
```bash
17+
poetry install --with dev
2618
```
2719

28-
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
29-
30-
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
31-
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
32-
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
33-
should work as you expect.
34-
35-
#### Create credentials
20+
### Create credentials
3621

3722
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/posthog)
3823
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_posthog/spec.json` file.
@@ -45,31 +30,23 @@ and place them into `secrets/config.json`.
4530
### Locally running the connector
4631

4732
```
48-
python main.py spec
49-
python main.py check --config secrets/config.json
50-
python main.py discover --config secrets/config.json
51-
python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
33+
poetry run source-posthog spec
34+
poetry run source-posthog check --config secrets/config.json
35+
poetry run source-posthog discover --config secrets/config.json
36+
poetry run source-posthog read --config secrets/config.json --catalog sample_files/configured_catalog.json
5237
```
5338

54-
### Locally running the connector docker image
55-
56-
#### Build
57-
58-
**Via [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) (recommended):**
39+
### Building the docker image
5940

41+
1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
42+
2. Run the following command to build the docker image:
6043
```bash
6144
airbyte-ci connectors --name=source-posthog build
6245
```
6346

64-
An image will be built with the tag `airbyte/source-posthog:dev`.
65-
66-
**Via `docker build`:**
47+
An image will be available on your host with the tag `airbyte/source-posthog:dev`.
6748

68-
```bash
69-
docker build -t airbyte/source-posthog:dev .
70-
```
71-
72-
#### Run
49+
### Running as a docker container
7350

7451
Then run any of the connector commands as follows:
7552

@@ -80,10 +57,9 @@ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-posthog:dev discover -
8057
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-posthog:dev read --config /secrets/config.json --catalog /sample_files/configured_catalog.json
8158
```
8259

83-
## Testing
60+
### Running our CI test suite
8461

8562
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
86-
8763
```bash
8864
airbyte-ci connectors --name=source-posthog test
8965
```
@@ -111,4 +87,4 @@ You've checked out the repo, implemented a million dollar feature, and you're re
11187
4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/posthog.md`).
11288
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
11389
6. Pat yourself on the back for being an awesome contributor.
114-
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
90+
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

‎airbyte-integrations/connectors/source-posthog/metadata.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ data:
99
connectorSubtype: api
1010
connectorType: source
1111
definitionId: af6d50ee-dddf-4126-a8ee-7faee990774f
12-
dockerImageTag: 1.1.5
12+
dockerImageTag: 1.1.6
1313
dockerRepository: airbyte/source-posthog
1414
documentationUrl: https://docs.airbyte.com/integrations/sources/posthog
1515
githubIssueLabel: source-posthog

‎airbyte-integrations/connectors/source-posthog/pyproject.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ requires = [ "poetry-core>=1.0.0",]
33
build-backend = "poetry.core.masonry.api"
44

55
[tool.poetry]
6-
version = "1.1.5"
6+
version = "1.1.6"
77
name = "source-posthog"
88
description = "Source implementation for Posthog."
99
authors = [ "Airbyte <[email protected]>",]

‎airbyte-integrations/connectors/source-posthog/source_posthog/manifest.yaml

+4-5
Original file line numberDiff line numberDiff line change
@@ -152,13 +152,12 @@ definitions:
152152
inject_into: "request_parameter"
153153
field_name: "limit"
154154
pagination_strategy:
155-
type: "CursorPagination"
156-
cursor_value: "{{ response['next'] }}"
155+
type: "OffsetIncrement"
157156
page_size: 10000
158157
page_token_option:
159-
type: RequestPath
160-
$parameters:
161-
url_base: "#/definitions/requester/url_base"
158+
type: RequestOption
159+
field_name: "offset"
160+
inject_into: "request_parameter"
162161

163162
streams:
164163
- "#/definitions/projects_stream"

‎docs/integrations/sources/posthog.md

+28-28
Original file line numberDiff line numberDiff line change
@@ -69,31 +69,31 @@ Want to use the PostHog API beyond these limits? Email Posthog at `customers@pos
6969
<details>
7070
<summary>Expand to review</summary>
7171

72-
| Version | Date | Pull Request | Subject |
73-
| :------ | :--------- | :------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------- |
74-
| 1.1.5 | 2024-08-10 | [43488](https://github.com/airbytehq/airbyte/pull/43488) | Update dependencies |
75-
| 1.1.4 | 2024-08-03 | [43232](https://github.com/airbytehq/airbyte/pull/43232) | Update dependencies |
76-
| 1.1.3 | 2024-07-27 | [42769](https://github.com/airbytehq/airbyte/pull/42769) | Update dependencies |
77-
| 1.1.2 | 2024-07-20 | [42151](https://github.com/airbytehq/airbyte/pull/42151) | Update dependencies |
78-
| 1.1.1 | 2024-07-13 | [41823](https://github.com/airbytehq/airbyte/pull/41823) | Update dependencies |
79-
| 1.1.0 | 2024-06-20 | [39763](https://github.com/airbytehq/airbyte/pull/39763) | Add `properties` and `uuid`
80-
attributes to persons stream |
81-
| 1.0.0 | 2023-12-04 | [28593](https://github.com/airbytehq/airbyte/pull/28593) | Fix events.event type |
82-
| 0.1.15 | 2023-10-28 | [31265](https://github.com/airbytehq/airbyte/pull/31265) | Fix Events stream datetime format |
83-
| 0.1.14 | 2023-08-29 | [29947](https://github.com/airbytehq/airbyte/pull/29947) | Add optional field to spec: `events_time_step` |
84-
| 0.1.13 | 2023-07-19 | [28461](https://github.com/airbytehq/airbyte/pull/28461) | Fixed EventsSimpleRetriever declaration |
85-
| 0.1.12 | 2023-06-28 | [27764](https://github.com/airbytehq/airbyte/pull/27764) | Update following state breaking changes |
86-
| 0.1.11 | 2023-06-09 | [27135](https://github.com/airbytehq/airbyte/pull/27135) | Fix custom EventsSimpleRetriever |
87-
| 0.1.10 | 2023-04-15 | [24084](https://github.com/airbytehq/airbyte/pull/24084) | Increase `events` streams batch size |
88-
| 0.1.9 | 2023-02-13 | [22906](https://github.com/airbytehq/airbyte/pull/22906) | Specified date formatting in specification |
89-
| 0.1.8 | 2022-11-11 | [18993](https://github.com/airbytehq/airbyte/pull/18993) | connector migrated to low-code, added projects,insights streams, added project based slicing for all other streams |
90-
| 0.1.7 | 2022-07-26 | [14585](https://github.com/airbytehq/airbyte/pull/14585) | Add missing 'properties' field to event attributes |
91-
| 0.1.6 | 2022-01-20 | [8617](https://github.com/airbytehq/airbyte/pull/8617) | Update connector fields title/description |
92-
| 0.1.5 | 2021-12-24 | [9082](https://github.com/airbytehq/airbyte/pull/9082) | Remove obsolete session_events and insights streams |
93-
| 0.1.4 | 2021-09-14 | [6058](https://github.com/airbytehq/airbyte/pull/6058) | Support self-hosted posthog instances |
94-
| 0.1.3 | 2021-07-20 | [4001](https://github.com/airbytehq/airbyte/pull/4001) | Incremental streams read only relevant pages |
95-
| 0.1.2 | 2021-07-15 | [4692](https://github.com/airbytehq/airbyte/pull/4692) | Use account information for checking the connection |
96-
| 0.1.1 | 2021-07-05 | [4539](https://github.com/airbytehq/airbyte/pull/4539) | Add `AIRBYTE_ENTRYPOINT` env variable for kubernetes support |
97-
| 0.1.0 | 2021-06-08 | [3768](https://github.com/airbytehq/airbyte/pull/3768) | Initial Release |
98-
99-
</details>
72+
| Version | Date | Pull Request | Subject |
73+
| :------ | :--------- | :------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------- |
74+
| 1.1.6 | 2024-08-13 | [44016](https://github.com/airbytehq/airbyte/pull/44016) | Fix `events` stream pagniator to workaround PostHog API issue [#13508](https://github.com/PostHog/posthog/issues/13508) |
75+
| 1.1.5 | 2024-08-10 | [43488](https://github.com/airbytehq/airbyte/pull/43488) | Update dependencies |
76+
| 1.1.4 | 2024-08-03 | [43232](https://github.com/airbytehq/airbyte/pull/43232) | Update dependencies |
77+
| 1.1.3 | 2024-07-27 | [42769](https://github.com/airbytehq/airbyte/pull/42769) | Update dependencies |
78+
| 1.1.2 | 2024-07-20 | [42151](https://github.com/airbytehq/airbyte/pull/42151) | Update dependencies |
79+
| 1.1.1 | 2024-07-13 | [41823](https://github.com/airbytehq/airbyte/pull/41823) | Update dependencies |
80+
| 1.1.0 | 2024-06-20 | [39763](https://github.com/airbytehq/airbyte/pull/39763) | Add `properties` and `uuid` attributes to persons stream |
81+
| 1.0.0 | 2023-12-04 | [28593](https://github.com/airbytehq/airbyte/pull/28593) | Fix events.event type |
82+
| 0.1.15 | 2023-10-28 | [31265](https://github.com/airbytehq/airbyte/pull/31265) | Fix Events stream datetime format |
83+
| 0.1.14 | 2023-08-29 | [29947](https://github.com/airbytehq/airbyte/pull/29947) | Add optional field to spec: `events_time_step` |
84+
| 0.1.13 | 2023-07-19 | [28461](https://github.com/airbytehq/airbyte/pull/28461) | Fixed EventsSimpleRetriever declaration |
85+
| 0.1.12 | 2023-06-28 | [27764](https://github.com/airbytehq/airbyte/pull/27764) | Update following state breaking changes |
86+
| 0.1.11 | 2023-06-09 | [27135](https://github.com/airbytehq/airbyte/pull/27135) | Fix custom EventsSimpleRetriever |
87+
| 0.1.10 | 2023-04-15 | [24084](https://github.com/airbytehq/airbyte/pull/24084) | Increase `events` streams batch size |
88+
| 0.1.9 | 2023-02-13 | [22906](https://github.com/airbytehq/airbyte/pull/22906) | Specified date formatting in specification |
89+
| 0.1.8 | 2022-11-11 | [18993](https://github.com/airbytehq/airbyte/pull/18993) | connector migrated to low-code, added projects,insights streams, added project based slicing for all other streams |
90+
| 0.1.7 | 2022-07-26 | [14585](https://github.com/airbytehq/airbyte/pull/14585) | Add missing 'properties' field to event attributes |
91+
| 0.1.6 | 2022-01-20 | [8617](https://github.com/airbytehq/airbyte/pull/8617) | Update connector fields title/description |
92+
| 0.1.5 | 2021-12-24 | [9082](https://github.com/airbytehq/airbyte/pull/9082) | Remove obsolete session_events and insights streams |
93+
| 0.1.4 | 2021-09-14 | [6058](https://github.com/airbytehq/airbyte/pull/6058) | Support self-hosted posthog instances |
94+
| 0.1.3 | 2021-07-20 | [4001](https://github.com/airbytehq/airbyte/pull/4001) | Incremental streams read only relevant pages |
95+
| 0.1.2 | 2021-07-15 | [4692](https://github.com/airbytehq/airbyte/pull/4692) | Use account information for checking the connection |
96+
| 0.1.1 | 2021-07-05 | [4539](https://github.com/airbytehq/airbyte/pull/4539) | Add `AIRBYTE_ENTRYPOINT` env variable for kubernetes support |
97+
| 0.1.0 | 2021-06-08 | [3768](https://github.com/airbytehq/airbyte/pull/3768) | Initial Release |
98+
99+
</details>

0 commit comments

Comments
 (0)