Skip to content

🐛 Source Monday: fetch display value when text is empty #37722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 1, 2024
Merged

🐛 Source Monday: fetch display value when text is empty #37722

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4eed90a
Source Monday: fetch display value when text is empty
davydov-d Apr 30, 2024
122f8a2
bump version and upd changelog
davydov-d Apr 30, 2024
76b34ea
fix format
davydov-d Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-monday/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 80a54ea2-9959-4040-aac1-eee42423ec9b
dockerImageTag: 2.1.1
dockerImageTag: 2.1.2
releases:
breakingChanges:
2.0.0:
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-monday/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
version = "2.1.1"
version = "2.1.2"
name = "source-monday"
description = "Source implementation for Monday."
authors = [ "Airbyte <[email protected]>",]
Expand Down
15 changes: 10 additions & 5 deletions airbyte-integrations/connectors/source-monday/source_monday/extractor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,14 @@ def extract_records(self, response: requests.Response) -> List[Record]:
if not result and self.field_path_incremental:
result = self.try_extract_records(response, self.field_path_incremental)

for item_index in range(len(result)):
if "updated_at" in result[item_index]:
result[item_index]["updated_at_int"] = int(
datetime.strptime(result[item_index]["updated_at"], "%Y-%m-%dT%H:%M:%S%z").timestamp()
)
for record in result:
if "updated_at" in record:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is the cursor value (updated_at) can we take it from the manifest or declare the constant otherwise?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frankly speaking - I'm not sure if this is a cursor value. There has been a change regarding the way we iterate the result, but nothing changed regarding this specific field. If you suggest declaring it as a constant, I'd prefer doing it in a separate PR

record["updated_at_int"] = int(datetime.strptime(record["updated_at"], "%Y-%m-%dT%H:%M:%S%z").timestamp())

column_values = record.get("column_values", [])
for values in column_values:
display_value, text = values.get("display_value"), values.get("text")
if display_value and not text:
values["text"] = display_value

return result
29 changes: 7 additions & 22 deletions airbyte-integrations/connectors/source-monday/source_monday/graphql_requester.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,29 +77,14 @@ def _build_query(self, object_name: str, field_schema: dict, **object_arguments)

arguments = self._get_object_arguments(**object_arguments)
arguments = f"({arguments})" if arguments else ""
fields = ",".join(fields)

# Essentially, we construct a query based on schema properties; however, some fields in the schema are conditional.
# These conditional fields can be obtained by defining them as inline fragments (The docs: https://spec.graphql.org/October2021/#sec-Inline-Fragments).
# This is an example of a query built for the Items stream, with a `display_value` property defined as an `MirrorValue` inline fragment:
# query {
# boards (limit:1) {
# items_page (limit:20) {
# <a field>,
# ...,
# column_values {
# id,
# text,
# type,
# value,
# ... on MirrorValue {display_value}
# }
# }
# }
# }
# When constructing a query, we replace the `display_value` field with the `... on MirrorValue {display_value}` inline fragment.
if object_name == "column_values" and "display_value" in fields:
fields = fields.replace("display_value", "... on MirrorValue{display_value}")
if object_name == "column_values":
fields.remove("display_value")
fields.extend(
["... on MirrorValue{display_value}", "... on BoardRelationValue{display_value}", "... on DependencyValue{display_value}"]
)

fields = ",".join(fields)

if object_name in ["items_page", "next_items_page"]:
query = f"{object_name}{arguments}{{cursor,items{{{fields}}}}}"
Expand Down
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-monday/unit_tests/test_extractor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def test_empty_activity_logs_extract_records():
def test_extract_records_incremental():
# Mock the response
response = MagicMock()
response_body = {"data": {"boards": [{"id": 1}]}}
response_body = {"data": {"boards": [{"id": 1, "column_values": [{"id": 11, "text": None, "display_value": "Hola amigo!"}]}]}}

response.json.return_value = response_body
extractor = MondayIncrementalItemsExtractor(
Expand All @@ -51,4 +51,4 @@ def test_extract_records_incremental():
records = extractor.extract_records(response)

# Assertions
assert records == [{"id": 1}]
assert records == [{"id": 1, "column_values": [{"id": 11, "text": "Hola amigo!", "display_value": "Hola amigo!"}]}]
12 changes: 11 additions & 1 deletion airbyte-integrations/connectors/source-monday/unit_tests/test_graphql_requester.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,12 +145,22 @@ def test_build_items_incremental_query(monday_requester):
field_schema = {
"id": {"type": "integer"},
"name": {"type": "string"},
"column_values": {
"properties": {
"id": {"type": ["null", "string"]},
"text": {"type": ["null", "string"]},
"type": {"type": ["null", "string"]},
"value": {"type": ["null", "string"]},
"display_value": {"type": ["null", "string"]}
}
}
}
stream_slice = {"ids": [1, 2, 3]}

built_query = monday_requester._build_items_incremental_query(object_name, field_schema, stream_slice)

assert built_query == "items(limit:100,ids:[1, 2, 3]){id,name}"
assert built_query == "items(limit:100,ids:[1, 2, 3]){id,name,column_values{id,text,type,value,... on MirrorValue{display_value}," \
"... on BoardRelationValue{display_value},... on DependencyValue{display_value}}}"


def test_get_request_headers(monday_requester):
Expand Down
55 changes: 28 additions & 27 deletions docs/integrations/sources/monday.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,30 +72,31 @@ The Monday connector should not run into Monday API limitations under normal usa

## Changelog

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:----------------------------------------------------------|:------------------------------------------------------------------------|
| 2.1.1 | 2024-04-05 | [36717](https://github.com/airbytehq/airbyte/pull/36717) | Add handling of complexityBudgetExhausted error. |
| 2.1.0 | 2024-04-03 | [36746](https://github.com/airbytehq/airbyte/pull/36746) | Pin airbyte-cdk version to `^0` |
| 2.0.4 | 2024-02-28 | [35696](https://github.com/airbytehq/airbyte/pull/35696) | Fix extraction for `null` value in stream `Activity logs` |
| 2.0.3 | 2024-02-21 | [35506](https://github.com/airbytehq/airbyte/pull/35506) | Support for column values of the mirror type for the `Items` stream. |
| 2.0.2 | 2024-02-12 | [35146](https://github.com/airbytehq/airbyte/pull/35146) | Manage dependencies with Poetry. |
| 2.0.1 | 2024-02-08 | [35016](https://github.com/airbytehq/airbyte/pull/35016) | Migrated to the latest airbyte cdk |
| 2.0.0 | 2024-01-12 | [34108](https://github.com/airbytehq/airbyte/pull/34108) | Migrated to the latest API version: 2024-01 |
| 1.1.4 | 2023-12-13 | [33448](https://github.com/airbytehq/airbyte/pull/33448) | Increase test coverage and migrate to base image |
| 1.1.3 | 2023-09-23 | [30248](https://github.com/airbytehq/airbyte/pull/30248) | Add new field "type" to board stream |
| 1.1.2 | 2023-08-23 | [29777](https://github.com/airbytehq/airbyte/pull/29777) | Add retry for `502` error |
| 1.1.1 | 2023-08-15 | [29429](https://github.com/airbytehq/airbyte/pull/29429) | Ignore `null` records in response |
| 1.1.0 | 2023-07-05 | [27944](https://github.com/airbytehq/airbyte/pull/27944) | Add incremental sync for Items and Boards streams |
| 1.0.0 | 2023-06-20 | [27410](https://github.com/airbytehq/airbyte/pull/27410) | Add new streams: Tags, Workspaces. Add new fields for existing streams. |
| 0.2.6 | 2023-06-12 | [27244](https://github.com/airbytehq/airbyte/pull/27244) | Added http error handling for `403` and `500` HTTP errors |
| 0.2.5 | 2023-05-22 | [225881](https://github.com/airbytehq/airbyte/pull/25881) | Fix pagination for the items stream |
| 0.2.4 | 2023-04-26 | [25277](https://github.com/airbytehq/airbyte/pull/25277) | Increase row limit to 100 |
| 0.2.3 | 2023-03-06 | [23231](https://github.com/airbytehq/airbyte/pull/23231) | Publish using low-code CDK Beta version |
| 0.2.2 | 2023-01-04 | [20996](https://github.com/airbytehq/airbyte/pull/20996) | Fix json schema loader |
| 0.2.1 | 2022-12-15 | [20533](https://github.com/airbytehq/airbyte/pull/20533) | Bump CDK version |
| 0.2.0 | 2022-12-13 | [19586](https://github.com/airbytehq/airbyte/pull/19586) | Migrate to low-code |
| 0.1.4 | 2022-06-06 | [14443](https://github.com/airbytehq/airbyte/pull/14443) | Increase retry_factor for Items stream |
| 0.1.3 | 2021-12-23 | [8172](https://github.com/airbytehq/airbyte/pull/8172) | Add oauth2.0 support |
| 0.1.2 | 2021-12-07 | [8429](https://github.com/airbytehq/airbyte/pull/8429) | Update titles and descriptions |
| 0.1.1 | 2021-11-18 | [8016](https://github.com/airbytehq/airbyte/pull/8016) | 🐛 Source Monday: fix pagination and schema bug |
| 0.1.0 | 2021-11-07 | [7168](https://github.com/airbytehq/airbyte/pull/7168) | 🎉 New Source: Monday |
| Version | Date | Pull Request | Subject |
|:--------|:-----------|:----------------------------------------------------------|:--------------------------------------------------------------------------------------------------|
| 2.1.2 | 2024-04-30 | [37722](https://github.com/airbytehq/airbyte/pull/37722) | Fetch `display_value` field for column values of `Mirror`, `Dependency` and `Connect Board` types |
| 2.1.1 | 2024-04-05 | [36717](https://github.com/airbytehq/airbyte/pull/36717) | Add handling of complexityBudgetExhausted error. |
| 2.1.0 | 2024-04-03 | [36746](https://github.com/airbytehq/airbyte/pull/36746) | Pin airbyte-cdk version to `^0` |
| 2.0.4 | 2024-02-28 | [35696](https://github.com/airbytehq/airbyte/pull/35696) | Fix extraction for `null` value in stream `Activity logs` |
| 2.0.3 | 2024-02-21 | [35506](https://github.com/airbytehq/airbyte/pull/35506) | Support for column values of the mirror type for the `Items` stream. |
| 2.0.2 | 2024-02-12 | [35146](https://github.com/airbytehq/airbyte/pull/35146) | Manage dependencies with Poetry. |
| 2.0.1 | 2024-02-08 | [35016](https://github.com/airbytehq/airbyte/pull/35016) | Migrated to the latest airbyte cdk |
| 2.0.0 | 2024-01-12 | [34108](https://github.com/airbytehq/airbyte/pull/34108) | Migrated to the latest API version: 2024-01 |
| 1.1.4 | 2023-12-13 | [33448](https://github.com/airbytehq/airbyte/pull/33448) | Increase test coverage and migrate to base image |
| 1.1.3 | 2023-09-23 | [30248](https://github.com/airbytehq/airbyte/pull/30248) | Add new field "type" to board stream |
| 1.1.2 | 2023-08-23 | [29777](https://github.com/airbytehq/airbyte/pull/29777) | Add retry for `502` error |
| 1.1.1 | 2023-08-15 | [29429](https://github.com/airbytehq/airbyte/pull/29429) | Ignore `null` records in response |
| 1.1.0 | 2023-07-05 | [27944](https://github.com/airbytehq/airbyte/pull/27944) | Add incremental sync for Items and Boards streams |
| 1.0.0 | 2023-06-20 | [27410](https://github.com/airbytehq/airbyte/pull/27410) | Add new streams: Tags, Workspaces. Add new fields for existing streams. |
| 0.2.6 | 2023-06-12 | [27244](https://github.com/airbytehq/airbyte/pull/27244) | Added http error handling for `403` and `500` HTTP errors |
| 0.2.5 | 2023-05-22 | [225881](https://github.com/airbytehq/airbyte/pull/25881) | Fix pagination for the items stream |
| 0.2.4 | 2023-04-26 | [25277](https://github.com/airbytehq/airbyte/pull/25277) | Increase row limit to 100 |
| 0.2.3 | 2023-03-06 | [23231](https://github.com/airbytehq/airbyte/pull/23231) | Publish using low-code CDK Beta version |
| 0.2.2 | 2023-01-04 | [20996](https://github.com/airbytehq/airbyte/pull/20996) | Fix json schema loader |
| 0.2.1 | 2022-12-15 | [20533](https://github.com/airbytehq/airbyte/pull/20533) | Bump CDK version |
| 0.2.0 | 2022-12-13 | [19586](https://github.com/airbytehq/airbyte/pull/19586) | Migrate to low-code |
| 0.1.4 | 2022-06-06 | [14443](https://github.com/airbytehq/airbyte/pull/14443) | Increase retry_factor for Items stream |
| 0.1.3 | 2021-12-23 | [8172](https://github.com/airbytehq/airbyte/pull/8172) | Add oauth2.0 support |
| 0.1.2 | 2021-12-07 | [8429](https://github.com/airbytehq/airbyte/pull/8429) | Update titles and descriptions |
| 0.1.1 | 2021-11-18 | [8016](https://github.com/airbytehq/airbyte/pull/8016) | 🐛 Source Monday: fix pagination and schema bug |
| 0.1.0 | 2021-11-07 | [7168](https://github.com/airbytehq/airbyte/pull/7168) | 🎉 New Source: Monday |
Loading