✨ Source auth0: add new streams `Organizations`, `OrganizationMembers`, `OrganizationMemberRoles` #27500

vasilisgav · 2023-06-20T13:42:54Z

What

Add support for organization related streams in auth0 source connector issue #27499

How

Connector in place had all the scaffolding ready, so adding a new stream was simple enough

🚨 User Impact 🚨

No breaking changes

Pre-merge Actions

New Stream for source Connector auth0

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.2.0
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

github-actions · 2023-06-20T13:43:16Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
The connector tests are passing in CI
You've updated the connector's metadata.yaml file (new!)
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

vasilisgav · 2023-06-20T13:44:04Z

unit tests execution

python -m pytest unit_tests/ -v
Test session starts (platform: darwin, Python 3.9.16, pytest 6.2.5, pytest-sugar 0.9.7)
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors/source-auth0/.hypothesis/examples')
rootdir: /Users/vgavriil/workspace/forks/airbyteAuth0, configfile: pytest.ini
plugins: hypothesis-6.54.6, requests-mock-1.9.3, mock-3.6.1, sugar-0.9.7, timeout-1.4.2, cov-3.0.0
collected 36 items

 airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py::TestAuthentication.test_init_token_authentication_init ✓3% ▍
 airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py::TestAuthentication.test_init_oauth2_authentication_init ✓6% ▋
 airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py::TestAuthentication.test_init_oauth2_authentication_wrong_credentials_record ✓8% ▉
 airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py::TestAuthentication.test_init_oauth2_authentication_wrong_oauth_config_bad_auth_type ✓11% █▎
 airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py::TestAuthentication.test_check_connection_ok ✓14% █▍
 airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py::TestAuthentication.test_check_connection_error_status_code ✓17% █▋
 airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py::TestAuthentication.test_check_connection_error_with_exception ✓19% █▉
 airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py::TestAuthentication.test_check_streams ✓22% ██▎
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_auth0_stream_request_params ✓25% ██▌
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_auth0_stream_parse_response ✓28% ██▊
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_auth0_stream_backoff_time ✓31% ███▏
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_auth0_stream_incremental_request_params ✓33% ███▍
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_incremental_auth0_stream_parse_response ✓36% ███▋
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_incremental_auth0_stream_backoff_time ✓39% ███▉
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_auth0_stream_incremental_backoff_time_empty ✓42% ████▎
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_auth0_stream_incremental_back_off_now ✓44% ████▌
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_auth0_stream_incremental_get_updated_state ✓47% ████▊
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestAuth0Stream.test_auth0_stream_http_method ✓50% █████
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestNextPageToken.test_next_page_token ✓53% █████▍
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestNextPageToken.test_next_page_token_invalid_cursor ✓56% █████▋
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestNextPageToken.test_next_page_token_missing_cursor ✓58% █████▉
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestNextPageToken.test_next_page_token_one_page_only ✓61% ██████▎
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestNextPageToken.test_next_page_token_last_page_incomplete ✓64% ██████▍
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestNextPageToken.test_next_page_token_last_page_complete ✓67% ██████▋
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamUsers.test_stream_users ✓   69% ██████▉
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamUsers.test_users_request_params_out_of_next_page_token ✓72% ███████▎
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamUsers.test_users_source_request_params_have_next_cursor ✓75% ███████▌
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamUsers.test_users_source_parse_response ✓78% ███████▊
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamClients.test_stream_clients ✓81% ████████▏
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamClients.test_clients_source_parse_response ✓83% ████████▍
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamOrganizations.test_stream_organizations ✓86% ████████▋
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamOrganizations.test_organizations_source_parse_response ✓89% ████████▉
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamOrganizationsMembers.test_stream_organizations ✓92% █████████▎
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamOrganizationsMembers.test_organizations_source_parse_response ✓94% █████████▌
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamOrganizationsMemberRoles.test_stream_organizations ✓97% █████████▊
 airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py::TestStreamOrganizationsMemberRoles.test_organizations_source_parse_response ✓100% ██████████
====================================================== warnings summary ======================================================
airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py: 8 warnings
airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py: 34 warnings
  /Users/vgavriil/.pyenv/versions/3.9/envs/airbyte3.9/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:45: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

airbyte-integrations/connectors/source-auth0/unit_tests/test_source.py: 8 warnings
airbyte-integrations/connectors/source-auth0/unit_tests/test_streams.py: 34 warnings
  /Users/vgavriil/.pyenv/versions/3.9/envs/airbyte3.9/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (1.25s):
      36 passed

vasilisgav · 2023-06-20T13:44:52Z

integrations tests execution

sh acceptance-test-docker.sh
Sending build context to Docker daemon  53.76kB
Step 1/17 : FROM python:3.9.13-alpine3.15 as base
 ---> 7811e68d771a
Step 2/17 : FROM base as builder
 ---> 7811e68d771a
Step 3/17 : WORKDIR /airbyte/integration_code
 ---> Using cache
 ---> ec960e3af630
Step 4/17 : RUN apk --no-cache upgrade     && pip install --upgrade pip     && apk --no-cache add tzdata build-base
 ---> Using cache
 ---> 01e45b88dde7
Step 5/17 : COPY setup.py ./
 ---> Using cache
 ---> c1ecd9f7578e
Step 6/17 : RUN pip install --prefix=/install .
 ---> Using cache
 ---> 09f147c9e21a
Step 7/17 : FROM base
 ---> 7811e68d771a
Step 8/17 : WORKDIR /airbyte/integration_code
 ---> Using cache
 ---> ec960e3af630
Step 9/17 : COPY --from=builder /install /usr/local
 ---> Using cache
 ---> 7bafde2291d7
Step 10/17 : COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
 ---> Using cache
 ---> 557a7fab26bf
Step 11/17 : RUN echo "Etc/UTC" > /etc/timezone
 ---> Using cache
 ---> c2be637a3d4b
Step 12/17 : COPY main.py ./
 ---> Using cache
 ---> 54e36006bbc2
Step 13/17 : COPY source_auth0 ./source_auth0
 ---> Using cache
 ---> dddc29e6dd12
Step 14/17 : ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ---> Using cache
 ---> 576cdd9f8f13
Step 15/17 : ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 ---> Using cache
 ---> 1d5290dc93d1
Step 16/17 : LABEL io.airbyte.version=0.3.0
 ---> Using cache
 ---> 369a7f7448fa
Step 17/17 : LABEL io.airbyte.name=airbyte/source-auth0
 ---> Using cache
 ---> 7c9ce0a73dae
Successfully built 7c9ce0a73dae
Successfully tagged airbyte/source-auth0:dev

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
latest: Pulling from airbyte/connector-acceptance-test
Digest: sha256:5c3bfc60ffe61acac0a3b3c701bfdf29b37f162b9dea0b2f59e98ebe1c62572a
Status: Image is up to date for airbyte/connector-acceptance-test:latest
docker.io/airbyte/connector-acceptance-test:latest
============================= test session starts ==============================
platform linux -- Python 3.9.11, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /test_input
plugins: hypothesis-6.54.6, timeout-1.4.2, cov-3.0.0, sugar-0.9.7, requests-mock-1.9.3, mock-3.6.1
collected 41 items

../../test_input/test_core.py .....................s............s..      [ 90%]
../../test_input/test_full_refresh.py
.                                  [ 92%]
../../test_input/test_incremental.py ...                                 [100%]


=========================== short test summary info ============================
SKIPPED [1] ../../actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/connector-acceptance-test/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../../actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/connector-acceptance-test/connector_acceptance_test/tests/test_core.py:695: This tests currently leads to too much failures. We need to fix the connectors at scale first.
================== 39 passed, 2 skipped in 456.37s (0:07:36) ===================

vasilisgav · 2023-06-20T13:52:55Z

Hi @marcosmarxm, tagging you because you reviewed the clients stream in #25985
cheers!

marcosmarxm · 2023-06-26T23:45:30Z

@vasilisgav we're going to take a look later this week sorry the missing return to you

sajarin

Can you bump the version in the metadata.yaml file?

...yte-integrations/connectors/source-auth0/source_auth0/schemas/organization_member_roles.json

…oles

archangelic

Thanks for your contribution! I have submitted some questions and changes to make!

airbyte-integrations/connectors/source-auth0/source_auth0/schemas/clients.json

airbyte-integrations/connectors/source-auth0/source_auth0/schemas/users.json

sajarin · 2023-07-03T20:06:09Z

@vasilisgav can you run the python format command?

./gradlew --no-daemon :airbyte-integrations:connectors:<connector_name>:airbytePythonFormat

vasilisgav · 2023-07-04T10:01:27Z

Hi @sajarin it is actually failing

./gradlew --no-daemon --stacktrace  :airbyte-integrations:connectors:source-auth0:airbytePythonFormat
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.6/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build
Building all of Airbyte.
/Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors
> Task :airbyte-integrations:connectors:source-auth0:checkPython FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:source-auth0:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (https://github.com/xvik/gradle-use-python-plugin/issues). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
> Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':airbyte-integrations:connectors:source-auth0:checkPython'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:142)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:140)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:128)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:69)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:322)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:309)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:302)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:288)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:462)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:379)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
Caused by: org.gradle.api.GradleException: Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (https://github.com/xvik/gradle-use-python-plugin/issues). You can disable virtualenv usage with 'python.scope = USER'.
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.checkPython(CheckPythonTask.groovy:82)
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.switchEnvironment(CheckPythonTask.groovy:166)
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.run(CheckPythonTask.groovy:61)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
        at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:236)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
        at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:221)
        at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:204)
        at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:187)
        at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:165)
        at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:89)
        at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:40)
        at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:53)
        at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:50)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:50)
        at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:40)
        at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:68)
        at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:38)
        at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
        at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
        at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
        at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:51)
        at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:29)
        at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:124)
        at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:80)
        at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:58)
        at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:48)
        at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:36)
        at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:181)
        at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:71)
        at org.gradle.internal.Either$Right.fold(Either.java:175)
        at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
        at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:69)
        at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:47)
        at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
        at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
        at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
        at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:110)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:56)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:56)
        at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:38)
        at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:73)
        at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:44)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
        at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:89)
        at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:50)
        at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:102)
        at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:57)
        at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:76)
        at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:50)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNoEmptySources(SkipEmptyWorkStep.java:254)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:91)
        at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:56)
        at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
        at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
        at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:43)
        at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:31)
        at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:40)
        at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:281)
        at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:40)
        at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
        at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
        at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
        at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:44)
        at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:33)
        at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:76)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:139)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:128)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:69)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:322)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:309)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:302)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:288)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:462)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:379)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command '.venv/bin/python''
        at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241)
        at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:218)
        at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:370)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87)
        at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
        ... 2 more
Caused by: net.rubygrapefruit.platform.NativeException: Could not start '.venv/bin/python'
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
        at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
        at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:98)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:71)
        ... 3 more
Caused by: java.io.IOException: Cannot run program ".venv/bin/python" (in directory "/Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors/source-auth0"): error=2, No such file or directory
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
        ... 6 more
Caused by: java.io.IOException: error=2, No such file or directory
        ... 7 more


* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 47s
11 actionable tasks: 1 executed, 10 up-to-date

But I believe I didn't introduce this error. It must have already been there. Regardless, can you help me tackle this? Cheers!

marcosmarxm · 2023-07-04T11:17:10Z

You must be in the root folder of the project not inside /airbyte-integrations/connectors.

vasilisgav · 2023-07-04T11:45:53Z

Actually I am indeed in the root directory

vgavriil@VG-Work:~/workspace/forks/airbyteAuth0 (Stream-Organizations)$
pwd
/Users/vgavriil/workspace/forks/airbyteAuth0
(airbyte3.9) vgavriil@VG-Work:~/workspace/forks/airbyteAuth0 (Stream-Organizations)$
./gradlew --no-daemon --stacktrace  :airbyte-integrations:connectors:source-auth0:airbytePythonFormat
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.6/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build
Building all of Airbyte.
/Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors
> Task :airbyte-integrations:connectors:source-auth0:checkPython FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:source-auth0:checkPython'.
...
...
...

May be something with python virtual env and gradlew not working so well together

vasilisgav · 2023-07-06T10:55:56Z

Thank you Both, @marcosmarxm and @sajarin I was able to fix the gradle issue
there were some failures (isort and black) which I address in my latest commit. Cheers!

Here is gradlew running successfully:

(airbyte3.9) vgavriil@VG-Work:~/workspace/forks/airbyteAuth0 (Stream-Organizations)$
./gradlew --no-daemon --stacktrace  :airbyte-integrations:connectors:source-auth0:airbytePythonFormat
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.6/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build
Building all of Airbyte.
/Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors

> Task :airbyte-integrations:connectors:source-auth0:checkPython
Using python 3.9.16 from /Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors/source-auth0/.venv (.venv/bin/python)
Using pip 21.3.1 from /Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors/source-auth0/.venv/lib/python3.9/site-packages/pip (python 3.9)

> Task :airbyte-integrations:connectors:source-auth0:pipInstall
[python] .venv/bin/python -m pip install pip==21.3.1
         Looking in indexes: https://pypi.org/simple, https://vgavriilidis:****@pypicloud.herokuapp.com/simple
         Requirement already satisfied: pip==21.3.1 in ./.venv/lib/python3.9/site-packages (21.3.1)
         WARNING: You are using pip version 21.3.1; however, version 23.1.2 is available.
         You should consider upgrading via the '/Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors/source-auth0/.venv/bin/python -m pip install --upgrade pip' command.
[python] .venv/bin/python -m pip list --format=columns
         Package           Version
         ----------------- -------
         attrs             23.1.0
         black             22.3.0
         click             8.1.3
         coverage          6.3.1
         flake8            4.0.1
         iniconfig         2.0.0
         isort             5.6.4
         mccabe            0.6.1
         mypy              0.930
         mypy-extensions   1.0.0
         packaging         23.1
         pathspec          0.11.1
         pip               21.3.1
         platformdirs      3.8.0
         pluggy            1.2.0
         py                1.11.0
         pycodestyle       2.8.0
         pyflakes          2.4.0
         pyproject-flake8  0.0.1a2
         pytest            6.2.5
         setuptools        65.5.0
         toml              0.10.2
         tomli             2.0.1
         typing_extensions 4.7.1
         wheel             0.37.1
         WARNING: You are using pip version 21.3.1; however, version 23.1.2 is available.
         You should consider upgrading via the '/Users/vgavriil/workspace/forks/airbyteAuth0/airbyte-integrations/connectors/source-auth0/.venv/bin/python -m pip install --upgrade pip' command.

> Task :airbyte-integrations:connectors:source-auth0:isortFormat
[python] .venv/bin/python -m isort --settings-file=/Users/vgavriil/workspace/forks/airbyteAuth0/pyproject.toml ./
         Skipped 2 files

> Task :airbyte-integrations:connectors:source-auth0:blackFormat
[python] .venv/bin/python -m black --config /Users/vgavriil/workspace/forks/airbyteAuth0/pyproject.toml ./
         All done! ✨ 🍰 ✨
         6 files left unchanged.

> Task :airbyte-integrations:connectors:source-auth0:flakeCheck
[python] .venv/bin/python -m pflake8 --config /Users/vgavriil/workspace/forks/airbyteAuth0/pyproject.toml ./

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 35s
16 actionable tasks: 5 executed, 11 up-to-date
(airbyte3.9) vgavriil@VG-Work:~/workspace/forks/airbyteAuth0 (Stream-Organizations)$

Can you have another look please?

sajarin · 2023-07-06T18:33:24Z

Thanks @vasilisgav, we just need to set up our integration account properly so that the tests pass on our end. Hope to get this merged soon!

vasilisgav · 2023-07-17T13:19:03Z

Hi @sajarin, @marcosmarxm , any update? I would like to contribute on additional streams, and I am waiting for the merge first.

sajarin · 2023-07-21T07:19:43Z

hey @vasilisgav sorry for the delay here. The connector is still broken on our integration account. We're missing some records on our sandbox account as well for the organization members, once that's updated we can proceed with the merge.

Is this currently blocking you from creating new PRs?

vasilisgav · 2023-07-28T12:36:55Z

@sajarin now that you mention it, not really blocking, I might hit some conflicts though, will solve them when the time comes.

marcosmarxm

Thanks for your contribution @vasilisgav I ran tests and made quick adjustments in #29001 and will get your contribution merged there together with my changes.

vasilisgav · 2023-08-03T13:29:58Z

Thank you @marcosmarxm !

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation community connectors/source/auth0 labels Jun 20, 2023

sajarin suggested changes Jun 27, 2023

View reviewed changes

...yte-integrations/connectors/source-auth0/source_auth0/schemas/organization_member_roles.json Outdated Show resolved Hide resolved

marcosmarxm added the team/tse Technical Support Engineers label Jun 27, 2023

vasilisgav added 3 commits June 28, 2023 10:32

Add new streams Organizations,OrganizationMembers,OrganizationMemberR…

fea3aa1

…oles

relax schema definition to allow additional fields

a6c2ed6

Bump image tag version

d249331

vasilisgav requested a review from sajarin June 28, 2023 07:38

archangelic suggested changes Jun 29, 2023

View reviewed changes

revert some changes to the old schemas

1fb6493

vasilisgav requested a review from archangelic June 30, 2023 15:18

Format python so gradle can pass

2967f59

marcosmarxm assigned sajarin Jul 27, 2023

marcosmarxm mentioned this pull request Aug 2, 2023

Source oauth0: new streams and fix incremental #29001

Merged

marcosmarxm approved these changes Aug 3, 2023

View reviewed changes

marcosmarxm closed this Aug 3, 2023

✨ Source auth0: add new streams Organizations, OrganizationMembers, OrganizationMemberRoles #27500

✨ Source auth0: add new streams Organizations, OrganizationMembers, OrganizationMemberRoles #27500

Uh oh!

Conversation

vasilisgav commented Jun 20, 2023

What

How

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Uh oh!

github-actions bot commented Jun 20, 2023

Before Merging a Connector Pull Request

Uh oh!

vasilisgav commented Jun 20, 2023

Uh oh!

vasilisgav commented Jun 20, 2023

Uh oh!

vasilisgav commented Jun 20, 2023

Uh oh!

marcosmarxm commented Jun 26, 2023

Uh oh!

sajarin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

archangelic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sajarin commented Jul 3, 2023

Uh oh!

vasilisgav commented Jul 4, 2023

Uh oh!

marcosmarxm commented Jul 4, 2023

Uh oh!

vasilisgav commented Jul 4, 2023

Uh oh!

vasilisgav commented Jul 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sajarin commented Jul 6, 2023

Uh oh!

vasilisgav commented Jul 17, 2023

Uh oh!

sajarin commented Jul 21, 2023

Uh oh!

vasilisgav commented Jul 28, 2023

Uh oh!

marcosmarxm left a comment

Choose a reason for hiding this comment

Uh oh!

vasilisgav commented Aug 3, 2023

Uh oh!

Uh oh!

✨ Source auth0: add new streams `Organizations`, `OrganizationMembers`, `OrganizationMemberRoles` #27500

✨ Source auth0: add new streams `Organizations`, `OrganizationMembers`, `OrganizationMemberRoles` #27500

vasilisgav commented Jul 6, 2023 •

edited

Loading