Improvement: Gitlab Source

[adamschmidt]

Tell us about the new connector you’d like to have

Gitlab Source

Describe the context around this modified connector

The Gitlab source connector where you're dealing with one group with no structure and a simple list of repos. The connector support breaks down for larger teams where groups have subgroups etc.

What I would like to do:

• Use a single group-level access key that has appropriate permissions for everything within it
• Be able to leave the groups and projects field of the source config empty so that all groups and projects that are accessible to the key are ingested (i.e. not have to individually nominate each team's groups within the org group)
• Ensure that groups within groups are recursed to ensure repos that are deep in a nested structure are ingested
• Save the source configuration without receiving an error message stating: The connection tests failed. non-json response

Describe the alternative you are considering or using

n/a

Are you willing to submit a PR?

Yes

[briveira]

Still seeing "non-json response" fails when adding a local GITLAB source with hundreds of projects in 0.36.7-alpha (after HOURS trying to add the new gitlab source)

[shogomuranushi]

You may be stuck in an Ingress or load balancer timeout. This may solve the problem.

AWS

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout

GKE

https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features?hl=ja#timeout

[briveira]

doubt it. Gitlab is on premises; airbyte running on my laptop, but thank you anyway (edited: ok, maybe not in aws or gke, but should check gitlab's reverse proxy for errors)