Skip to content

Commit 8974887

Browse files
omri2001GoshaDo
omri2001
and
GoshaDo
authored
feat: Gitlab provider (#27)
* fix: gitlab get scopes added * fix: finished newgitlabClient * fix: ListFiles complete * fix: get File and getFiles * fix: SetStatus + PinkHook still need to do tests on them, not sure if working * fix: set and del webhook done * fix: small change setWebhook * fix: done handlePaylod will now start tests * fix: pr review changes and fixed scopes * fix: first test done * fix: some pr changes resolving * docs: added provider and url to chart * fix: done utils tests * fix: started gitlab tests * fix: added some more tests and fixing bugs * fix: finished tests now runnning qa * fix: added some docs * fix: added some validations and docs * fix: argo ingress * fix: gitlab init ruby script * ci: add gitlab local support * ci: add gitlab local support * ci: fix e2e service account permissions * fix: gitlab setup script * ci: change rules to working branch * ci: start work on actions * fix: cleaning up * fix: context and git provider factory * fix: add context * fix: changes to make it work * ci: add local gitlab at localhost:8080 (#24) * fix: changes to make it work * fix: pipe works, small fixes left * fix: pipe works, small fixes left * test: fixed unit tests * test: running e2e * test: running e2e * test: gitlab e2e check * fix: gitlab unsetting webhook * fix: pr changes * fix: gitlab ruby init script finished * fix: gitlab init change * docs: updated docs for gitlab * fix: gitlab script cleanup * fix: some space * ci: changed running branch for testing * ci: changed order of jobs in e2e * ci: e2e to run on branch * test: gitlab e2e test * test: gitlab e2e test * test: gitlab e2e test * test: gitlab e2e test * test: gitlab e2e test * test: fix gitlab test * fix: gitlab rails script * ci: e2e test revert to main * ci: e2e check on main * ci: parallel e2e jobs * fix: gitlab e2e * fix: e2e tests * docs: align with main * fix: changed e2e * fix: gitlab license as env * fix: gitlab script add sleep * fix: lock gitlab helm version * fix: gitlab script * fix: gitlab ruby script * fix: gitlab ruby script --------- Co-authored-by: goshado <[email protected]> Co-authored-by: GoshaDo <[email protected]>
1 parent fa7656a commit 8974887

27 files changed

+1391
-60
lines changed

‎.github/workflows/e2e.yaml

+130-14
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@ on:
55
branches:
66
- "main"
77
paths:
8-
- '**'
9-
- '!docs/**'
8+
- "**"
9+
- "!docs/**"
1010
pull_request:
1111
branches:
1212
- "main"
@@ -19,8 +19,125 @@ permissions:
1919
contents: read
2020

2121
jobs:
22-
e2e-env-init:
23-
name: E2E Tests (on development)
22+
gitlab-e2e-env:
23+
env:
24+
GITLAB_LICENSE: ${{ secrets.GITLAB_LICENSE }}
25+
name: Gitlab E2E Tests (on development)
26+
runs-on: ubuntu-latest
27+
timeout-minutes: 15
28+
steps:
29+
- uses: actions/checkout@v3
30+
- uses: docker/setup-qemu-action@v2
31+
- uses: docker/setup-buildx-action@v2
32+
with:
33+
driver-opts: network=host
34+
- uses: actions/setup-go@v4
35+
with:
36+
go-version: "1.20"
37+
cache: true
38+
- name: Install kind
39+
run: |
40+
curl -sSLo kind "https://github.com/kubernetes-sigs/kind/releases/download/v0.19.0/kind-linux-amd64"
41+
chmod +x kind
42+
sudo mv kind /usr/local/bin/kind
43+
kind version
44+
- name: Install Kubectl
45+
run: |
46+
curl -sSLO "https://storage.googleapis.com/kubernetes-release/release/v1.26.1/bin/linux/amd64/kubectl"
47+
chmod +x kubectl
48+
sudo mv kubectl /usr/local/bin/kubectl
49+
kubectl version --client --output=yaml
50+
- name: Kubernetes KinD Cluster
51+
run: |
52+
make init-kind
53+
- name: install workflows
54+
run: |
55+
make init-argo-workflows
56+
- name: install gitlab
57+
run: |
58+
tokens=$(make init-gitlab | tail -n1)
59+
GROUP_TOKEN=$(echo "$tokens" | grep -oP "(?<=GROUP_TOKEN )\S+")
60+
echo "GITLAB_TOKEN=$GROUP_TOKEN" >> $GITHUB_ENV
61+
- name: Build Docker Image
62+
uses: docker/build-push-action@v4
63+
with:
64+
context: .
65+
push: true
66+
tags: localhost:5001/piper:latest
67+
cache-from: type=gha
68+
cache-to: type=gha,mode=max
69+
- name: Check tunnel existence
70+
run: |
71+
echo "NGROK_URL=$(cat ~/ngrok.log | grep -o 'url=https://.*' | cut -d '=' -f 2)" >> $GITHUB_ENV
72+
cat ~/ngrok.log | grep -o 'url=https://.*' | cut -d '=' -f 2
73+
- name: init piper
74+
run: |
75+
helm upgrade --install piper ./helm-chart \
76+
-f ./examples/template.values.dev.yaml \
77+
--set piper.gitProvider.name="gitlab" \
78+
--set piper.gitProvider.token="${{ env.GITLAB_TOKEN }}" \
79+
--set piper.gitProvider.url="http://gitlab-webservice-default.gitlab:8080" \
80+
--set piper.gitProvider.webhook.url="http://piper.default/webhook" \
81+
--set piper.gitProvider.webhook.repoList="piper-e2e-test" \
82+
--set piper.gitProvider.organization.name="pied-pipers" \
83+
--set image.repository=localhost:5001 \
84+
--set piper.argoWorkflows.server.address="${{ env.NGROK_URL }}/argo" \
85+
--set-string env\[0\].name=GIT_WEBHOOK_AUTO_CLEANUP,env\[0\].value="true" && \
86+
sleep 20 && kubectl logs deployment/piper
87+
kubectl wait \
88+
--for=condition=ready pod \
89+
--selector=app=piper \
90+
--timeout=60s
91+
- uses: actions/checkout@v3
92+
with:
93+
repository: "quickube/piper-e2e-test"
94+
path: piper-e2e-test
95+
ref: "main"
96+
- name: inject some changes to piper-e2e-test repo
97+
run: |
98+
mkdir ./gitlab
99+
cd ./gitlab
100+
git clone http://oauth2:${{ env.GITLAB_TOKEN }}@localhost:8080/pied-pipers/piper-e2e-test.git
101+
cp -r ../piper-e2e-test/.workflows ./piper-e2e-test/
102+
cd ./piper-e2e-test
103+
git config user.name 'piper-user'
104+
git config user.email '[email protected]'
105+
git add -A
106+
git commit -m "add stuff"
107+
git push
108+
git checkout -b ${{ github.ref_name }}-test
109+
rm ./.workflows/triggers.yaml
110+
cat <<EOF > ./.workflows/triggers.yaml
111+
- events:
112+
- merge_request
113+
- merge_request.open
114+
branches: ["*"]
115+
onStart: ["main.yaml"]
116+
onExit: ["exit.yaml"]
117+
templates: ["templates.yaml"]
118+
EOF
119+
git add -A
120+
git commit -m "${{ github.ref_name }}-test"
121+
git push --set-upstream origin ${{ github.ref_name }}-test -o merge_request.create
122+
- name: Wait for workflow creation
123+
run: |
124+
sleep 10
125+
- name: Check Result
126+
run: |
127+
kubectl logs deployment/piper
128+
kubectl get workflows.argoproj.io -n workflows
129+
BRANCH_VALID_STRING=$(echo ${{ github.ref_name }}-test | tr '[:upper:]' '[:lower:]' | tr '_' '-' | tr -cd 'a-z0-9.\-')
130+
131+
## check if created
132+
RESULT=$(kubectl get workflows.argoproj.io -n workflows --selector=branch=$BRANCH_VALID_STRING --no-headers | grep piper-e2e-test)
133+
[ ! -z "$RESULT" ] && echo "CRD created $RESULT" || { echo "Workflow not exists, existing..."; exit 1; }
134+
135+
## check if status phase not Failed, if yes, show message
136+
RESULT=$(kubectl get workflows.argoproj.io -n workflows --selector=branch=$BRANCH_VALID_STRING --no-headers -o custom-columns="Status:status.phase")
137+
MESSAGE=$(kubectl get workflows.argoproj.io -n workflows --selector=branch=$BRANCH_VALID_STRING --no-headers -o custom-columns="Status:status.message")
138+
[ ! "$RESULT" == "Failed" ] && echo "CRD created $MESSAGE" || { echo "Workflow Failed $MESSAGE, existing..."; exit 1; }
139+
github-e2e-env:
140+
name: Github E2E Tests (on development)
24141
runs-on: ubuntu-latest
25142
timeout-minutes: 15
26143
steps:
@@ -48,7 +165,7 @@ jobs:
48165
chmod +x kind
49166
sudo mv kind /usr/local/bin/kind
50167
kind version
51-
- name: Install Kubectl
168+
- name: Install Kubectl
52169
run: |
53170
curl -sSLO "https://storage.googleapis.com/kubernetes-release/release/v1.26.1/bin/linux/amd64/kubectl"
54171
chmod +x kubectl
@@ -79,24 +196,24 @@ jobs:
79196
run: |
80197
helm upgrade --install piper ./helm-chart \
81198
-f ./examples/template.values.dev.yaml \
199+
--set piper.gitProvider.name="github" \
82200
--set piper.gitProvider.token="${{ secrets.GIT_TOKEN }}" \
83201
--set piper.gitProvider.webhook.url="${{ env.NGROK_URL }}/piper/webhook" \
84202
--set piper.gitProvider.webhook.repoList={piper-e2e-test} \
85203
--set piper.gitProvider.organization.name="quickube" \
86204
--set image.repository=localhost:5001 \
87205
--set piper.argoWorkflows.server.address="${{ env.NGROK_URL }}/argo" \
88-
--set-string env\[0\].name=GIT_WEBHOOK_AUTO_CLEANUP,env\[0\].value="true" \
89-
--set-string rookout.token="${{ secrets.ROOKOUT_TOKEN }}" && \
206+
--set-string env\[0\].name=GIT_WEBHOOK_AUTO_CLEANUP,env\[0\].value="true" && \
90207
sleep 20 && kubectl logs deployment/piper
91-
kubectl wait \
208+
kubectl wait \
92209
--for=condition=ready pod \
93210
--selector=app=piper \
94211
--timeout=60s
95212
- uses: actions/checkout@v3
96213
with:
97-
repository: 'quickube/piper-e2e-test'
214+
repository: "quickube/piper-e2e-test"
98215
path: piper-e2e-test
99-
ref: 'main'
216+
ref: "main"
100217
- name: inject some changes to piper-e2e-test repo
101218
run: |
102219
cd ./piper-e2e-test
@@ -116,26 +233,25 @@ jobs:
116233
- name: Wait for workflow creation
117234
run: |
118235
sleep 10
119-
120236
- name: Close Pull Request
121237
uses: peter-evans/close-pull@v3
122238
with:
123239
token: ${{ secrets.GIT_TOKEN }}
124240
pull-request-number: ${{ steps.cpr.outputs.pull-request-number }}
125-
repository: 'quickube/piper-e2e-test'
241+
repository: "quickube/piper-e2e-test"
126242
comment: Auto-closing pull request
127243
delete-branch: true
128244
- name: Check Result
129245
run: |
130246
kubectl logs deployment/piper
131247
kubectl get workflows.argoproj.io -n workflows
132248
BRANCH_VALID_STRING=$(echo ${{ github.ref_name }}-test | tr '[:upper:]' '[:lower:]' | tr '_' '-' | tr -cd 'a-z0-9.\-')
133-
249+
134250
## check if created
135251
RESULT=$(kubectl get workflows.argoproj.io -n workflows --selector=branch=$BRANCH_VALID_STRING --no-headers | grep piper-e2e-test)
136252
[ ! -z "$RESULT" ] && echo "CRD created $RESULT" || { echo "Workflow not exists, existing..."; exit 1; }
137253
138254
## check if status phase not Failed, if yes, show message
139255
RESULT=$(kubectl get workflows.argoproj.io -n workflows --selector=branch=$BRANCH_VALID_STRING --no-headers -o custom-columns="Status:status.phase")
140256
MESSAGE=$(kubectl get workflows.argoproj.io -n workflows --selector=branch=$BRANCH_VALID_STRING --no-headers -o custom-columns="Status:status.message")
141-
[ ! "$RESULT" == "Failed" ] && echo "CRD created $MESSAGE" || { echo "Workflow Failed $MESSAGE, existing..."; exit 1; }
257+
[ ! "$RESULT" == "Failed" ] && echo "CRD created $MESSAGE" || { echo "Workflow Failed $MESSAGE, existing..."; exit 1; }

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,9 @@ venv.bak/
104104
# mkdocs documentation
105105
/site
106106

107+
#mirrord config
108+
.mirrord/
109+
107110
# mypy
108111
.mypy_cache/
109112
*.iml

‎Dockerfile

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM golang:1.20-alpine3.16 as builder
1+
FROM golang:1.20-alpine3.16 AS builder
22

33
WORKDIR /piper
44

@@ -25,7 +25,7 @@ RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache
2525
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build go build -gcflags='all=-N -l' -tags=alpine -buildvcs=false -trimpath ./cmd/piper
2626

2727

28-
FROM alpine:3.16 as piper-release
28+
FROM alpine:3.16 AS piper-release
2929

3030
ENV GIN_MODE=release
3131

‎cmd/piper/piper.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -53,4 +53,4 @@ func main() {
5353
defer stop()
5454
event_handler.Start(ctx, stop, cfg, globalClients)
5555
server.Start(ctx, stop, cfg, globalClients)
56-
}
56+
}

‎docs/configuration/environment_variables.md

+6-3
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,16 @@ The helm chart populates them using [values.yaml](https://github.com/quickube/pi
55

66
### Git
77

8-
* GIT_PROVIDER
9-
The git provider that Piper will use, possible variables: GitHub . We plan to support Bitbucket and GitLab, as well.
8+
- GIT_PROVIDER
9+
The git provider that Piper will use, possible variables: GitHub | Gitlab | Bitbucket
1010

1111
* GIT_TOKEN
1212
The git token that will be used to connect to the git provider.
1313

14-
* GIT_ORG_NAME
14+
- GIT_URL
15+
the git url that will be used, only relevant when running gitlab self hosted
16+
17+
- GIT_ORG_NAME
1518
The organization name.
1619

1720
* GIT_ORG_LEVEL_WEBHOOK

‎docs/configuration/health_check.md

+2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
## Health Check
22

3+
currently not supported for gitlab / bitbucket
4+
35
The following examples shows a health check being executed every 1 minute as configured in the helm chart under `livenessProbe`, and triggered by `/healthz` endpoint:
46

57
```yaml

‎docs/getting_started/installation.md

+5-4
Original file line numberDiff line numberDiff line change
@@ -36,13 +36,14 @@ Piper will use git to fetch the `.workflows` folder and receive events using web
3636

3737
To pick which git provider you are using provide `gitProvider.name` configuration in helm chart (Currently we only support GitHub and Bitbucket).
3838

39-
You must also configure your organization (GitHub) or workspace (Bitbucket) name using `gitProvider.organization.name` in the helm chart.
39+
Also configure you organization (Github), workspace (Bitbucket) or group (Gitlab) name using `gitProvider.organization.name` in helm chart.
4040

4141
#### Git Token Permissions
4242

43-
The token should have access for creating webhooks and read repositories content.
44-
For GitHub, configure `admin:org` and `write:org` permissions in Classic Token.
45-
For Bitbucket, configure `Repositories:read`, `Webhooks:read and write` and `Pull requests:read` permissions (for multiple repos use workspace token).
43+
The token should have access for creating webhooks and read repositories content.</br>
44+
<b>For GitHub</b>, configure `admin:org` and `write:org` permissions in Classic Token. </br>
45+
<b>For Bitbucket</b>, configure `Repositories:read`, `Webhooks:read and write` and `Pull requests:read` permissions (for multiple repos use workspace token). </br>
46+
<b>For Gitlab</b>, configure `read_api`, `write_repository` and `api` (for multiple repos use group token with owner role). </br>
4647

4748
#### Token
4849

‎examples/template.values.dev.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
piper:
22
gitProvider:
3-
name: github
3+
name: "" # github/bitbucket/gitlab | env: GIT_PROVIDER
44
token: "GIT_TOKEN"
55
organization:
66
name: "ORG_NAME"

‎gitlab.values.yaml

+16-3
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,33 @@
11
gitlab:
22
toolbox:
3-
enabled: false
3+
enabled: true
4+
extraVolumes: |-
5+
- name: piper-config
6+
configMap:
7+
name: piper-setup
8+
extraVolumeMounts: |-
9+
- mountPath: /tmp/scripts/piper-setup.rb
10+
name: piper-config
11+
subPath: piper-setup.rb
12+
readOnly: true
413
gitlab-shell:
5-
enabled: false
14+
enabled: true
615
gitlab-pages:
716
enabled: false
817
gitlab-exporter:
918
enabled: false
1019
kas:
1120
minReplicas: 1
1221
webservice:
22+
enabled: true
1323
minReplicas: 1
1424
ingress:
1525
requireBasePath: false
1626
global:
27+
gitlab:
28+
license:
29+
key: license_key
30+
secret: gitlab-license
1731
hosts:
1832
domain: localhost
1933
https: false
@@ -38,7 +52,6 @@ prometheus:
3852
certmanager:
3953
installCRDs: false
4054
install: false
41-
4255
nginx-ingress:
4356
controller:
4457
ingressClassResource:

‎go.mod

+5-2
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ require (
1313
github.com/ktrysmt/go-bitbucket v0.9.66
1414
github.com/stretchr/testify v1.8.4
1515
github.com/tidwall/gjson v1.16.0
16+
github.com/xanzy/go-gitlab v0.113.0
1617
golang.org/x/net v0.17.0
1718
gopkg.in/yaml.v3 v3.0.1
1819
k8s.io/apimachinery v0.24.3
@@ -45,6 +46,8 @@ require (
4546
github.com/google/uuid v1.3.0 // indirect
4647
github.com/gorilla/websocket v1.5.0 // indirect
4748
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
49+
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
50+
github.com/hashicorp/go-retryablehttp v0.7.7 // indirect
4851
github.com/hashicorp/golang-lru v0.5.4 // indirect
4952
github.com/imdario/mergo v0.3.13 // indirect
5053
github.com/josharian/intern v1.0.0 // indirect
@@ -53,7 +56,7 @@ require (
5356
github.com/kr/pretty v0.3.1 // indirect
5457
github.com/leodido/go-urn v1.2.4 // indirect
5558
github.com/mailru/easyjson v0.7.7 // indirect
56-
github.com/mattn/go-isatty v0.0.19 // indirect
59+
github.com/mattn/go-isatty v0.0.20 // indirect
5760
github.com/mitchellh/mapstructure v1.5.0 // indirect
5861
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
5962
github.com/modern-go/reflect2 v1.0.2 // indirect
@@ -71,7 +74,7 @@ require (
7174
golang.org/x/arch v0.3.0 // indirect
7275
golang.org/x/crypto v0.17.0 // indirect
7376
golang.org/x/oauth2 v0.11.0 // indirect
74-
golang.org/x/sys v0.15.0 // indirect
77+
golang.org/x/sys v0.20.0 // indirect
7578
golang.org/x/term v0.15.0 // indirect
7679
golang.org/x/text v0.14.0 // indirect
7780
golang.org/x/time v0.3.0 // indirect

0 commit comments

Comments
 (0)