Gitea support

[hulto]

Hey!
I just started playing with Garm and I'm really enjoying it!
I'd like to use it with my gitea server as well as some GitHub repos.

Are there plans to add support for Gitea? If not is that something I can contribute?

[gabriel-samfira]

Hi @hulto!

As long as Gitea/Forgejo supports:

• ephemeral runners
• registration tokens/JIT runners
• can send webhooks with job runs so we can schedule runners to a pool
• Has a way to allow us to list compatible agent downloads for linux/windows/mac/etc

It should be possible. However, I think there is an inherent danger in trying to support 2 different forges in the same project, as they may deviate significantly in the future and it may become cumbersome or downright impossible to consolidate. It may be wise to attempt to rearchitect parts of GARM and split off the bits that deal with the interaction between GARM and the forge. Similar to how providers work. But that would be a huge undertaking.

In the short term, if gitea has the above mentioned features, an attempt could be made to support it, if you have the time. Up to this point the focus has been to support GitHub and GHES. So the code mostly revolves around that assumption. But I think some thin abstractions could make gitea fit.

My experience with gitea is limited, so I am not sure how much I could help, but if I can, I will. If you want to tackle this, I suggest creating a proof of concept, hacked with an axe, just to see if it's feasible. Something that won't take months of your time, just to find that we can't merge it.

If having the two forges coexist does not seem possible, another option would be to fork GARM, yank out github and replace it with gitea. The we can merge back if/when the architecture of GARM permits pluggable forges, as we could then separately maintain plugins for gitea, github, etc.

So, plenty of options 😊.

[ChristopherHX]

I got so far with gitea + garm that the jobs are listed. By patching random stuff on both gitea and garm.

~/garm/garm-src$ ./bin/garm-cli job list
+----+------+-----------+------------+-------------+------------+------------------+-----------+
| ID | NAME | STATUS    | CONCLUSION | RUNNER NAME | REPOSITORY | REQUESTED LABELS | LOCKED BY |
+----+------+-----------+------------+-------------+------------+------------------+-----------+
|  5 | _    | queued    |            |             | test/test  | ubuntu           |           |
+----+------+-----------+------------+-------------+------------+------------------+-----------+
|  6 | _    | completed | cancelled  |             | test/test  | ubuntu           |           |
+----+------+-----------+------------+-------------+------------+------------------+-----------+

My problems are dropped error messages due to security.

• Gitea Webhooks don't send any webhook installation level information e.g. repository, organization (user and global) (enterprise not available)

Does garm only try to create runners if it could fetch runner installation packages?

• runner package download is 404 due to
  • not beeing implemented in gitea
  • go-github is configured via WithEnterpriseURLs this leads to weirdness of that method
    • GET http://rpi2server.fritz.box:3044/api/v1/api/v3/repos/test/test/actions/runners/downloads: 404, manually setting these exported fields is needed for gitea

[gabriel-samfira]

Hi @ChristopherHX

Adding Gitea will be a bit more involved. There are a few assumptions in GARM that are related to GitHub. Each entity (Repo, Org, Enterprise) has a "pool manager" (needs to be reworked). That pool manager makes sure that the runners in each pool are properly created, deleted, etc. To ensure that GARM itselt has valid credentials to manage resources inside an entity, we periodically fetch the tools. We need them anyway, and in some cases, tools have a bearer token we need to ass to be able to download them (GHES). So checking for tools has a dual purpose:

• Validate credentials
• keep the tools object in cache for the entity

For Gitea we don't have enterprises, which is fine. But if Gitea doesn't have "organizations", only repos and global, we need to create a new entity type.

We also need to determine how webhooks work in Gitea. For example, in GitHub we can have an enterprise which has multiple orgs which in turn have multiple repos. I can install GARM at all 3 levels. If we have repo "testrepo" in org "testorg" which is part of "testenterprise" and I have GARM installed on all 3 levels, then whenever "testrepo" triggers a workflow, all 3 will send a webhook to GARM for the same job.

Does Gitea have a similar behavior? If so, we need a way to determine who triggered that webhook, so we know which pools are entitled to react and create a runner.

To get this to work we will need to:

• Abstract the forge specific code that deals with webhooks
• After determining the forge type, we need to:
  • Determine which entity types are available for this implementation
  • Save the job in the db and associate it with the entity
• Create a new credentials type for the new forge type. Github has apps and PATs. Something similar needs to exist in Gitea.
• Determine if JIT runners is officially supported, if not, this bit will be hidden away in the pool manager implementaion.
• Abstract away the pool manager to consume jobs and create runners associated with a specific pool
  • Based on the origin forge, we need to generate proper userdata that sets up the runner and registers it in the forge

There are probably more points to consider, but if we are to add Gitea support, I need to cleanup the code a bit and remove some assumptions that are tightly coupled to GitHub. I would like to do this anyway, because I want us to be able to support Gitea as well as Forgejo. Porentially others in the future.

It might take a while though. In the meantime if you'd like to create a PoC, do not let me discourage you 🙂 .

[ChristopherHX]

Gitea has

• System/Global Runners (/admin/runners/registration-token) Gitea Only (replacement of enterprise)
• User Runners (/user/actions/runners/registration-token) Gitea Only
• Organization Runners (/orgs/{org}/actions/runners/registration-token) GitHub Compatible
• Repo Runners (/repos/{owner}/{repo}/actions/runners/registration-token) GitHub Compatible <<<--- I tried this in the POC.

Abstract the forge specific code that deals with webhooks

For me this is only now this missing header then Gitea is able to use the GitHub code of garm

garm/apiserver/controllers/controllers.go

Line 108 in 99f5660

hookType := r.Header.Get("X-Github-Hook-Installation-Target-Type")

X-Github-Hook-Installation-Target-Type, Gitea send the other X-Github headers already.

I hardcoded repository in the POC.

Create a new credentials type for the new forge type. Github has apps and PATs. Something similar needs to exist in Gitea.

Not needed at first, I used PAT like GitHub. Gitea does it very good to behave like the GitHub Rest Api. Oidc etc. are enhancements.

e.g. the runner registration token retrieval is 100% the same if the GHES config does not append /api/v3 to a /api/v1 path.

Determine if JIT runners is officially supported, if not, this bit will be hidden away in the pool manager implementaion.

Currently no.

At first I just thought this is straight forward (by just needed to provide a custom runner setup template), but yes this is sightly more complex, I come back to this topic in March.

This seems to be a used endpoint as well
failed to fetch github runners: fetching runners: GET http://rpi2server.fritz.box:3044/api/v1/repos/test/test/actions/runners?per_page=100

DeleteRunner not available as well

[ChristopherHX]

So checking for tools has a dual purpose:

• Validate credentials
• keep the tools object in cache for the entity

Yes it really blocked progress, so the lxd pluginhas created runners for me by just returning the json from GitHub docs from Gitea for runner downloads. (at least until this feature is replicated, could also be skipped in a garm POC).

• repository github api ok for my gitea test branch and patched garm to work with /api/v1 instead of /api/v3
• get registration token ok (auto fallback, because jit tokens not supported, can take a while for me to add that stuff have some pending PRs to Gitea at this time)
• start lxd container ok
• track job status in garm ok
• authentication via PAT ok

I omited the code I used, so ping me if you want the information. Added Gitea as repository level GHES instance.

Now I'am at step adding an extra spec to download and setup act_runner instead of GitHub's runner (while I have a way to proxy GitHub runners to gitea e.g. binary compatible, this is not my goal right now)

For arm64 systems only.... due to act_runner url

garm-cli pool update --extra-specs='{"enable_boot_debug": true,"runner_install_template":"IyEvYmluL2Jhc2gKCnNldCAtZQpzZXQgLW8gcGlwZWZhaWwKCnt7LSBpZiAuRW5hYmxlQm9vdERlYnVnIH19CnNldCAteAp7ey0gZW5kIH19CgpDQUxMQkFDS19VUkw9Int7IC5DYWxsYmFja1VSTCB9fSIKTUVUQURBVEFfVVJMPSJ7eyAuTWV0YWRhdGFVUkwgfX0iCkJFQVJFUl9UT0tFTj0ie3sgLkNhbGxiYWNrVG9rZW4gfX0iCgpSVU5fSE9NRT0iL2hvbWUve3sgLlJ1bm5lclVzZXJuYW1lIH19L2FjdGlvbnMtcnVubmVyIgoKaWYgWyAteiAiJE1FVEFEQVRBX1VSTCIgXTt0aGVuCgllY2hvICJubyB0b2tlbiBpcyBhdmFpbGFibGUgYW5kIE1FVEFEQVRBX1VSTCBpcyBub3Qgc2V0IgoJZXhpdCAxCmZpCgpmdW5jdGlvbiBjYWxsKCkgewoJUEFZTE9BRD0iJDEiCglbWyAkQ0FMTEJBQ0tfVVJMID1+IF4oLiopL3N0YXR1cygvKT8kIF1dIHx8IENBTExCQUNLX1VSTD0iJHtDQUxMQkFDS19VUkx9L3N0YXR1cyIKCWN1cmwgLS1yZXRyeSA1IC0tcmV0cnktZGVsYXkgNSAtLXJldHJ5LWNvbm5yZWZ1c2VkIC0tZmFpbCAtcyAtWCBQT1NUIC1kICIke1BBWUxPQUR9IiAtSCAnQWNjZXB0OiBhcHBsaWNhdGlvbi9qc29uJyAtSCAiQXV0aG9yaXphdGlvbjogQmVhcmVyICR7QkVBUkVSX1RPS0VOfSIgIiR7Q0FMTEJBQ0tfVVJMfSIgfHwgZWNobyAiZmFpbGVkIHRvIGNhbGwgaG9tZTogZXhpdCBjb2RlICgkPykiCn0KCmZ1bmN0aW9uIHN5c3RlbUluZm8oKSB7CglpZiBbIC1mICIvZXRjL29zLXJlbGVhc2UiIF07dGhlbgoJCS4gL2V0Yy9vcy1yZWxlYXNlCglmaQoJT1NfTkFNRT0ke05BTUU6LSIifQoJT1NfVkVSU0lPTj0ke1ZFUlNJT05fSUQ6LSIifQoJQUdFTlRfSUQ9JHsxOi1udWxsfQoJIyBzdHJpcCBzdGF0dXMgZnJvbSB0aGUgY2FsbGJhY2sgdXJsCglbWyAkQ0FMTEJBQ0tfVVJMID1+IF4oLiopL3N0YXR1cygvKT8kIF1dICYmIENBTExCQUNLX1VSTD0iJHtCQVNIX1JFTUFUQ0hbMV19IiB8fCB0cnVlCglTWVNJTkZPX1VSTD0iJHtDQUxMQkFDS19VUkx9L3N5c3RlbS1pbmZvLyIKCVBBWUxPQUQ9IntcIm9zX25hbWVcIjogXCIkT1NfTkFNRVwiLCBcIm9zX3ZlcnNpb25cIjogXCIkT1NfVkVSU0lPTlwiLCBcImFnZW50X2lkXCI6ICRBR0VOVF9JRH0iCgljdXJsIC0tcmV0cnkgNSAtLXJldHJ5LWRlbGF5IDUgLS1yZXRyeS1jb25ucmVmdXNlZCAtLWZhaWwgLXMgLVggUE9TVCAtZCAiJHtQQVlMT0FEfSIgLUggJ0FjY2VwdDogYXBwbGljYXRpb24vanNvbicgLUggIkF1dGhvcml6YXRpb246IEJlYXJlciAke0JFQVJFUl9UT0tFTn0iICIke1NZU0lORk9fVVJMfSIgfHwgdHJ1ZQp9CgpmdW5jdGlvbiBzZW5kU3RhdHVzKCkgewoJTVNHPSIkMSIKCWNhbGwgIntcInN0YXR1c1wiOiBcImluc3RhbGxpbmdcIiwgXCJtZXNzYWdlXCI6IFwiJE1TR1wifSIKfQoKZnVuY3Rpb24gc3VjY2VzcygpIHsKCU1TRz0iJDEiCglJRD0kezI6LW51bGx9CgljYWxsICJ7XCJzdGF0dXNcIjogXCJpZGxlXCIsIFwibWVzc2FnZVwiOiBcIiRNU0dcIiwgXCJhZ2VudF9pZFwiOiAkSUR9Igp9CgpmdW5jdGlvbiBmYWlsKCkgewoJTVNHPSIkMSIKCWNhbGwgIntcInN0YXR1c1wiOiBcImZhaWxlZFwiLCBcIm1lc3NhZ2VcIjogXCIkTVNHXCJ9IgoJZXhpdCAxCn0KCmZ1bmN0aW9uIGRvd25sb2FkQW5kRXh0cmFjdFJ1bm5lcigpIHsKCXNlbmRTdGF0dXMgImRvd25sb2FkaW5nIHRvb2xzIGZyb20gaHR0cHM6Ly9kbC5naXRlYS5jb20vYWN0X3J1bm5lci8wLjIuMTEvYWN0X3J1bm5lci0wLjIuMTEtbGludXgtYXJtNjQiCglta2RpciAtcCAiJFJVTl9IT01FIiB8fCBmYWlsICJmYWlsZWQgdG8gY3JlYXRlIGFjdGlvbnMtcnVubmVyIGZvbGRlciIKCWN1cmwgLS1yZXRyeSA1IC0tcmV0cnktZGVsYXkgNSAtLXJldHJ5LWNvbm5yZWZ1c2VkIC0tZmFpbCAtTCAtbyAiJFJVTl9IT01FL2FjdF9ydW5uZXIiICJodHRwczovL2RsLmdpdGVhLmNvbS9hY3RfcnVubmVyLzAuMi4xMS9hY3RfcnVubmVyLTAuMi4xMS1saW51eC1hcm02NCIgfHwgZmFpbCAiZmFpbGVkIHRvIGRvd25sb2FkIHRvb2xzIgogICAgc3VkbyBjaG93biB7eyAuUnVubmVyVXNlcm5hbWUgfX06e3sgLlJ1bm5lckdyb3VwIH19IC1SIC9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fSB8fCBmYWlsICJmYWlsZWQgdG8gY2hhbmdlIG93bmVyIgogICAgc3VkbyBjaG1vZCAreCAiJFJVTl9IT01FL2FjdF9ydW5uZXIiIHx8IGZhaWwgImZhaWxlZCB0byBjaGFuZ2UgcGVybWlzc2lvbnMiCiAgICBzdWRvIHRlZSAiJFJVTl9IT01FL2FjdF9ydW5uZXIuc2VydmljZSIgPDxFT0YKW1VuaXRdCkRlc2NyaXB0aW9uPWFjdF9ydW5uZXIKQWZ0ZXI9bmV0d29yay50YXJnZXQKCltTZXJ2aWNlXQpVc2VyPXt7IC5SdW5uZXJVc2VybmFtZSB9fQpFbnZpcm9ubWVudD1IT01FPS9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fQpFeGVjU3RhcnQ9JFJVTl9IT01FL2FjdF9ydW5uZXIgZGFlbW9uCldvcmtpbmdEaXJlY3Rvcnk9JFJVTl9IT01FCktpbGxNb2RlPXByb2Nlc3MKS2lsbFNpZ25hbD1TSUdJTlQKVGltZW91dFN0b3BTZWM9NW1pbgoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0CkVPRgogICAgc3VkbyBzeXN0ZW1jdGwgZW5hYmxlICRSVU5fSE9NRS9hY3RfcnVubmVyLnNlcnZpY2UKfQoKaWYgWyAhIC1kICIkUlVOX0hPTUUiIF07dGhlbgoJZG93bmxvYWRBbmRFeHRyYWN0UnVubmVyCgljZCAiJFJVTl9IT01FIgplbHNlCglzZW5kU3RhdHVzICJ1c2luZyBjYWNoZWQgcnVubmVyIGZvdW5kIGluICRSVU5fSE9NRSIKCWNkICIkUlVOX0hPTUUiCmZpCgoKc2VuZFN0YXR1cyAiY29uZmlndXJpbmcgcnVubmVyIgp7ey0gaWYgLlVzZUpJVENvbmZpZyB9fQp7ey0gZWxzZX19CgpHSVRIVUJfVE9LRU49JChjdXJsIC0tcmV0cnkgNSAtLXJldHJ5LWRlbGF5IDUgLS1yZXRyeS1jb25ucmVmdXNlZCAtLWZhaWwgLXMgLVggR0VUIC1IICdBY2NlcHQ6IGFwcGxpY2F0aW9uL2pzb24nIC1IICJBdXRob3JpemF0aW9uOiBCZWFyZXIgJHtCRUFSRVJfVE9LRU59IiAiJHtNRVRBREFUQV9VUkx9L3J1bm5lci1yZWdpc3RyYXRpb24tdG9rZW4vIikKCnNldCArZQphdHRlbXB0PTEKd2hpbGUgdHJ1ZTsgZG8KCUVSUk9VVD0kKG1rdGVtcCkKCXt7LSBpZiAuR2l0SHViUnVubmVyR3JvdXAgfX0KCXt7LSBlbHNlfX0KCiAgICAjIC0tZXBoZW1lcmFsIG9uY2Ugc3VwcG9ydGVkCiAgICAjICJ7eyAuUnVubmVyTGFiZWxzIH19IgogICAgLi9hY3RfcnVubmVyIHJlZ2lzdGVyIC0tbm8taW50ZXJhY3RpdmUgLS1pbnN0YW5jZSAiJChlY2hvICJ7eyAuUmVwb1VSTCB9fSIgfCBjdXQgLWQnLycgLWYxLTMpIiAtLXRva2VuICIkR0lUSFVCX1RPS0VOIiAtLW5hbWUgInt7IC5SdW5uZXJOYW1lIH19IiAtLWxhYmVscyAidWJ1bnR1LGdlbmVyaWMiIDI+JEVSUk9VVAoJe3stIGVuZH19CglpZiBbICQ/IC1lcSAwIF07IHRoZW4KCQlybSAkRVJST1VUIHx8IHRydWUKCQlzZW5kU3RhdHVzICJydW5uZXIgc3VjY2Vzc2Z1bGx5IGNvbmZpZ3VyZWQgYWZ0ZXIgJGF0dGVtcHQgYXR0ZW1wdChzKSIKCQlicmVhawoJZmkKCUxBU1RfRVJSPSQoY2F0ICRFUlJPVVQpCgllY2hvICIkTEFTVF9FUlIiCgoJaWYgWyAkYXR0ZW1wdCAtZ3QgNSBdO3RoZW4KCQlybSAkRVJST1VUIHx8IHRydWUKCQlmYWlsICJmYWlsZWQgdG8gY29uZmlndXJlIHJ1bm5lcjogJExBU1RfRVJSIgoJZmkKCglzZW5kU3RhdHVzICJmYWlsZWQgdG8gY29uZmlndXJlIHJ1bm5lciAoYXR0ZW1wdCAkYXR0ZW1wdCk6ICRMQVNUX0VSUiAocmV0cnlpbmcgaW4gNSBzZWNvbmRzKSIKCWF0dGVtcHQ9JCgoYXR0ZW1wdCsxKSkKCXJtICRFUlJPVVQgfHwgdHJ1ZQoJc2xlZXAgNQpkb25lCnNldCAtZQp7ey0gZW5kfX0KCmlmIFsgLWUgIi9zeXMvZnMvc2VsaW51eCIgXTt0aGVuCglzdWRvIGNoY29uIC1SIC1oIHVzZXJfdTpvYmplY3RfcjpiaW5fdDpzMCAvaG9tZS9ydW5uZXIvIHx8IGZhaWwgImZhaWxlZCB0byBjaGFuZ2Ugc2VsaW51eCBjb250ZXh0IgpmaQoKQUdFTlRfSUQ9IiIKe3stIGlmIC5Vc2VKSVRDb25maWcgfX0Ke3stIGVsc2V9fQpzdWRvIHN5c3RlbWN0bCBzdGFydCBhY3RfcnVubmVyIHx8IGZhaWwgImZhaWxlZCB0byBzdGFydCBzZXJ2aWNlIgp7ey0gZW5kfX0Kc3lzdGVtSW5mbyAkQUdFTlRfSUQKc3VjY2VzcyAicnVubmVyIHN1Y2Nlc3NmdWxseSBpbnN0YWxsZWQiICRBR0VOVF9JRAo="}'  <POOL_ID>

Debugging tips for the runner setup of the lxd garm backend would help me a lot.

runnerinstalltemplate gitea prototype label ubuntu,generic hardcoded in POC (the label using : cause register to fail with exit code 0 :( )

For arm64 systems only.... due to act_runner url

#!/bin/bash

set -e
set -o pipefail

{{- if .EnableBootDebug }}
set -x
{{- end }}

CALLBACK_URL="{{ .CallbackURL }}"
METADATA_URL="{{ .MetadataURL }}"
BEARER_TOKEN="{{ .CallbackToken }}"

RUN_HOME="/home/{{ .RunnerUsername }}/actions-runner"

if [ -z "$METADATA_URL" ];then
	echo "no token is available and METADATA_URL is not set"
	exit 1
fi

function call() {
	PAYLOAD="$1"
	[[ $CALLBACK_URL =~ ^(.*)/status(/)?$ ]] || CALLBACK_URL="${CALLBACK_URL}/status"
	curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d "${PAYLOAD}" -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "${CALLBACK_URL}" || echo "failed to call home: exit code ($?)"
}

function systemInfo() {
	if [ -f "/etc/os-release" ];then
		. /etc/os-release
	fi
	OS_NAME=${NAME:-""}
	OS_VERSION=${VERSION_ID:-""}
	AGENT_ID=${1:-null}
	# strip status from the callback url
	[[ $CALLBACK_URL =~ ^(.*)/status(/)?$ ]] && CALLBACK_URL="${BASH_REMATCH[1]}" || true
	SYSINFO_URL="${CALLBACK_URL}/system-info/"
	PAYLOAD="{\"os_name\": \"$OS_NAME\", \"os_version\": \"$OS_VERSION\", \"agent_id\": $AGENT_ID}"
	curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d "${PAYLOAD}" -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "${SYSINFO_URL}" || true
}

function sendStatus() {
	MSG="$1"
	call "{\"status\": \"installing\", \"message\": \"$MSG\"}"
}

function success() {
	MSG="$1"
	ID=${2:-null}
	call "{\"status\": \"idle\", \"message\": \"$MSG\", \"agent_id\": $ID}"
}

function fail() {
	MSG="$1"
	call "{\"status\": \"failed\", \"message\": \"$MSG\"}"
	exit 1
}

function downloadAndExtractRunner() {
	sendStatus "downloading tools from https://dl.gitea.com/act_runner/0.2.11/act_runner-0.2.11-linux-arm64"
	mkdir -p "$RUN_HOME" || fail "failed to create actions-runner folder"
	curl --retry 5 --retry-delay 5 --retry-connrefused --fail -L -o "$RUN_HOME/act_runner" "https://dl.gitea.com/act_runner/0.2.11/act_runner-0.2.11-linux-arm64" || fail "failed to download tools"
    sudo chown {{ .RunnerUsername }}:{{ .RunnerGroup }} -R /home/{{ .RunnerUsername }} || fail "failed to change owner"
    sudo chmod +x "$RUN_HOME/act_runner" || fail "failed to change permissions"
    sudo tee "$RUN_HOME/act_runner.service" <<EOF
[Unit]
Description=act_runner
After=network.target

[Service]
User={{ .RunnerUsername }}
Environment=HOME=/home/{{ .RunnerUsername }}
ExecStart=$RUN_HOME/act_runner daemon
WorkingDirectory=$RUN_HOME
KillMode=process
KillSignal=SIGINT
TimeoutStopSec=5min

[Install]
WantedBy=multi-user.target
EOF
    sudo systemctl enable $RUN_HOME/act_runner.service
}

if [ ! -d "$RUN_HOME" ];then
	downloadAndExtractRunner
	cd "$RUN_HOME"
else
	sendStatus "using cached runner found in $RUN_HOME"
	cd "$RUN_HOME"
fi


sendStatus "configuring runner"
{{- if .UseJITConfig }}
{{- else}}

GITHUB_TOKEN=$(curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "${METADATA_URL}/runner-registration-token/")

set +e
attempt=1
while true; do
	ERROUT=$(mktemp)
	{{- if .GitHubRunnerGroup }}
	{{- else}}

    # --ephemeral once supported
    # "{{ .RunnerLabels }}" once `:` supported
    ./act_runner register --no-interactive --instance "$(echo "{{ .RepoURL }}" | cut -d'/' -f1-3)" --token "$GITHUB_TOKEN" --name "{{ .RunnerName }}" --labels "ubuntu,generic" 2>$ERROUT
	{{- end}}
	if [ $? -eq 0 ]; then
		rm $ERROUT || true
		sendStatus "runner successfully configured after $attempt attempt(s)"
		break
	fi
	LAST_ERR=$(cat $ERROUT)
	echo "$LAST_ERR"

	if [ $attempt -gt 5 ];then
		rm $ERROUT || true
		fail "failed to configure runner: $LAST_ERR"
	fi

	sendStatus "failed to configure runner (attempt $attempt): $LAST_ERR (retrying in 5 seconds)"
	attempt=$((attempt+1))
	rm $ERROUT || true
	sleep 5
done
set -e
{{- end}}

if [ -e "/sys/fs/selinux" ];then
	sudo chcon -R -h user_u:object_r:bin_t:s0 /home/runner/ || fail "failed to change selinux context"
fi

AGENT_ID=""
{{- if .UseJITConfig }}
{{- else}}
sudo systemctl start act_runner || fail "failed to start service"
{{- end}}
systemInfo $AGENT_ID
success "runner successfully installed" $AGENT_ID

This branch of Gitea used by the POC: https://github.com/ChristopherHX/gitea/tree/workflow_job_webhook PRs pending for non stub parts

• runner download stub (only included here)
• runner register token POST method (only included here right now, idk why it uses GET)

Version of GARM that can configure go-github for gitea: https://github.com/actions-oss/gitea-garm-poc, otherwise didn't need edits if my version of gitea is used.

The --ephemeral flag act_runner can be found here: https://gitea.com/gitea/act_runner/pulls/649
Then the runner_setup_template needs to be updated with a new download url + the cli flag, then the runners that pick a task get removed from the webui automatically.

Open Bugs

• No colon support for labels https://gitea.com/gitea/act_runner/issues/664 (can be resolved by garm by using = or some other symbol for label
• No error reported due to Exitcode 0 on register failure https://gitea.com/gitea/act_runner/issues/665

[gabriel-samfira]

Hi @ChristopherHX

Amazing progress! I am currently fighting off a flu or something. Will have some input for you as soon as my brain works again.

[ChristopherHX]

I hope you will recover soon. My main focus shifted in the meantime to different topics, since the POC is running and multiple required Gitea side changes are ready for review.

[ChristopherHX]

Update

• X-GitHub-Hook-Installation-Target-Type webhook http header landed in gitea 1.24 nightly
• workflow_job webhook landed in gitea 1.24 nightly
• Ephemeral runners landed in gitea + act_runner 1.24 nightly
• runner cleanup api pending review milestone 1.24 (no longer database migrations yeah)

Gitea DEMO Branch 2.0, you can switch from Gitea 1.24 nightly to this one and back at the moment, same database version.

• https://github.com/ChristopherHX/gitea/tree/garm-demo-v2
  • cherry picked the runner deletion bugfix from UI
  • contains the runners api
  • runner downloads api stub only in this branch

#!/bin/bash

set -e
set -o pipefail

{{- if .EnableBootDebug }}
set -x
{{- end }}

CALLBACK_URL="{{ .CallbackURL }}"
METADATA_URL="{{ .MetadataURL }}"
BEARER_TOKEN="{{ .CallbackToken }}"

RUN_HOME="/home/{{ .RunnerUsername }}/actions-runner"

if [ -z "$METADATA_URL" ];then
	echo "no token is available and METADATA_URL is not set"
	exit 1
fi

function call() {
	PAYLOAD="$1"
	[[ $CALLBACK_URL =~ ^(.*)/status(/)?$ ]] || CALLBACK_URL="${CALLBACK_URL}/status"
	curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d "${PAYLOAD}" -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "${CALLBACK_URL}" || echo "failed to call home: exit code ($?)"
}

function systemInfo() {
	if [ -f "/etc/os-release" ];then
		. /etc/os-release
	fi
	OS_NAME=${NAME:-""}
	OS_VERSION=${VERSION_ID:-""}
	AGENT_ID=${1:-null}
	# strip status from the callback url
	[[ $CALLBACK_URL =~ ^(.*)/status(/)?$ ]] && CALLBACK_URL="${BASH_REMATCH[1]}" || true
	SYSINFO_URL="${CALLBACK_URL}/system-info/"
	PAYLOAD="{\"os_name\": \"$OS_NAME\", \"os_version\": \"$OS_VERSION\", \"agent_id\": $AGENT_ID}"
	curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d "${PAYLOAD}" -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "${SYSINFO_URL}" || true
}

function sendStatus() {
	MSG="$1"
	call "{\"status\": \"installing\", \"message\": \"$MSG\"}"
}

function success() {
	MSG="$1"
	ID=${2:-null}
	call "{\"status\": \"idle\", \"message\": \"$MSG\", \"agent_id\": $ID}"
}

function fail() {
	MSG="$1"
	call "{\"status\": \"failed\", \"message\": \"$MSG\"}"
	exit 1
}

function downloadAndExtractRunner() {
	sendStatus "downloading tools from https://dl.gitea.com/act_runner/nightly/act_runner-nightly-linux-arm64.xz"
	mkdir -p "$RUN_HOME" || fail "failed to create actions-runner folder"
	curl --retry 5 --retry-delay 5 --retry-connrefused --fail -L -o "$RUN_HOME/act_runner.xz" "https://dl.gitea.com/act_runner/nightly/act_runner-nightly-linux-arm64.xz" || fail "failed to download tools"
	xz -d "$RUN_HOME/act_runner.xz" || fail "failed to extract tools"
    sudo chown {{ .RunnerUsername }}:{{ .RunnerGroup }} -R /home/{{ .RunnerUsername }} || fail "failed to change owner"
    sudo chmod +x "$RUN_HOME/act_runner" || fail "failed to change permissions"
    sudo tee "$RUN_HOME/act_runner.service" <<EOF
[Unit]
Description=act_runner
After=network.target

[Service]
User={{ .RunnerUsername }}
Environment=HOME=/home/{{ .RunnerUsername }}
ExecStart=$RUN_HOME/act_runner daemon
WorkingDirectory=$RUN_HOME
KillMode=process
KillSignal=SIGINT
TimeoutStopSec=5min

[Install]
WantedBy=multi-user.target
EOF
    sudo systemctl enable $RUN_HOME/act_runner.service
}

if [ ! -d "$RUN_HOME" ];then
	downloadAndExtractRunner
	cd "$RUN_HOME"
else
	sendStatus "using cached runner found in $RUN_HOME"
	cd "$RUN_HOME"
fi


sendStatus "configuring runner"
{{- if .UseJITConfig }}
{{- else}}

GITHUB_TOKEN=$(curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "${METADATA_URL}/runner-registration-token/")

set +e
attempt=1
while true; do
	ERROUT=$(mktemp)
	{{- if .GitHubRunnerGroup }}
	{{- else}}

    # "{{ .RunnerLabels }}" once `:` supported
    ./act_runner register --ephemeral --no-interactive --instance "$(echo "{{ .RepoURL }}" | cut -d'/' -f1-3)" --token "$GITHUB_TOKEN" --name "{{ .RunnerName }}" --labels "ubuntu,generic" 2>$ERROUT
	{{- end}}
	if [ $? -eq 0 ]; then
		rm $ERROUT || true
		sendStatus "runner successfully configured after $attempt attempt(s)"
		break
	fi
	LAST_ERR=$(cat $ERROUT)
	echo "$LAST_ERR"

	if [ $attempt -gt 5 ];then
		rm $ERROUT || true
		fail "failed to configure runner: $LAST_ERR"
	fi

	sendStatus "failed to configure runner (attempt $attempt): $LAST_ERR (retrying in 5 seconds)"
	attempt=$((attempt+1))
	rm $ERROUT || true
	sleep 5
done
set -e
{{- end}}

if [ -e "/sys/fs/selinux" ];then
	sudo chcon -R -h user_u:object_r:bin_t:s0 /home/runner/ || fail "failed to change selinux context"
fi

AGENT_ID=""
{{- if .UseJITConfig }}
{{- else}}
sudo systemctl start act_runner || fail "failed to start service"
{{- end}}
systemInfo $AGENT_ID
success "runner successfully installed" $AGENT_ID

EDIT Script now uses xz for runner, since this is part of the ubuntu image...

[gabriel-samfira]

Hi @ChristopherHX

Would you recommend we use go-github as the client to interact with Gitea or does Gitea maintain its own client? If we are to add support for Gitea, I would rather we use a proper client that is maintained by Gitea which may mirror any new additions to the API, etc.

I would like to take the proper time to do it right and make sure it is maintainable moving forward.

[ChristopherHX]

does Gitea maintain its own client

I think yes it is https://gitea.com/gitea/go-sdk, but I never used or contributed to it.

At the moment it seems only go-github implements the required api methods. However I could look into adding them to the official gitea sdk.

I would like to take the proper time to do it right

Sure, good idea. Keep in mind Forgejo (since you mentioned it in this issue) state is unknown about this, they seem to go another path my last state has been ephemeral runner protocol change rejected.

For global and user runner support might be no other option than using a gitea sdk in long term.

[gabriel-samfira]

The gitea API seems to have swagger annotations. I suspect that a client could be generated for all endpoints. GARM has something similar. Our client is completely generated.

Sure, good idea. Keep in mind Forgejo (since you mentioned it in this issue) state is unknown about this, they seem to go another path my last state has been ephemeral runner protocol change rejected.

I am following their discussion as well. They seem to lack the ephemeral flag for their runners. The one-job command just tries to fetch one job, run it then exit. That is not really secure, unless the job is run in a container/VM which would never gain access to the machine that has the runner token set.

I would love to support both Gitea and Forgejo long term, but right now, gitea seems to have everything we expect for ephemeral runners. So might as well start there.

Webhooks are great to start with. What would be awesome (and this is just spit balling ony part) long term is to have some longpoll or websocket/grpc endpoint where we can subscribe for jobs. Something like how scale sets work in github. This would be good for multiple reasons:

• No need to expose the GARM api to the forge for webhooks. GARM would subscribe as a client.
• Webhooks may be triggered while GARM is down, in which case GARM will not know about pending jobs. This can be mitigated by idle runners, but if you're using a public cloud, this may incur cost.
• Efficient scheduling of runners. We can acquire a specific job by a specific entity (user, global, org) and spin up a runner for it. If we can't, we release the job and let other potential entities/auto scalers try. Or we can retry.

But for now, webhooks are great.

[ChristopherHX]

The gitea API seems to have swagger annotations. I suspect that a client could be generated for all endpoints.

Yes, but based on https://gitea.com/gitea/awesome-gitea#sdk something like this has been archived in favour of the go-sdk..

Gitea-sdk - Gitea SDK generated by Swagger. (Archived, use the official Golang SDK)

That is not really secure, unless the job is run in a container/VM which would never gain access to the machine that has the runner token set.

Yes exactly, this is my perspective as well.

What would be awesome (and this is just spit balling ony part) long term is to have some longpoll or websocket/grpc endpoint where we can subscribe for jobs.

I had read a message from techknowlogick (Gitea TOC) about that would be possible to implement.
The queue may needs a rearchitecture / use the workflow_job webhook as event source to implement this internally.

act_runner uses grpc, with short polling at the moment. I wish that would be long polling like actions/runner does e.g. up to 50s blocked poll returning as soon a job is available.
Subscribing would be great....

Depending on how this would work you might come to the point that the VM you spawn need a token for logs and job status at the moment you would end up with the runner token of the autoscaler.
What do you think about this proposal?
go-gitea/gitea#32629 (comment)

If the act_runner would be allowed to pick a free job by it's id taken out of an autoscaler/webhook event you could use a new ephemeral runner to run it and keep the autoscaler token private to the scaler.

Alternative deep integration into Gitea

---

act_runner could be modified to do the following

• provide a poller as go api that calls a golang callback / could be implemented here as well (garm scaler side, act_runner may provide the library)
• provide a way to accept a task json via stdin or file including the runner creds (act_runner side)
• before autoscaler polls again (garm scaler side)
  • register a new runner for the autoscaler
  • replace the token of the poller
  • the old runner ownership went to the VM
    • Again proposal Grant Job Update and Job Log right to ACTIONS_RUNTIME_TOKEN go-gitea/gitea#32629 (comment) could allow reusing the autoscaler token by not transferring ownership to the VM/ephemeral runner

Optionally the workflow_job webhook in Gitea could allow to do a better forcast about jobs that might be queues soon, status "waiting" (in Gitea slang blocked, webhook follows GitHubs naming)

At least my implementation should announce every workflow jobs including ones that are waiting for dependent jobs

[ChristopherHX]

I now tried the third party k8s provider as well https://gitea.com/gitea/helm-actions/issues/8. This operator does not seem to need any extra spec, because it drops the runner setup template. I have built my own docker image using the expected abi and it works.

garm works great for my experiments 👍, the runner scaled from zero to n for a Gitea Actions matrix using the workflow_job metric.

I guess my idea of using the garm-operator with my custom default branch build didn't work correctly, because of some incompatible changes between release garm v1.5.0 and patched main or I just didn't understand how this thing works.