Description
Problem Description
I'm experiencing race conditions when multiple ImageUpdateAutomation
objects attempt to push commits to the same branch (main) in a monorepo setup. The push operations fail with git reference locking errors, and the official documentation doesn't provide clear guidance on handling this scenario.
Environment
- Flux Version: v2.x (image-automation-controller)
- Repository Setup: Monorepo with multiple services containing app code and k8s manifests.
- Git Provider: GitHub
- Deployment: Multiple ImageUpdateAutomation objects, each managing different services
Repository Structure
├── deployment
│ ├── api
│ │ ├── base
│ │ │ ├── deployment.yaml
│ │ │ ├── image-policy.yaml
│ │ │ ├── image-update-automation.yaml
│ │ ├── dev
│ │ │ ├── deployment.yaml
│ │ │ ├── image-policy.yaml
│ │ │ ├── image-update-automation.yaml
│ │ ├── Dockerfile
│ │ ├── main.go
│ │ ├── prod
│ │ │ ├── deployment.yaml
│ │ │ ├── image-policy.yaml
│ │ │ ├── image-update-automation.yaml
│ │ ├── project.json
│ │ └── sandbox
│ │ ├── deployment.yaml
│ │ ├── image-policy.yaml
│ │ ├── image-update-automation.yaml
Current Configuration
Each service has its own ImageUpdateAutomation
with a specific path:
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: service-a-automation
spec:
sourceRef:
kind: GitRepository
name: platform-repo
update:
path: ./services/service-a/deployment/prod
git:
checkout:
ref:
branch: main
commit:
author:
email: [email protected]
name: flux
messageTemplate: |
Automated image update for service-a
Files:
{{ range $filename, _ := .Updated.Files -}}
- {{ $filename }}
{{ end -}}
# No push section specified = pushes to main
Each service/env combo has an imagepolicy specified that is referenced in deployment.yaml.
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
name: service-a
spec:
filterTags:
pattern: ^develop-(?P<date>\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2}Z)-.*
extract: "$date"
policy:
alphabetical:
order: asc
Errors Encountered
"message": "failed to update source: failed to push to remote: command error on refs/heads/main: cannot lock ref 'refs/heads/main': is at 78b5676cca83c5da14a80abceea8252719274736 but expected f32e161e8f41132538da67a74c8552e35757b31f"
"message": "failed to update source: failed to push to remote: object not found"
Analysis
This error occurs when:
- Multiple
ImageUpdateAutomation
objects run concurrently - Each clones the repository at the same commit
- Each makes changes and attempts to push to
main
- The first push succeeds, subsequent pushes fail due to the branch having moved
- When multiple ImageUpdateAutomations fail to push to
main
branch, they retry fail and next retry is being pushed further by exponential backoff retry. - This significantly slows down time to deploy cluster from a moment new image is pushed to ECR repository.
Questions and Concerns
1. Documentation Gap
The official Flux documentation extensively shows examples pushing directly to main
:
spec:
git:
checkout:
ref:
branch: main
# No push section = pushes to main
However, there's no clear guidance on:
- How to handle multiple automations in the same repository
- When race conditions occur and how to prevent them
- Best practices for monorepo setups
2. Monorepo Best Practices
For monorepo setups with multiple services, what's the recommended approach?
Option A: Single ImageUpdateAutomation
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: platform-automation
spec:
update:
path: ./services # Update all services
# Single automation, no race conditions
Option B: Service-specific staging branches
spec:
git:
checkout:
ref:
branch: main
push:
branch: auto-updates-service-a # Unique per service
Or anything else that I'm missing?
Request for Guidance
-
What's the current recommended approach for handling multiple ImageUpdateAutomation objects in the same repository?
-
Are there plans to improve the documentation around these scenarios?
Reproduction Steps
- Create a monorepo with multiple services
- Set up multiple
ImageUpdateAutomation
objects, each with differentupdate.path
- Configure all to push to
main
branch - Trigger image updates for multiple services simultaneously
- Observe race condition errors