Skip to content

Race Condition When Multiple ImageUpdateAutomation Objects Push to Same Branch #939

Open
@ivan-chepurin-immutable

Description

@ivan-chepurin-immutable

Problem Description

I'm experiencing race conditions when multiple ImageUpdateAutomation objects attempt to push commits to the same branch (main) in a monorepo setup. The push operations fail with git reference locking errors, and the official documentation doesn't provide clear guidance on handling this scenario.

Environment

  • Flux Version: v2.x (image-automation-controller)
  • Repository Setup: Monorepo with multiple services containing app code and k8s manifests.
  • Git Provider: GitHub
  • Deployment: Multiple ImageUpdateAutomation objects, each managing different services

Repository Structure

├── deployment
│   ├── api
│   │   ├── base
│   │   │   ├── deployment.yaml
│   │   │   ├── image-policy.yaml
│   │   │   ├── image-update-automation.yaml
│   │   ├── dev
│   │   │   ├── deployment.yaml
│   │   │   ├── image-policy.yaml
│   │   │   ├── image-update-automation.yaml
│   │   ├── Dockerfile
│   │   ├── main.go
│   │   ├── prod
│   │   │   ├── deployment.yaml
│   │   │   ├── image-policy.yaml
│   │   │   ├── image-update-automation.yaml
│   │   ├── project.json
│   │   └── sandbox
│   │       ├── deployment.yaml
│   │       ├── image-policy.yaml
│   │       ├── image-update-automation.yaml

Current Configuration

Each service has its own ImageUpdateAutomation with a specific path:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: service-a-automation
spec:
  sourceRef:
    kind: GitRepository
    name: platform-repo
  update:
    path: ./services/service-a/deployment/prod
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: [email protected]
        name: flux
      messageTemplate: |
        Automated image update for service-a
        
        Files:
        {{ range $filename, _ := .Updated.Files -}}
        - {{ $filename }}
        {{ end -}}
    # No push section specified = pushes to main

Each service/env combo has an imagepolicy specified that is referenced in deployment.yaml.

---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: service-a
spec:
  filterTags:
    pattern: ^develop-(?P<date>\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2}Z)-.*
    extract: "$date"
  policy:
    alphabetical:
      order: asc

Errors Encountered

"message": "failed to update source: failed to push to remote: command error on refs/heads/main: cannot lock ref 'refs/heads/main': is at 78b5676cca83c5da14a80abceea8252719274736 but expected f32e161e8f41132538da67a74c8552e35757b31f"
"message": "failed to update source: failed to push to remote: object not found"

Analysis

This error occurs when:

  1. Multiple ImageUpdateAutomation objects run concurrently
  2. Each clones the repository at the same commit
  3. Each makes changes and attempts to push to main
  4. The first push succeeds, subsequent pushes fail due to the branch having moved
  5. When multiple ImageUpdateAutomations fail to push to main branch, they retry fail and next retry is being pushed further by exponential backoff retry.
  6. This significantly slows down time to deploy cluster from a moment new image is pushed to ECR repository.

Questions and Concerns

1. Documentation Gap

The official Flux documentation extensively shows examples pushing directly to main:

spec:
  git:
    checkout:
      ref:
        branch: main
    # No push section = pushes to main

However, there's no clear guidance on:

  • How to handle multiple automations in the same repository
  • When race conditions occur and how to prevent them
  • Best practices for monorepo setups

2. Monorepo Best Practices

For monorepo setups with multiple services, what's the recommended approach?

Option A: Single ImageUpdateAutomation

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: platform-automation
spec:
  update:
    path: ./services  # Update all services
  # Single automation, no race conditions

Option B: Service-specific staging branches

spec:
  git:
    checkout:
      ref:
        branch: main
    push:
      branch: auto-updates-service-a  # Unique per service

Or anything else that I'm missing?

Request for Guidance

  1. What's the current recommended approach for handling multiple ImageUpdateAutomation objects in the same repository?

  2. Are there plans to improve the documentation around these scenarios?

Reproduction Steps

  1. Create a monorepo with multiple services
  2. Set up multiple ImageUpdateAutomation objects, each with different update.path
  3. Configure all to push to main branch
  4. Trigger image updates for multiple services simultaneously
  5. Observe race condition errors

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions