Skip to content

ingress-controller: update ingresses only if the load balancer is active #745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

madumalt
Copy link
Collaborator

@madumalt madumalt commented May 27, 2025

Issue Description

As per the existing behavior, once the stack update completes, the
ingress controller immediately updates the status of all ingresses and
routegroups to reference the new load balancer. This update may
sometimes happens before the new load balancer has marked its targets
(skipper-ingress) as healthy, leading clients being routed to a load
balancer that is not ready to serve traffic yet.

Fix

To improve this behavior we retrieve the ELBs via aws api and build
the models including the ELB state of the corresponding ELB. Then
before updating the ingresses/routegroups (updateIngress func) we
check whether the ELB is in active state, if not we skip updating
the ingresses/routegroups status with the ELB hostname.

@madumalt madumalt marked this pull request as draft May 27, 2025 04:14
@AlexanderYastrebov
Copy link
Member

Overall approach to check loadbalancer status looks OK.
There should be a way to get loadbalancer ARN from CF stack by logical id (which is "LB", see how CF stack is created).
DescribeLoadBalancers API allows fetching multiple load balancers so I think we should collect them first and then buildManagedModel and processes updates.
Please also add tests.

@madumalt
Copy link
Collaborator Author

so I think we should collect them first and then buildManagedModel and processes updates.

made the changes to address this

@madumalt
Copy link
Collaborator Author

madumalt commented May 28, 2025

TBD

  • TestResourceConversionOneToOne tests in worker_test.go is failing
  • Add tests for the newly introduced funtions (getLoadBalancerStackResource, GetLoadBalancer)

@madumalt madumalt force-pushed the check-lb-state-before-ingress-update branch 3 times, most recently from de27916 to 152337a Compare June 10, 2025 16:56
@madumalt madumalt marked this pull request as ready for review June 10, 2025 17:01
type ELBV2API interface {
elbv2.DescribeTargetGroupsAPIClient
elbv2.DescribeTargetHealthAPIClient
elbv2.DescribeLoadBalancersAPIClient
Copy link
Member

@szuecs szuecs Jun 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this likely needs also an adjustment of our documentation about what AWS policy you need, and if so we would need also to change minor version in VERSION file and communicate the change in the release.

see also https://github.com/zalando-incubator/kube-ingress-aws-controller/blob/master/deploy/requirements.md

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes introduced here need permission for elasticloadbalancing:DescribeLoadBalancers and cloudformation:ListStackReources Actions. The requirements.md already have them documents as follows.

{
"Version": "2012-10-17",
"Statement": [
  ...
  {
        "Action": "elasticloadbalancing:DescribeLoadBalancers",
        "Resource": "*",
        "Effect": "Allow"
  },
  ...
  {
        "Action": "cloudformation:List*",
        "Resource": "*",
        "Effect": "Allow"
  },
  ...
]
}

@madumalt madumalt force-pushed the check-lb-state-before-ingress-update branch from 8e8a668 to e0596e6 Compare June 16, 2025 11:44
Thilina Madumal and others added 3 commits June 16, 2025 17:27
As per the existing behavior, once the stack update completes, the
ingress controller immediately updates the status of all ingresses and
routegroups to reference the new load balancer. This update may
sometimes happens before the new load balancer has marked its targets
(skipper-ingress) as healthy, leading clients being routed to a load
balancer that is not ready to serve traffic yet.

To improve this behavior we retrieve the ELBs via aws api and build
the models including the ELB state of the corresponding ELB. Then
before updating the ingresses/routegroups (updateIngress func) we
check whether the ELB is in active state, if not we skip updating
the ingresses/routegroups status with the ELB hostname.

Signed-off-by: Thilina Madumal <[email protected]>
Refactor fake certificate provider to enable creation of multiple
certificates with the same CA and to allow manual mocking of
GetCertificates result.

Follow up on zalando-incubator#615 and zalando-incubator#587

Signed-off-by: Alexander Yastrebov <[email protected]>
Controller manages loadbalancer lifecycle based on available
certificates.

Prevent controller from deleting loadbalancers when there are no
certificates found e.g. due to a bug like zalando-incubator#725 or a tag filter
misconfiguration (zalando-incubator#658).

Signed-off-by: Alexander Yastrebov <[email protected]>
@madumalt madumalt force-pushed the check-lb-state-before-ingress-update branch from e0596e6 to 2169f9c Compare June 16, 2025 15:28
@madumalt
Copy link
Collaborator Author

Closing this PR in favour of #748. I messed the commit history while rebasing, it was easier to cherrypick the changes and create a new PR rather than trying to fix the commit history I messed up here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants