templates/openshift: grab bootstrap log on failure #2624
Conversation
openshift-ci-robot
added
the
size/M
|
/assign @crawford |
This uses the Terraform state to discover the IP address of the bootstrap node (ideally, the installer will provide this information in a form which easier to consume in the future). It then connects to the gatewayd instance on that machine and pulls the logs for various services. Hopefully, these logs will be useful when diagnosing installation failures.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: sttts If they are not already assigned, you can assign the PR to them by writing The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Oh nice! I'll give this a run locally and then LGTM if it looks good. |
|
As I say in all cases where I see this -- where else can this logic live? Python in Bash in template YAML is not maintainable. |
Medium term this should move directly into the installer itself. So this is more an ad-hoc tool to get us visibility into the bootstrapping phase now if things go badly wrong in CI. We are blind, so this would help a lot in diagnosis when testgrid is completely red again. |
| # jq '.modules[].resources."aws_instance.bootstrap".primary.attributes."public_ip" | select(.)' | ||
| bootstrap_ip=$(python -c \ | ||
| 'import sys, json; d=reduce(lambda x,y: dict(x.items() + y.items()), map(lambda x: x["resources"], json.load(sys.stdin)["modules"])); k="aws_instance.bootstrap"; print d[k]["primary"]["attributes"]["public_ip"] if k in d else ""' \ | ||
| /tmp/artifacts/installer/terraform.tfstate |
There was a problem hiding this comment.
You need to pipe that file into the script explicitly, iow:
python -c '...' < /tmp/artifacts/installer/terraform.tfstate
and you'll be good.
|
@stevekuznetsov when I was working on #2353, I was explicitly told (in slack) to just use a heredoc instead of trying to break the install configs out and into a separate file. If you can provide guidance on the correct method, I'd be happy to help restructure this. |
|
/close |
|
@crawford: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Python rewrite of #2581