cmd: gather the logs from bootstrap instead of printing commands #1822

abhinavdahiya · 2019-06-05T22:10:54Z

xref: https://jira.coreos.com/browse/CORS-1071

This changes the gather bootstrap to establish an SSH connection to bootstrap host to gather the logs, rather than printing the commands for
the user to execute.

$ ./openshift-install --dir dev gather bootstrap --log-level debug --bootstrap 192.168.126.10 --master 192.168.126.11
DEBUG OpenShift Installer unreleased-master-1100-gc3be773d1eedc9356c5f9af955bcb8db25729e0b
DEBUG Built from commit c3be773d1eedc9356c5f9af955bcb8db25729e0b
INFO Pulling logs from bootstrap for debugging
DEBUG Gathering bootstrap journals ...
DEBUG Gathering bootstrap containers ...
DEBUG Gathering rendered assets...
DEBUG Gathering cluster resources ...
DEBUG Waiting for logs ...
DEBUG error: the server doesn't have a resource type "nodes"
DEBUG error: the server doesn't have a resource type "pods"
DEBUG error: the server doesn't have a resource type "nodes"
DEBUG error: the server doesn't have a resource type "configmaps"
DEBUG error: the server doesn't have a resource type "clusterversion"
DEBUG error: the server doesn't have a resource type "pods"
DEBUG error: the server doesn't have a resource type "clusteroperators"
DEBUG error: the server doesn't have a resource type "csr"
DEBUG error: the server doesn't have a resource type "apiservices"
DEBUG error: the server doesn't have a resource type "endpoints"
DEBUG error: the server doesn't have a resource type "namespaces"
DEBUG error: the server doesn't have a resource type "roles"
DEBUG error: the server doesn't have a resource type "kubecontrollermanager"
DEBUG error: the server doesn't have a resource type "machineconfigpools"
DEBUG error: the server doesn't have a resource type "kubeapiserver"
DEBUG error: the server doesn't have a resource type "nodes"
DEBUG error: the server doesn't have a resource type "secrets"
DEBUG error: the server doesn't have a resource type "pods"
DEBUG error: the server doesn't have a resource type "events"
DEBUG error: the server doesn't have a resource type "services"
DEBUG error: the server doesn't have a resource type "openshiftapiserver"
DEBUG error: the server doesn't have a resource type "rolebindings"
DEBUG Error from server (NotFound): the server could not find the requested resource
DEBUG error: the server doesn't have a resource type "secrets"
DEBUG error: the server doesn't have a resource type "machineconfigs"
DEBUG Gather remote logs
DEBUG Collecting info from 192.168.126.11
DEBUG scp: ./installer-masters-gather.sh: Permission denied
 EBUG Warning: Permanently added '192.168.126.11' (ECDSA) to the list of known hosts.
DEBUG Gathering master journals ...
DEBUG Gathering master containers ...
DEBUG Waiting for logs ...
DEBUG Log bundle written to ~/log-bundle.tar.gz
INFO Bootstrap gather logs captured here "dev/log-bundle.tar.gz"

/cc @jhixson74 @patrickdillon @wking

openshift-ci-robot · 2019-06-05T22:10:55Z

@abhinavdahiya: GitHub didn't allow me to request PR reviews from the following users: jhixson74.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

xref: https://jira.coreos.com/browse/CORS-1071

/cc @jhixson74 @patrickdillon @wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

abhinavdahiya · 2019-06-05T22:15:40Z

/cc @jhixson74

openshift-ci-robot · 2019-06-05T22:15:41Z

@abhinavdahiya: GitHub didn't allow me to request PR reviews from the following users: jhixson74.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @jhixson74

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

abhinavdahiya · 2019-06-05T22:44:58Z

/retest

abhinavdahiya · 2019-06-06T19:13:39Z

/cc @openshift/openshift-team-installer

cmd/openshift-install/gather.go

smarterclayton · 2019-06-06T20:36:49Z

Awesome. Can we invoke gather in the e2e tear down trap?

cmd/openshift-install/gather.go

pkg/gather/ssh/ssh.go

pkg/gather/ssh/agent.go

wking · 2019-06-06T21:01:48Z

Can we invoke gather in the e2e tear down trap?

We already gather in CI's teardown container. I don't see a need to teardown in the trap.

The package allows users to create a new SSH client that - Uses an SSH agent to provide authentication - Either uses the keys from paths explicity provided, or loads all the keys from `~/.ssh` to use for authentication The package also provides a way to Run a command using an SSH client. The client must be configured to allow SSH agent forwarding. The package also provides a way to pull a file from remote server to a file on the local host using a SSH client.

…ogs on bootstrap failure With installer PR [1], the installer will automatically gather the logs from bootstrap in case of bootstrap failure. It uses the `~/.ssh` directlory to load the private SSH keys for authentication, when not specifically invoked using `gather bootstrap --key <>` [1]: openshift/installer#1822

wking · 2019-06-09T21:57:08Z

/lgtm

openshift-ci-robot · 2019-06-09T21:57:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2019-06-09T22:04:44Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-06-09T23:16:41Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-06-09T23:52:58Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-06-10T00:40:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2019-06-10T01:58:03Z

@abhinavdahiya: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-scaleup-rhel7	`4326be6`	link	`/test e2e-aws-scaleup-rhel7`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Since openshift/installer@cad7f02a8b (cmd: gather the logs from bootstrap instead of printing commands, 2019-06-05, openshift/installer#1822). This functionality is new in 4.2.z; you still need to SSH in 4.1.z.

wking · 2019-12-18T09:47:08Z

/cherrypick release-4.1

openshift-cherrypick-robot · 2019-12-18T09:47:25Z

@wking: #1822 failed to apply on top of branch "release-4.1":

Using index info to reconstruct a base tree...
M	Gopkg.lock
M	Gopkg.toml
Falling back to patching base and 3-way merge...
Auto-merging Gopkg.toml
Auto-merging Gopkg.lock
Applying: pkg: add gather/ssh package for utilities for gathering using ssh
Applying: cmd: gather the logs from bootstrap instead of printing commands
error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	cmd/openshift-install/gather.go
Falling back to patching base and 3-way merge...
Auto-merging cmd/openshift-install/gather.go
CONFLICT (content): Merge conflict in cmd/openshift-install/gather.go
Patch failed at 0003 cmd: gather the logs from bootstrap instead of printing commands

In response to this:

/cherrypick release-4.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…ather The installer does this automatically since openshift/installer@cad7f02a8b (cmd: gather the logs from bootstrap instead of printing commands, 2019-06-05, openshift/installer#1822). You'd still need to SSH in to gather manually in 4.1.z, but if we think that's important we should backport the gather logic to the installer's 4.1 branch [1]. [1]: openshift/installer#2840

abhinavdahiya requested a review from patrickdillon June 5, 2019 22:10

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 5, 2019

openshift-ci-robot requested a review from wking June 5, 2019 22:10

openshift-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jun 5, 2019

abhinavdahiya force-pushed the gather_agent branch from 30af121 to e747b85 Compare June 5, 2019 22:19

vendor: add github.com/pkg/sftp and golang.org/x/crypto/ssh/agent

9a237b5

abhinavdahiya force-pushed the gather_agent branch from e747b85 to e87bb07 Compare June 6, 2019 19:11

abhinavdahiya changed the title ~~WIP: cmd: gather the logs from bootstrap instead of printing commands~~ cmd: gather the logs from bootstrap instead of printing commands Jun 6, 2019

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 6, 2019

openshift-ci-robot requested a review from a team June 6, 2019 19:13