Skip to content

Upgrade of bookkeeper cluster is failing intermittently #213

Closed
@anishakj

Description

@anishakj

Description

Upgrade of bookkeeper cluster is failing intermittently. In operator logs, seeing below output

time="2022-06-23T13:56:25Z" level=info msg="syncing cluster version from 4.14.3-8-de07afb7f to 4.14.3-13-378101a"
time="2022-06-23T13:56:25Z" level=info msg="Reconciling BookkeeperCluster store-ns/openshift-upgrade-bk\n"
time="2022-06-23T13:56:25Z" level=info msg="updating statefulset (openshift-upgrade-bk-bookie) template image to 'asdrepo.isus.emc.com:8099/bookkeeper:4.14.3-13-378101a'"
time="2022-06-23T13:56:25Z" level=info msg="failed to reconcile bookkeeper cluster (openshift-upgrade-bk): failed to reconcile cluster status: failed to update cluster status: Operation cannot be fulfilled on bookkeeperclusters.bookkeeper.pravega.io \"openshift-upgrade-bk\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1655992585.8642533,"logger":"controller.bookkeepercluster","msg":"Reconciler error","reconciler group":"bookkeeper.pravega.io","reconciler kind":"BookkeeperCluster","name":"openshift-upgrade-bk","namespace":"store-ns","error":"failed to reconcile cluster status: failed to update cluster status: Operation cannot be fulfilled on bookkeeperclusters.bookkeeper.pravega.io \"openshift-upgrade-bk\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
time="2022-06-23T13:56:25Z" level=info msg="Reconciling BookkeeperCluster store-ns/openshift-upgrade-bk\n"
time="2022-06-23T13:56:25Z" level=info msg="statefulset (openshift-upgrade-bk-bookie) status: 4 updated, 4 ready, 4 target"
2022/06/23 13:56:25 Updating version history adding version 4.14.3-13-378101a

Importance

must-have

Location

upgrade.go

Suggestions for an improvement

Handle upgrade in this scenario as well

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions