Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Etcd Upgrade Fails When Persistent Storage Flag is Disabled #78398

Closed
saurabhnetskope opened this issue Feb 27, 2025 · 11 comments
Closed

Etcd Upgrade Fails When Persistent Storage Flag is Disabled #78398

saurabhnetskope opened this issue Feb 27, 2025 · 11 comments
Assignees
@juan131
Labels
etcd solved stale 15 days without activity tech-issues The user has a technical issue about an application

Comments

@saurabhnetskope
Copy link

saurabhnetskope commented Feb 27, 2025 •
edited by carrodher
Loading

Name and Version

bitnami/etcd:3.5.18

What architecture are you using?

amd64

What steps will reproduce the bug?

Description:
We are running a 3-pod etcd cluster without persistent storage, relying on emptyDir. However, after the recent change introduced in commit 1aff4e2, the etcd upgrade is failing.

Steps to Reproduce:

  1. Deploy a 3-pod etcd cluster with persistent storage disabled (emptyDir used instead).
  2. Attempt to perform a rolling upgrade.
  3. Observe that the upgrade fails.

Root Cause:
The issue lies in the is_new_etcd_cluster function. To determine if the cluster is new or existing, this function executes:

is_new_etcd_cluster() {
    local -a extra_flags
    read -r -a extra_flags <<<"$(etcdctl_auth_flags)"
    is_boolean_yes "$ETCD_ON_K8S" && extra_flags+=("--endpoints=$(etcdctl_get_endpoints)")
    ! debug_execute etcdctl endpoint status --cluster "${extra_flags[@]}"
}

During a rolling upgrade, not all endpoints will be responsive. When etcdctl_get_endpoints includes its own endpoint, the command:

! debug_execute etcdctl endpoint status --cluster "${extra_flags[@]}"

fails, leading to the upgrade issue.

Proposed Fix:
Modify etcdctl_get_endpoints to exclude the pod's own endpoint before executing etcdctl endpoint status --cluster. This will prevent failures when checking the cluster status during a rolling upgrade.

Expected Behavior:
The etcd cluster should successfully upgrade even when persistent storage is disabled, allowing rolling upgrades to complete without failure.

Environment Details:

  • Etcd Cluster: 3 pods
  • Storage: emptyDir
  • Affected Commit: 1aff4e2
  • Kubernetes Environment: [Provide Kubernetes version]
  • Helm Chart Version (if applicable): [Provide chart version]

What is the expected behavior?

The etcd cluster should successfully upgrade even when persistent storage is disabled, allowing rolling upgrades to complete without failure.

What do you see instead?

  • Prevents successful rolling upgrades for etcd clusters without persistent storage.
  • Could impact production environments relying on emptyDir for ephemeral etcd storage.
@saurabhnetskope saurabhnetskope added the tech-issues The user has a technical issue about an application label Feb 27, 2025
@github-actions github-actions bot added the triage Triage is needed label Feb 27, 2025
@saurabhnetskope saurabhnetskope mentioned this issue Feb 27, 2025
Merged
@saurabhnetskope
Copy link
Author

saurabhnetskope commented Feb 27, 2025 •
edited by carrodher
Loading

this is fixed function.

is_new_etcd_cluster() {
    local -a extra_flags
    read -r -a extra_flags <<< "$(etcdctl_auth_flags)"
    
    is_boolean_yes "$ETCD_ON_K8S" && extra_flags+=("--endpoints=$(etcdctl_get_endpoints true)")
    
    ! debug_execute etcdctl endpoint status --cluster "${extra_flags[@]}"
}

@carrodher
Copy link
Member

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

@carrodher carrodher added the etcd label Feb 27, 2025
@juan131
Copy link
Contributor

juan131 commented Mar 3, 2025

@saurabhnetskope please check upgrading notes on latest release:

Remove support for manual scaling with kubectl or autoscaler. Upgrading of any kind including increasing replica count must be done with helm upgrade exclusively. CD automation tools that respect Helm hooks such as ArgoCD can also be used.

@saurabhnetskope
Copy link
Author

@juan131 helm upgrade is not working when persistent storage is disabled.

@juan131
Copy link
Contributor

juan131 commented Mar 3, 2025

Could you please provide the chart parameters (provided via values.yaml or using the --set flag) you used to upgrade your Helm release? Please exclusively provide the parameters you customized avoiding the ones with default values.

@alop
Copy link

alop commented Mar 13, 2025

Using 3.5.18-debian-12-r0 through 3.5.19-debian-12-r0, we're seeing this problem. The initial cluster setup works, but after it transitions from new to existing the pods will never rejoin the cluster, even though it is healthy:

etcd-6 etcd etcd 21:27:44.08 INFO  ==>
etcd-6 etcd etcd 21:27:44.08 INFO  ==> Welcome to the Bitnami etcd container
etcd-6 etcd etcd 21:27:44.08 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
etcd-6 etcd etcd 21:27:44.08 INFO  ==> Did you know there are enterprise versions of the Bitnami catalog? For enhanced secure software supply chain features, unlimited pulls from Docker, LTS support, or application customization, see Bitnami Premium or Tanzu Application Catalog. See https://www.arrow.com/globalecs/na/vendors/bitnami/ for more information.
etcd-6 etcd etcd 21:27:44.08 INFO  ==>
etcd-6 etcd etcd 21:27:44.09 INFO  ==> ** Starting etcd setup **
etcd-6 etcd etcd 21:27:44.10 INFO  ==> Validating settings in ETCD_* env vars..
etcd-6 etcd etcd 21:27:44.10 INFO  ==> Initializing etcd
etcd-6 etcd etcd 21:27:44.10 INFO  ==> Generating etcd config file using env variables
etcd-6 etcd etcd 21:27:44.11 INFO  ==> There is no data from previous deployments
etcd-6 etcd etcd 21:27:50.17 INFO  ==> Bootstrapping a new cluster
etcd-6 etcd etcd 21:27:50.18 INFO  ==> ** etcd setup finished! **
etcd-6 etcd
etcd-6 etcd etcd 21:27:50.19 INFO  ==> ** Starting etcd **
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212635Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://etcd-6.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2379,http://etcd.muon-tenant-appconnector-stage-1.svc.cluster.local:2379"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212698Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTH_TOKEN","variable-value":"simple"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212706Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"periodic"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212710Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"1h"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212715Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212722Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212729Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212755Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"https://etcd-6.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2380"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212764Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"etcd-0=https://etcd-0.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2380,etcd-1=https://etcd-1.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2380,etcd-2=https://etcd-2.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2380,etcd-3=https://etcd-3.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2380,etcd-4=https://etcd-4.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2380,etcd-5=https://etcd-5.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2380,etcd-6=https://etcd-6.etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local:2380"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212771Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"existing"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212776Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212783Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212792Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"https://0.0.0.0:2380"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212800Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"warn"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212815Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"etcd-6"}
etcd-6 etcd {"level":"info","ts":"2025-03-13T21:27:50.212826Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"true"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212878Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFRAG_SERVICE_PORT_DUMMY=80"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212896Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212902Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFRAG_PORT_80_TCP_PORT=80"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212907Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SERVICE_HOST=172.20.212.158"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212917Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT_2380_TCP_ADDR=172.20.212.158"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212921Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT_2379_TCP=tcp://172.20.212.158:2379"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212928Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212933Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFRAG_SERVICE_PORT=80"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212943Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOT_HISTORY_LIMIT=1"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212949Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFRAG_PORT=tcp://172.20.228.55:80"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212960Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212971Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT_2380_TCP_PROTO=tcp"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212977Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212985Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.212998Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213005Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT_2379_TCP_PORT=2379"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213012Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFAULT_CONF_DIR=/opt/bitnami/etcd/conf.default"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213019Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT_2379_TCP_ADDR=172.20.212.158"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213027Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT_2380_TCP_PORT=2380"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213038Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFRAG_PORT_80_TCP_ADDR=172.20.228.55"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213047Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFRAG_PORT_80_TCP=tcp://172.20.228.55:80"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213059Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT_2380_TCP=tcp://172.20.212.158:2380"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213072Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=etcd-headless.muon-tenant-appconnector-stage-1.svc.cluster.local"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213079Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213086Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213090Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213094Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFRAG_SERVICE_HOST=172.20.228.55"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213099Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DEFRAG_PORT_80_TCP_PROTO=tcp"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213103Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213106Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213111Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SERVICE_PORT_CLIENT=2379"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213115Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213119Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213124Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213128Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SERVICE_PORT_PEER=2380"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213132Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213145Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT_2379_TCP_PROTO=tcp"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213150Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213154Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SERVICE_PORT=2379"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213157Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_PORT=tcp://172.20.212.158:2379"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213162Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213166Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213197Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213246Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.213493Z","caller":"transport/listener.go:281","msg":"automatically generate certificates","certificate-validity-bound-not-after":"2026-03-13T21:27:50.213491Z"}
etcd-6 etcd {"level":"warn","ts":"2025-03-13T21:27:50.326800Z","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}
etcd-6 etcd {"level":"panic","ts":"2025-03-13T21:27:50.341028Z","logger":"raft","caller":"etcdserver/zap_raft.go:101","msg":"tocommit(4642) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*zapRaftLogger).Panicf\n\tgo.etcd.io/etcd/server/v3/etcdserver/zap_raft.go:101\ngo.etcd.io/etcd/raft/v3.(*raftLog).commitTo\n\tgo.etcd.io/etcd/raft/[email protected]/log.go:237\ngo.etcd.io/etcd/raft/v3.(*raft).handleHeartbeat\n\tgo.etcd.io/etcd/raft/[email protected]/raft.go:1508\ngo.etcd.io/etcd/raft/v3.stepFollower\n\tgo.etcd.io/etcd/raft/[email protected]/raft.go:1434\ngo.etcd.io/etcd/raft/v3.(*raft).Step\n\tgo.etcd.io/etcd/raft/[email protected]/raft.go:975\ngo.etcd.io/etcd/raft/v3.(*node).run\n\tgo.etcd.io/etcd/raft/[email protected]/node.go:356"}
etcd-6 etcd panic: tocommit(4642) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?
etcd-6 etcd
etcd-6 etcd goroutine 346 [running]:
etcd-6 etcd go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001b0c00, {0x0, 0x0, 0x0})
etcd-6 etcd 	go.uber.org/[email protected]/zapcore/entry.go:234 +0x2dc
etcd-6 etcd go.uber.org/zap.(*SugaredLogger).log(0xc000396138, 0x4, {0x114c3a6?, 0x40f1fb?}, {0xc002358660?, 0xf21240?, 0x411600?}, {0x0, 0x0, 0x0})
etcd-6 etcd 	go.uber.org/[email protected]/sugar.go:227 +0xec
etcd-6 etcd go.uber.org/zap.(*SugaredLogger).Panicf(...)
etcd-6 etcd 	go.uber.org/[email protected]/sugar.go:159
etcd-6 etcd go.etcd.io/etcd/server/v3/etcdserver.(*zapRaftLogger).Panicf(0x1222?, {0x114c3a6?, 0x7ea?}, {0xc002358660?, 0x0?, 0x0?})
etcd-6 etcd 	go.etcd.io/etcd/server/v3/etcdserver/zap_raft.go:101 +0x45
etcd-6 etcd go.etcd.io/etcd/raft/v3.(*raftLog).commitTo(0xc0002f2cb0, 0x1222)
etcd-6 etcd 	go.etcd.io/etcd/raft/[email protected]/log.go:237 +0xf3
etcd-6 etcd go.etcd.io/etcd/raft/v3.(*raft).handleHeartbeat(_, {0x8, 0x9f34b836f55aa4e2, 0xab8b3db3172e0f08, 0xc, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
etcd-6 etcd 	go.etcd.io/etcd/raft/[email protected]/raft.go:1508 +0x39
etcd-6 etcd go.etcd.io/etcd/raft/v3.stepFollower(_, {0x8, 0x9f34b836f55aa4e2, 0xab8b3db3172e0f08, 0xc, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
etcd-6 etcd 	go.etcd.io/etcd/raft/[email protected]/raft.go:1434 +0x3b8
etcd-6 etcd go.etcd.io/etcd/raft/v3.(*raft).Step(_, {0x8, 0x9f34b836f55aa4e2, 0xab8b3db3172e0f08, 0xc, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
etcd-6 etcd 	go.etcd.io/etcd/raft/[email protected]/raft.go:975 +0x12d5
etcd-6 etcd go.etcd.io/etcd/raft/v3.(*node).run(0xc00060dec0)
etcd-6 etcd 	go.etcd.io/etcd/raft/[email protected]/node.go:356 +0x925
etcd-6 etcd created by go.etcd.io/etcd/raft/v3.RestartNode in goroutine 1
etcd-6 etcd 	go.etcd.io/etcd/raft/[email protected]/node.go:244 +0x259
- etcd-6 › etcd
+ etcd-6 › etcd
etcd-6 etcd etcd 21:27:51.44 INFO  ==>
etcd-6 etcd etcd 21:27:51.44 INFO  ==> Welcome to the Bitnami etcd container
etcd-6 etcd etcd 21:27:51.44 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
etcd-6 etcd etcd 21:27:51.45 INFO  ==> Did you know there are enterprise versions of the Bitnami catalog? For enhanced secure software supply chain features, unlimited pulls from Docker, LTS support, or application customization, see Bitnami Premium or Tanzu Application Catalog. See https://www.arrow.com/globalecs/na/vendors/bitnami/ for more information.
etcd-6 etcd etcd 21:27:51.45 INFO  ==>
etcd-6 etcd etcd 21:27:51.45 INFO  ==> ** Starting etcd setup **
etcd-6 etcd etcd 21:27:51.46 INFO  ==> Validating settings in ETCD_* env vars..
etcd-6 etcd etcd 21:27:51.47 INFO  ==> Initializing etcd
etcd-6 etcd etcd 21:27:51.47 INFO  ==> Generating etcd config file using env variables
etcd-6 etcd etcd 21:27:51.48 INFO  ==> Detected data from previous deployments
etcd-6 etcd etcd 21:27:51.60 INFO  ==> member id: 9f34b836f55aa4e2
etcd-6 etcd etcd 21:27:52.18 INFO  ==> Cluster is healthy

@carrodher carrodher assigned juan131 and unassigned carrodher Mar 14, 2025
@carrodher carrodher removed the triage Triage is needed label Mar 14, 2025
@juan131
Copy link
Contributor

juan131 commented Mar 17, 2025

@alop what chart version are you upgrading from and to?

@alop
Copy link

alop commented Mar 17, 2025

@juan131 No chart upgrade - using 9.8.1 - simply updating the image tag passed in from 3.5.14-debian-12-r4 to 3.5.18-debian-12-r7

@juan131
Copy link
Contributor

juan131 commented Mar 18, 2025

On image version 3.5.17-debian-12-r3 (see notable changes) we introduced some changes that affect upgrading, find more information in the links below:

If you want to upgrade to 3.5.18-debian-12-r7 we highly recommend you to previously upgrade your chart release to 11.y.z, otherwise there won't be a pre-upgrade job that solves the upgrading issues.

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 3, 2025

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

@github-actions github-actions bot added the stale 15 days without activity label Apr 3, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 8, 2025

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

@github-actions github-actions bot added the solved label Apr 8, 2025
@bitnami-bot bitnami-bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@juan131 juan131

Labels
etcd solved stale 15 days without activity tech-issues The user has a technical issue about an application
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@alop @juan131 @bitnami-bot @carrodher @saurabhnetskope