Skip to content

feat(test): use labels in meta e2e test #377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Mar 28, 2023
Merged

feat(test): use labels in meta e2e test #377

merged 29 commits into from
Mar 28, 2023

Conversation

CAJan93
Copy link
Contributor

@CAJan93 CAJan93 commented Mar 3, 2023

What's changed and what's your intention?

Follow-up from #352

Checklist

  • I have written the necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

Copy link
Collaborator

@arkbriar arkbriar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CAJan93 CAJan93 marked this pull request as ready for review March 6, 2023 08:32
@codecov
Copy link

codecov bot commented Mar 6, 2023

Codecov Report

Merging #377 (f244b54) into main (f683f47) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #377      +/-   ##
==========================================
- Coverage   71.61%   71.58%   -0.04%     
==========================================
  Files          53       53              
  Lines        5641     5635       -6     
==========================================
- Hits         4040     4034       -6     
  Misses       1529     1529              
  Partials       72       72              
Flag Coverage Δ
unittests 71.58% <100.00%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/factory/risingwave_object_factory.go 86.44% <100.00%> (-0.06%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

leader="$p"
# Iterate over the etcd election kv pairs. Delete leader lease if found, else abort the test
local del_lease=false
kubectl -n rwc-2-mytenant run del-leader-lease --image='anldisr/etcdctl' --restart='Never' --command=true -- etcdctl --endpoints=risingwave-etcd-mytenant:2379 get _ --prefix=true --write-out="json"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we can utilize kubectl exec to execute the command without the need to fetch the image for this particular pod. For example:

$ k exec -it etcd-0 -- env ETCDCTL_API=3 etcdctl get a --prefix=true --write-out="json" --user=root:iGGUrEKBiX
{"header":{"cluster_id":16217987567572491554,"member_id":17870784734561721408,"revision":1,"raft_term":3}}

In addition, the use of get _ could be overly aggressive and may result in timeouts. To avoid this issue, it would be better to implement lease list as an alternative solution.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As always thank you for your guidance 😁

I totally agree with you, using exec instead of spewing a new pod is definitely better.

Regarding performance: I am not sure how we could use lease list in this case. That command returns

{
  "cluster_id": 6604019301136558000,
  "member_id": 1109445486625178000,
  "revision": 13954,
  "raft_term": 4,
  "leases": [
    {
      "id": 4734838729492364000
    },
    {
      "id": 4734838729492364000
    },
    {
      "id": 4734838729492364000
    }
  ]
}

We receive the leases, but we do not know which keys are attached to these leases. I do not see how this benefits us here. Please let me know if I miss something.

I will use etcdctl get __meta_election_ --prefix=true from now on. Maybe that is a bit more performant, since we narrow down the search a bit more

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind. I believe etcdctl get __meta_election_ --prefix=true already solves the problem. I didn't know more about the lease list than you.

Copy link
Collaborator

@arkbriar arkbriar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM! BTW, I found some nits. Would you mind fixing them before merging?

del_lease=true
echo "found leader lease. Deleting it"
# delete leader lease
k8s::kubectl exec -it etcd-0 -- env ETCDCTL_API=3 etcdctl del "$(echo "$i" | jq -r .key | base64 --decode)" --user=root:iGGUrEKBiX
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The iGGUrEKBiX is the password from my local... I presume it won't work in the tests. Could you try removing the --user and see if it works?

@arkbriar
Copy link
Collaborator

arkbriar commented Mar 13, 2023

It seems the e2e test failed with an explicit reason:

Unable to use a TTY - input is not a terminal or the right kind of file
Error: context deadline exceeded
command terminated with exit code 1
./test/e2e/tests/risingwave/meta.sh: line 17: : No such file or directory
[ERRO] [e2e] [ns/open-kruise-e2e-3] [risingwave::multi_meta_failover_fencing] Could not delete leader lease
[ERRO] [e2e] [ns/open-kruise-e2e-3] [risingwave::multi_meta_failover_fencing] Failed to delete leader lease
[ERRO] [e2e] [ns/open-kruise-e2e-3] [risingwave::multi_meta_failover_fencing] [cost: 1m01s] Failed!

Could you help have a look @CAJan93?

@CAJan93
Copy link
Contributor Author

CAJan93 commented Mar 15, 2023

I think this is blocked by risingwavelabs/risingwave#8238
I am currently getting Leader did not restart

@CAJan93
Copy link
Contributor Author

CAJan93 commented Mar 28, 2023

Tried this with the nightly-20230327 image and ./test/e2e/e2e.sh -p risingwave::multi_meta_failover_fencing passed

@CAJan93 CAJan93 merged commit 183a315 into main Mar 28, 2023
@CAJan93 CAJan93 deleted the update_e2e_test branch March 28, 2023 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants