Skip to content

Add a lock around cgroupd communication. #2620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 13, 2025

Conversation

hanwen-flow
Copy link
Contributor

Threads are put into cgroups through the cgroupd thread, which communicates with other threads using a socketpair. Each thread received a dup'd copy of the socket, and did the following

sendmsg(socket_dup_fd, my_cgroup_set);

// wait for ack.
while (1) {
    recvmsg(socket_dup_fd, &h, MSG_PEEK);
    if (h.pid != my_pid) continue;
    recvmsg(socket_dup_fd, &h, 0);
}
close(socket_dup_fd);

When restoring many threads, most threads would be spinning in the above loop waiting for their PID to appear.

In my test-case, restoring a process with a 11.5G heap and 491 threads could take anywhere between 10 seconds and 60 seconds to complete.

To avoid the spinning, we drop the loop and MSG_PEEK, and add a lock around the above code. This does not decrease parallelism, as the cgroupd daemon uses a single thread anyway.

With the lock in place, the same restore takes a consistent 9.2 seconds on my machine (Thinkpad P14s, AMD Ryzen 8840HS).

There is a similar "daemon" thread for user namespaces. That already is protected with a similar userns_sync_lock in __userns_call().

Fixes #2614

@hanwen-flow
Copy link
Contributor Author

PTAL

Copy link
Member

@minhbq-99 minhbq-99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 3 empty lines which have the tabs at the beginning so the linter is complaining.

I think the commit title should be "restorer: Add a lock around cgroupd communication" to follow the project's convention.

These are nits. Overall, the approach LGTM.

@hanwen-flow
Copy link
Contributor Author

thanks, PTAL.

I tried running 'make lint', but it wants to run 'ruff' which isn't available on Ubuntu 24.

Threads are put into cgroups through the cgroupd thread, which
communicates with other threads using a socketpair.

Previously, each thread received a dup'd copy of the socket, and did
the following

    sendmsg(socket_dup_fd, my_cgroup_set);

    // wait for ack.
    while (1) {
        recvmsg(socket_dup_fd, &h, MSG_PEEK);
        if (h.pid != my_pid) continue;
        recvmsg(socket_dup_fd, &h, 0);
    }
    close(socket_dup_fd);

When restoring many threads, many threads would be spinning in the
above loop waiting for their PID to appear.

In my test-case, restoring a process with a 11.5G heap and 491 threads
could take anywhere between 10 seconds and 60 seconds to complete.

To avoid the spinning, we drop the loop and MSG_PEEK, and add a lock
around the above code. This does not decrease parallelism, as the
cgroupd daemon uses a single thread anyway.

With the lock in place, the same restore consistently takes around 10
seconds on my machine (Thinkpad P14s, AMD Ryzen 8840HS).

There is a similar "daemon" thread for user namespaces. That already
is protected with a similar userns_sync_lock in __userns_call().

Fixes checkpoint-restore#2614

Signed-off-by: Han-Wen Nienhuys <[email protected]>
@rst0git
Copy link
Member

rst0git commented Mar 13, 2025

it wants to run 'ruff' which isn't available on Ubuntu 24.

Have you tried pip3 install ruff?

@avagin avagin merged commit 8d5cef5 into checkpoint-restore:criu-dev Mar 13, 2025
35 of 41 checks passed
@avagin
Copy link
Member

avagin commented Mar 13, 2025

Merged. Thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Restoring threads very slow.
4 participants