High CPU usage in scheduler

[tanjianfeng]

With below program, we find that the scheduler shows ~380% CPU usage vs ~200% in runc.

How to test:
$ docker run -it --cpu-period=1000 --cpu-quota=8000 ...
$ gcc -o threads thread.c -lpthread
$ ./threads 1024 100000

#include <stdio.h>
#include <pthread.h>
#include <sys/time.h>
#include <time.h>
#include <string.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdint.h>

struct worker_arg {
        int seq;
        int iterations;

        pthread_mutex_t count_lock;
        pthread_cond_t count_nonzero;
        unsigned count;
};


static void
park(struct worker_arg *arg)
{
        pthread_mutex_lock(&arg->count_lock);
        while (arg->count == 0)
                pthread_cond_wait(&arg->count_nonzero, &arg->count_lock);
        arg->count--;
        pthread_mutex_unlock(&arg->count_lock);
}

static void
unpark(struct worker_arg *arg)
{
        pthread_mutex_lock(&arg->count_lock);
        if (arg->count == 0)
                pthread_cond_signal(&arg->count_nonzero);
        arg->count++;
        pthread_mutex_unlock(&arg->count_lock);
}

static void *
worker(void *p)
{
        struct worker_arg *arg = (struct worker_arg *)p;

        while (arg->iterations--) {
                /* do some work */
                //printf("%d %d\n", arg->seq, arg->iterations);
                park(arg);
        }

        return NULL;
}

int
main(int argc, char **argv)
{
        int i, j, N, M;
        void *dummy;
        struct worker_arg *args;
        pthread_t *tids;

        if (argc != 3) {
                fprintf(stderr, "Usage: %s <number of threads> <iterations>\n", argv[0]);
                exit(1);
        }

        N = atoi(argv[1]);
        M = atoi(argv[2]);

        tids = malloc(sizeof(*tids) * N);
        args = malloc(sizeof(*args) * N);

        for (i = 0; i < N; i++) {
                args[i].seq = i;
                args[i].iterations = M;

                args[i].count = 0;
                pthread_mutex_init(&args[i].count_lock, NULL);
                pthread_cond_init(&args[i].count_nonzero, NULL);

                pthread_create(&tids[i], NULL, worker, &args[i]);
        }

        for (i = 0; i < M; i++) {
                for (j = 0; j < N; j++)
                        unpark(&args[j]);
        }

        for (i = 0; i < N; i++)
                pthread_join(tids[i], &dummy);
}

[prattmic]

When you say 'CPU in the scheduler', I assume you mean the host Linux kernel scheduler?

Which platform (or platforms) do you see this behavior on?

[tanjianfeng]

When you say 'CPU in the scheduler', I assume you mean the host Linux kernel scheduler?

I mean scheduler in broader meaning. The example above uses futex syscall to wait/wakeup threads. So it involves (1) futex implementation in sentry; (2) golang sheduler; (3) maybe also linux host CFS.

Which platform (or platforms) do you see this behavior on?

kvm platorm.

[prattmic]

For reference, this workload tends to suffer from golang/go#43997.

[github-actions]

A friendly reminder that this issue had no activity for 120 days.

[ayushr2]

golang/go@ecfce58 must have helped with this. Is this still an issue?

[github-actions]

A friendly reminder that this issue had no activity for 120 days.

[github-actions]

This issue has been closed due to lack of activity.