Supplemental GID missing for group with same name as user

[burgerdev]

Description

Originally, I observed this with a mongodb container. The image user is mongodb and there is a group mongodb, but it's not the primary group of the user (nogroup is). containerd omits the mongodb group from the supplemental GID list.

Steps to reproduce the issue

1. Create a GCP Debian Bookworm VM.
2. Install latest k3s (v1.32.5+k3s1, comes with containerd v2.0.5-k3s1.32¹).
3. Apply a specially crafted reproducer (source: https://github.com/burgerdev/weird-images/tree/74b32e2/src/gid)

   kubectl apply -f https://raw.githubusercontent.com/burgerdev/weird-images/74b32e268da916576f307f6f997ae289322f5aa6/src/gid/gid.yaml

Describe the results you received and expected

Observed

In the container logs, I see the UID, GID and supplemental groups:

Uid:	2	2	2	2
Gid:	1	1	1	1
Groups:	1 

Expected

The process should also have the supplemental group 2 (name).

What version of containerd are you using?

v2.0.5-k3s1.32

Any other relevant information

MongoDB setup

Image that I used when I discovered the bug: quay.io/mongodb/mongodb-community-server@sha256:8b73733842da21b6bbb6df4d7b2449229bb3135d2ec8c6880314d88205772a11

The unusual group setup comes from here, afaict:

https://github.com/mongodb/mongo/blob/cd9cc7450f3f665771a10f5c15f05bd6991c4d69/debian/mongodb-org-server.postinst#L25-L27

containerd bug

I assume it's this snippet that triggers the bug

containerd/pkg/oci/spec_opts.go

Lines 851 to 854 in bcd000f

	// we only want supplemental groups
	if g.Name == username {
	return false
	}

The code seems to assume that a group with the same name as the user is always the primary group of the user.

It might be possible to just delete this check, given that the primary ID is always added

containerd/pkg/oci/spec_opts.go

Line 833 in bcd000f

defer ensureAdditionalGids(s)

but only if it hasn't been added already

containerd/pkg/oci/spec_opts.go

Lines 127 to 135 in bcd000f

	func ensureAdditionalGids(s *Spec) {
	setProcess(s)
	for _, f := range s.Process.User.AdditionalGids {
	if f == s.Process.User.GID {
	return
	}
	}
	s.Process.User.AdditionalGids = append([]uint32{s.Process.User.GID}, s.Process.User.AdditionalGids...)
	}

Show configuration if it is related to CRI plugin.

No response

Footnotes

1. This was the simplest way for me to reproduce the bug. I know that it's a bit removed from upstream containerd main, but since I believe to have found the root cause (see last section), it may be sufficient as a demo. ↩

[thaJeztah]

Heh, I was actually looking at some of these earlier Today, in preparation of upstreaming some code that was forked in BuildKit and Moby (docker engine), but there's quite some complexity in those code-paths, and there's also multiple impleementations which seem to slightly differ (between the CRI code and non-CRI code, although for some one wraps the other);

containerd/internal/cri/server/container_create_linux.go

Lines 39 to 64 in 0bf07cd

	securityContext := config.GetLinux().GetSecurityContext()
	userstr := "0" // runtime default
	if securityContext.GetRunAsUsername() != "" {
	userstr = securityContext.GetRunAsUsername()
	} else if securityContext.GetRunAsUser() != nil {
	userstr = strconv.FormatInt(securityContext.GetRunAsUser().GetValue(), 10)
	} else if imageConfig.User != "" {
	userstr, _, _ = strings.Cut(imageConfig.User, ":")
	}
	switch securityContext.GetSupplementalGroupsPolicy() {
	case runtime.SupplementalGroupsPolicy_Merge:
	// merging group defined in /etc/passwd
	// and SupplementalGroups defined in security context
	specOpts = append(specOpts,
	customopts.WithAdditionalGIDs(userstr),
	customopts.WithSupplementalGroups(securityContext.GetSupplementalGroups()),
	)
	case runtime.SupplementalGroupsPolicy_Strict:
	// no merging group defined in /etc/passwd
	specOpts = append(specOpts,
	customopts.WithSupplementalGroups(securityContext.GetSupplementalGroups()),
	)
	default:
	return nil, fmt.Errorf("not implemented in this containerd release: SupplementalGroupsPolicy=%d", securityContext.GetSupplementalGroupsPolicy())
	}

Out of interest, I was wondering what a docker run would currently do (but I still need to look if it's currently depending on those code-paths from containerd, or using its own implementation);

Checking /etc/passwd and /etc/group;

docker run -it --rm --user mongodb mongodb/mongodb-community-server sh -c 'cat /etc/passwd | grep mongodb'
mongodb:x:101:65534::/home/mongodb:/usr/sbin/nologin

docker run -it --rm --user mongodb mongodb/mongodb-community-server sh -c 'cat /etc/group | grep mongodb'
mongodb:x:101:mongodb

docker run -it --rm --user mongodb mongodb/mongodb-community-server sh -c 'cat /etc/group | grep 65534'
nogroup:x:65534:

default user (as defined in the image)

docker run -it --rm mongodb/mongodb-community-server id
uid=101(mongodb) gid=65534(nogroup) groups=65534(nogroup),101(mongodb)

explicitly setting the user to "mongodb" (not passing a group);

docker run -it --rm --user mongodb mongodb/mongodb-community-server id
uid=101(mongodb) gid=65534(nogroup) groups=65534(nogroup),101(mongodb)

And with your reproducer image;

docker build \
    -t test \
    -f 'https://raw.githubusercontent.com/burgerdev/weird-images/74b32e268da916576f307f6f997ae289322f5aa6/src/gid/Containerfile' \
    'https://github.com/burgerdev/weird-images.git#74b32e268da916576f307f6f997ae289322f5aa6:src/gid'

docker run -it --rm test
Uid:	2	2	2	2
Gid:	1	1	1	1
Groups:	1 2
^C

docker run -it --rm --entrypoint="" test id
uid=2(name) gid=1(daemon) groups=1(daemon),2(name)

edit: also tried with docker build to see if it's correct;

echo -e 'FROM mongodb/mongodb-community-server\nRUN id' | docker build --progress=plain --no-cache -
# ...
#6 [2/2] RUN id
#6 0.325 uid=101(mongodb) gid=65534(nogroup) groups=65534(nogroup),101(mongodb)
#6 DONE 0.4s


echo -e 'FROM mongodb/mongodb-community-server\nUSER root\nRUN id\nUSER mongodb\nRUN id' | docker build --progress=plain --no-cache -
#5 [2/3] RUN id
#5 0.232 uid=0(root) gid=0(root) groups=0(root)
#5 DONE 0.3s

#6 [3/3] RUN id
#6 0.307 uid=101(mongodb) gid=65534(nogroup) groups=65534(nogroup),101(mongodb)
#6 DONE 0.4s

[burgerdev]

Thanks for investigating this, @thaJeztah. I also tested this with docker and can confirm your observation, but discounted it because I thought docker may be introducing some special treatment here. Now I too wonder which code path results in the correct additionalGIDs - I've only found the one with the bug and one that does not look at /etc/group.

[thaJeztah]

Thanks! Yes, it was also a bit "note to self" to compare the behavior; I had some "half-completed" branches to try and unify more of the implementations; especially to avoid (sometimes subtle) differences in behavior.

Big disclaimer that I'm not a kubernetes user, so it's possible that some differences were by design (kubernetes/enhancements#3620), but I'm also aware that correct behavior for (supplemental) groups were a bit under-defined (also see opencontainers/image-spec#1011, opencontainers/runtime-spec#1180 for example).