Rootless Podman Quadlet #3487
Replies: 41 comments 105 replies
-
Known issuesI've encountered some problems, however it is unknown whether these are actual bugs related to Podman or just usage errors.
For, now, you can manually update the mastercontainer by running systemctl --user stop nextcloud-aio-mastercontainer.service
podman pull docker.io/nextcloud/all-in-one:latest
systemctl --user start nextcloud-aio-mastercontainer.service |
Beta Was this translation helpful? Give feedback.
-
|
FYI Podman 4.8.0 has been released a few days ago and is already present in the Fedora 39 repos. Should be only a matter of the CoreOS auto updater to pick that up. |
Beta Was this translation helpful? Give feedback.
-
|
@jennydaman Your quick fix doesn't work anymore as @szaimen did this commit: f935993 |
Beta Was this translation helpful? Give feedback.
-
|
@p-fruck and @loeffelpan done |
Beta Was this translation helpful? Give feedback.
-
|
Is there any workarounds for borgBackup? In fact, the database always starts with: |
Beta Was this translation helpful? Give feedback.
-
|
Why don't you change appearances of the user UID in |
Beta Was this translation helpful? Give feedback.
-
|
Will this solution also work with a Podman compose file? What is de advantage of using a systemd unit file? |
Beta Was this translation helpful? Give feedback.
-
|
On the surface level, the main advantage of Quadlet is convenient for people who like systemd and are already comfortable with its syntax. |
Beta Was this translation helpful? Give feedback.
-
|
@jennydaman: thanks for the clarification! I didn't know that it was possible to start containers (with Podman) in unit files! Seems like a clean/efficient solution! |
Beta Was this translation helpful? Give feedback.
-
|
Soft reset instructions systemctl --user stop nextcloud-aio-mastercontainer
containers=$(podman ps -a -f 'name=^nextcloud-aio' --format='{{.Names}}')
podman stop $containers
podman rm $containers
systemctl --user start nextcloud-aio-mastercontainer |
Beta Was this translation helpful? Give feedback.
-
|
Hey, I'm using podman (rootless) in favor of docker on FCOS. My first attempt to get this working like described above ended in an error at domain check. Maybe this is a networking issue as caddy is proxying to Next step was to skip domaincheck but that ended in an Bonus: |
Beta Was this translation helpful? Give feedback.
-
|
Hey there @loeffelpan,
are you running behind cloudflare? If so, this might be the reason for your domaincheck issues. When disabling the domaincheck, do the application errors look similar to the ones described in this comment? #3487 (reply in thread)
If so, this will be fixed in a newer version of podman. 4.7.0 doesn't contain this fix and I haven't tested 4.7.2 until yet, but I don't think it includes my fix already.
Until the fixed podman version is released, you can also patch the source code of your nextcloud AIO as described by @jennydaman here: #3487 (reply in thread)
|
Beta Was this translation helpful? Give feedback.
-
|
How do you turn off domaincheck? Can I use this? Environment=SKIP_DOMAIN_VALIDATION=true |
Beta Was this translation helpful? Give feedback.
-
|
@kri164 Yes. Correct. |
Beta Was this translation helpful? Give feedback.
-
|
@p-fruck You're correct. With Is the need of If I an correct this should be mentioned in the initial post for anyone who is joining this discussion. |
Beta Was this translation helpful? Give feedback.
-
|
Interestingly, my local setup is created with the network in bridge mode by default (podman 4.7.0). Might have to do with my system default (Fedora CoreOS 39 beta), though I did not check any further. @loeffelpan what exactly would you like to mention? Afaict the default network mode of podman might deviate depending on how the given linux distribution packages podman, so I would simply mention that bridge network must be used (as already done by @jennydaman ) |
Beta Was this translation helpful? Give feedback.
-
|
I think this should not differ between distros. I'm also on podman 4.7.0 with FCOS 38. I was pointing to this one from the initial post.
There is Allover, I think @jennydaman should modify this as simply needed and not a workaround. Regardless of distro/package differences or rootful/rootless setups. |
Beta Was this translation helpful? Give feedback.
-
|
podman -v podman logs nextcloud-aio-mastercontainer |
Beta Was this translation helpful? Give feedback.
-
|
This error seems like its the same I mentioned above, related to podman. |
Beta Was this translation helpful? Give feedback.
-
|
I'm confused that this question has not beeing asked. Maybe I missed something. |
Beta Was this translation helpful? Give feedback.
-
|
Thats quite a good idea. Thanks! @szaimen |
Beta Was this translation helpful? Give feedback.
-
|
@loeffelpan and @szaimen thanks for bringing |
Beta Was this translation helpful? Give feedback.
-
|
In my case (on FCOS) |
Beta Was this translation helpful? Give feedback.
-
|
Can confirm that its disabled by default. This should not be anything CoreOS specific though. You can enable it in the ignition file for new installations, but a simple |
Beta Was this translation helpful? Give feedback.
-
|
I did this. But for some reason (maybe zincati update finaliziation?) its disabled again. Any idea? |
Beta Was this translation helpful? Give feedback.
-
|
Anyone using collabora in this setup? Without collabora nextcloud is working fine. |
Beta Was this translation helpful? Give feedback.
-
|
Not using collabora yet but I had some DNS issues trying to access my IdP when configuring OIDC for SSO. Can you try to |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
It depends. For me curl was working but the php request didn't until I executed nslookup for the domain once. Apparently the dns entry had to be cached first for some odd reason |
Beta Was this translation helpful? Give feedback.
-
|
To solve this I started another discussion which brought me on the right track. So running collabora in our rootless setup isn't possible right now - regardless docker or podman. |
Beta Was this translation helpful? Give feedback.
-
|
I followed the above steps on Fedora Silverblue 39 and was unable to start the |
Beta Was this translation helpful? Give feedback.
-
|
no dice for me, unfortunately |
Beta Was this translation helpful? Give feedback.
-
|
Is SELinux enabled on Silverblue? As it's basically Fedora, it should. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe this helps? https://github.com/nextcloud/all-in-one#are-there-known-problems-when-selinux-is-enabled |
Beta Was this translation helpful? Give feedback.
-
|
Indeed that would help. But disabling security is not my way to go. I'm looking for a way to allow socket access by rule for SELinux instead. |
Beta Was this translation helpful? Give feedback.
-
|
Honestly, I feel like running with SELinux in enforcing mode and disabling security opts for the master container only might be the way to go. If you want to build your own SELinux policy, maybe this reply can help ;) |
Beta Was this translation helpful? Give feedback.
-
|
Some notes from my last install. Hope it helps. cgroupsRunning containers as rootless requires cgroups version 2. Version 1 or hybrid mode v1/v2 doesnt work. Check the status SELinuxCheck the status with command NetavarkTodays versions of Podman comes with Netavark module, but existing installations are configured for CNI. If you have problems with intercontainer DNS resolution, switch it. Backup restoreBackup restore doesnt work during initial installation. Error is "Permission denied". Not sure if common bug or rootless install only. As a workaround, you can symlink backup directory to /tmp and use it for import |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for these instructions. I was able to get this working through cloudflare tunnel by adding a "Bypass - include everyone" rule for the zero trust app. (Allow - include everyone did not work because it continued to ask for cloudflare OTP, which broke the mobile apps with "Malformed server configuration"). I will need to make modifications to config.php, namely |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I don't think so but you could use some kind of script/init-container which executes the |
Beta Was this translation helpful? Give feedback.
-
|
For future reference, I solved this (from the host OS) by editing the file at |
Beta Was this translation helpful? Give feedback.
-
|
Fedora 39 here. Any ideas? |
Beta Was this translation helpful? Give feedback.
-
|
Maybe this one help https://github.com/dpw/selinux-dockersock |
Beta Was this translation helpful? Give feedback.
-
|
In case you do not want to deal with SELinux policies, you might want disable SELinux labeling for socket as explained in the docs |
Beta Was this translation helpful? Give feedback.
-
|
@p-fruck That seems to work. |
Beta Was this translation helpful? Give feedback.
-
|
Any way to create a policy on Fedora Core OS? I can't figure it out. |
Beta Was this translation helpful? Give feedback.
-
|
I think this question should be asked in the podman discussions. There official answer seems to be disabling SELinux labels for the specified container though. Feel free to link to link your discussion here ;) |
Beta Was this translation helpful? Give feedback.
-
|
On any mastercontainer update I got this error (in the logs of watchtower container): Same on starting newly created mastercontainer. |
Beta Was this translation helpful? Give feedback.
-
|
Am I the only one with problems on updates (of mastercontainer or any other container of aio)? |
Beta Was this translation helpful? Give feedback.
-
|
for me, at the moment, automated updates always fail to restart containers, and occasionally crash the ncaio container as well. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah. Containers can be created but not started. Seems due to the above error concerning memory swapiness. Maybe someone can take a look? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure that I have the same issue as you, although the symptoms are similar (ncaio crashing or not starting containers) here are my latest watchtower logs. note the date is feb 2 although just this morning (feb 4) I experienced aio stopping all containers and failing to start them If the problem is docker.io limits, I can temporarily solve by authenticating, but it seems like something deeper is going on here |
Beta Was this translation helpful? Give feedback.
-
|
You're right. Seems to be not the same error. |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
-
|
Hey! It seems that for some reason podman needs catatonit in order to create other containers. I didn't know that it could have saved a couple of days of troubleshooting. I'm just documenting it here in case that someone has the same problem! |
Beta Was this translation helpful? Give feedback.
-
|
I got stuck on the first step. I can't even install it successfully. than i got error: |
Beta Was this translation helpful? Give feedback.
-
|
Re-read the instructions. Specifically: Change /run/user/1001/podman/podman.sock to the path of your Podman socket (podman info --format '{{ .Host.RemoteSocket.Path }}') There is no |
Beta Was this translation helpful? Give feedback.
-
|
You actually can do this if you enable to podman socket as root and having the podman-docker package installed or just point directly to the podman socket. The problem seems to be that the socket doesn't work with SELinux enabled, even with passing |
Beta Was this translation helpful? Give feedback.
-
|
Here's my currently working example: |
Beta Was this translation helpful? Give feedback.
-
It's working for me |
Beta Was this translation helpful? Give feedback.
-
|
For context, I believe this is similar to why you also need to disable SELinux when you're doing podman in Kubernetes as well. I realize that it isn't ideal for everyone, but it is a reality for some similar podman deployment operations and not only a Nextcloud AIO issue. (In this case, you do not need to disable SELinux on the host, but just disable it in the specific podman run!)
|
Beta Was this translation helpful? Give feedback.
-
|
Hello! I'm attempting to run Nextcloud in this fashion but I have some concerns. For context, my environment is is using Podman Quadlets on a Fedora VM. Users and permissions are managed using FreeIPA. All persistent volumes are stored on NFS shares that are also Kerberized. I am running this as a specific user using the above configurations (although I am heavily considering the manual installation in a Pod to allow this to run rootless under root). The reason I bring this up is that I can't actually start the Nextcloud container itself as it can't chown the data directory to Note:
|
Beta Was this translation helpful? Give feedback.
-
|
Ok I think I just need to map UID 33 into the container, but I'm figuring out how to get NFS to export UID 33. |
Beta Was this translation helpful? Give feedback.
-
|
Ref. https://www.redhat.com/sysadmin/rootless-podman-nfs but you have an extra complexity with sssd. |
Beta Was this translation helpful? Give feedback.
-
|
Also, this discussion: containers/podman#16018 |
Beta Was this translation helpful? Give feedback.
-
|
I created a tool to manage podman and quadlets from repositories, and with it this installation can be run like so: The tool is using git repositories with folders containing all the quadlet files. The default the tool works with is https://github.com/rgolangh/podman-quadlets , but you can use your own of course. If you're interested please check-out https://github.com/rgolangh/pq - PRs or issues/discussions and feedback are welcomed. |
Beta Was this translation helpful? Give feedback.
-
|
This is cool - it seems like a sort of helm chart way to deploy things with podman. Do you also have a way to update the existing deployment with newer image refs? |
Beta Was this translation helpful? Give feedback.
-
|
the beauty is that podman does that for you using podman auto-update.service and to opt-in just add this to the quadlet file: This should result in a label on the container podman creates to be picked up by auto-update. |
Beta Was this translation helpful? Give feedback.
-
|
Excellent work! I love that you also have a repo ref for Valkey as well! |
Beta Was this translation helpful? Give feedback.
-
|
When going into https://127.0.0.1:8443 I get an |
Beta Was this translation helpful? Give feedback.
-
|
Can you post your caddyfile? |
Beta Was this translation helpful? Give feedback.
-
|
After updating to version 30.0.2, the AppAPI feature is flooding my logs with errors related to accessing the docker.sock. Suggestions? Disable AppAPI? Point it in the right direction somehow? |
Beta Was this translation helpful? Give feedback.
-
|
Install podman-docker and |
Beta Was this translation helpful? Give feedback.
-
|
Both were already installed prior to this update. Are you suggesting podman.socket needs to be enabled for the root user? podman.socket is enabled as the podman user. My intent is to use podman as a rootless user. |
Beta Was this translation helpful? Give feedback.
-
Do you have DOCKER_HOST set? and in the unit file of nextcloud: |
Beta Was this translation helpful? Give feedback.
-
|
It is worth making sure podman.socket is active as the user |
Beta Was this translation helpful? Give feedback.
-
|
Anyone managed to make onlyoffice work? I cannot access the host from within the container. |
Beta Was this translation helpful? Give feedback.
-
|
Hi ! Could you PLEASE provide a "caddy.container" file for your setup ? EDIT: Guess i've got it: Did forget to set "network=host" in caddy.container 🤦 |
Beta Was this translation helpful? Give feedback.
-
About self/automatic updates
Not tried in practice, yet, but is not this a solved problem in the podman world? To use it you basically just need to add a label to the containers ( (IMHO; this is also better than how the Docker world does it, it's all handled by podman – it could apparently even handle rollbacks.) Maybe you can add this to the guide, @jennydaman? |
Beta Was this translation helpful? Give feedback.
-
|
I didn't know about podman-auto-update before, thanks for sharing! As a sysadmin I'm not comfortable with the idea of automatic updates. Nextcloud-AIO has a self update feature, which is different from automatic updates:
People who are interested in auto-updates should be able to find your comment with a quick CTRL-F |
Beta Was this translation helpful? Give feedback.
-
|
One can also add the You could add it as a commented out parameter in the description: [Unit]
Description=Nextcloud AIO Master Container
Documentation=https://github.com/nextcloud/all-in-one/blob/main/docker-rootless.md
After=local-fs.target
Requires=podman.socket
[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=docker.io/nextcloud/all-in-one:latest
PublishPort=127.0.0.1:11001:8080
Volume=nextcloud_aio_mastercontainer:/mnt/docker-aio-config
Volume=/run/user/1001/podman/podman.sock:/var/run/docker.sock:ro
Network=bridge
SecurityLabelDisable=true
# AutoUpdate=registry # Uncomment if you want the AIO container to be automatically updated
Environment=APACHE_PORT=11000
Environment=APACHE_IP_BINDING=127.0.0.1
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1001/podman/podman.sock
[Install]
WantedBy=multi-user.target default.target |
Beta Was this translation helpful? Give feedback.
-
|
Updated the master container and can't start the container with the following error: Anyone else got issues or a fix? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Seems like the nextcloud-aio-mastercontainer is attaching to the default podman network? |
Beta Was this translation helpful? Give feedback.
-
So something in the Rocky Linux update has broken rootless networking. Gets me to the login page for the mastercontainer but results in a Slim application error on login. |
Beta Was this translation helpful? Give feedback.
-
|
It was wireguard! Rootless NetNS routing table was borked. Removing the wireguard interface from network manager and rebooting has restored internet. Will investigate further when I have more time. |
Beta Was this translation helpful? Give feedback.
-
|
Weird, I failed to make it work with |
Beta Was this translation helpful? Give feedback.
-
|
Updating the AIO master container from version 10.9.0 to a newer version that uses Suspected reason: The new version of This can be verified using the command: Rolling back to the old master container version fixes the issue, so I suspect that changes in the master container Docker image have broken compatibility with Podman. If someone have same issue and have solution - please share it :) Client: Podman Engine |
Beta Was this translation helpful? Give feedback.
-
|
The issue was that during the update process, Podman did not remove the old master container when pulling the new image from the new image registry. As a result, two containers with the same name existed in the system. This broke Podman's network handling logic, causing the "second" container not to join the correct network — and this issue was not logged in any way. Solution: Manually delete all containers downloaded from Docker Hub using the CLI. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for this discussion. For me, no trouble with the network. Only one with "double" ports because a unifi quadlet already used them and nextcloud talk could not start. Disable talk and all works. However, I can't see how on the initial post step 5 ( My current container file: I used traefik with this config (just the nextcloud parts) |
Beta Was this translation helpful? Give feedback.
-
|
I am using CoreOS, podman quadlets and caddy as a reverse proxy. My domain verification fails currently. in the AOI interface, I get this error when I want to confirm cloud.domain.be:
What's wrong here? |
Beta Was this translation helpful? Give feedback.
-
|
Recent podman releases will default to pasta for rootless networking. I had the same problem where the public ip for my domain lives on the same host as my rootless nextcloud setup, where I couldn't connect to it from within any container. The workaround was to revert back to slirp4netns instead. Now edit Reboot and all of your rootless containers should be able to connect with all host interfaces. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the idea. I think how I could implement a smaller work around that may not impact my other podman containers.
Edit: I forgot that the domaincheck container is created automatically and I do not know how I could add Edit2: As I have setup a specific unix user for nextcloud, I could configure slirp4netns only for this user using a file Note that CoreOS comes with slirp4netns already installed. Awesome! |
Beta Was this translation helpful? Give feedback.
-
|
That's correct as the Creating a separate user for nextcloud with a Because slirp4netns seems to perform a little worse compared to pasta in terms of CPU usage, it's good to limit it only to nextcloud and a couple of other containers which require connecting to the host interfaces. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if helpfull I use this setup with traefik instead caddy and I'm realy suprised how well it works. |
Beta Was this translation helpful? Give feedback.
-
|
I feel a little bit suspicious by the use of As you can read on podman documentation this practice is discouraged if the container is listening on TCP. I don't know if this is the case because usually it's hidden behind a reverse proxy, so is this causing a security issue? |
Beta Was this translation helpful? Give feedback.
-
|
I think you misunderstand the documentation. The Podman documentation you linked advises against exposing the Podman API over TCP, and it implies exposing the Podman API over a Unix socket is acceptable. Here, the nextcloud-aio-mastercontainer is given access to the Podman API over a Unix socket, mounted as a volume, and the nextcloud-aio API is exposed over TCP. In other words, we are not exposing the Podman API over TCP. |
Beta Was this translation helpful? Give feedback.
-
|
In any case, keep in mind that Docker is insecure by default (dockerd runs as root), and the nextcloud-aio project targets Docker. When we use Podman as a substitute in projects meant for Docker, it is often necessary to cripple Podman’s security to have it mimic Docker’s behavior. |
Beta Was this translation helpful? Give feedback.
-
|
@jennydaman thanks for this insight. I think I’ll try to set up Nextcloud manually, not with aio I mean, and put all in a pod. Did you try a similar setup? |
Beta Was this translation helpful? Give feedback.
-
|
Haven’t tried it myself. Sounds like a big improvement to efficiency at the cost of maintenance. |
Beta Was this translation helpful? Give feedback.
and others
-
This guide sets up Nextcloud-AIO using Podman in rootless mode and Quadlet behind a reverse proxy.
0. Install Podman
Podman version 4.8.0 or above is required. Older versions of Podman require workarounds, see edit history and discussion below.
1. Set up a reverse proxy
Consult the upstream documentation on how to do this: https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
An example
Caddyfilemight look like this:2. Create Systemd Unit File
Create the file
~/.config/containers/systemd/nextcloud-aio-mastercontainer.containerwith the following content:Important notes:
/run/user/1001/podman/podman.sockto the path of your Podman socket (podman info --format '{{ .Host.RemoteSocket.Path }}')nextcloud-aio. The optionNetwork=bridgeenables this behavior, for details see podman network connect not implemented for slirp4netns containers/podman#19577 (comment)Also, create the file
~/.config/containers/systemd/nextcloud-aio-mastercontainer.volume:3. Start the Services
4. (Optional) Configure Containers to Restart After Reboot
Podman is daemonless so unlike Docker, containers do not restart automatically after reboot.
To enable restart of Podman containers after reboot, see containers/podman#20418 (comment)
5. Business As Usual
Go to https://cloud.example.com:8443 to access the Nextcloud AIO interface and start the Nextcloud server.
Notes
Backups and mastercontainer self-updating might not work, these details have yet to been sorted out.
Beta Was this translation helpful? Give feedback.
All reactions