No exception / error logged though starting fails

[webdevilopers]

We use capistrano for deployment. It seems to have some problems with symlinks or similar.
But we could break it down to the main problem.

When we config our supervisor path to e.g. tmp or below the Symfony kernel root dir everything works as expected and supervisor gets started:

rabbit_mq_supervisor:
    supervisor_instance_identifier:     "%supervisor_instance_identifier%"
    paths:
        #workspace_directory:            "/tmp/supervisor/%supervisor_instance_identifier%/"
        workspace_directory:            "%kernel.root_dir%/../../../supervisor/%supervisor_instance_identifier%/"

But if we keep the supervisor dir inside the app folder for instance the starting fails.

In both cases the files are created correctly.

When I run the command on the server with the non-working dir I don't get any response - no log entries - no process starting:

SYMFONY_ENV=prod app/console rabbitmq-supervisor:control start

Though the exception has been added in #22 there still seems to be something missing.

Is there a debug or trace argument I can use? Or some way to hack into this:

• https://github.com/Phobetor/rabbitmq-supervisor-bundle/blob/master/Services/Supervisor.php#L98-L100

[webdevilopers]

Just recognized we were running the following version:

"phobetor/rabbitmq-supervisor-bundle": "2.0.0",

It thought that release
https://github.com/Phobetor/rabbitmq-supervisor-bundle/releases/tag/2.0.0
would alread include the Logger seen here:
https://github.com/Phobetor/rabbitmq-supervisor-bundle/blob/master/Services/Supervisor.php#L98-L100

But it doesn't. I must use dev-master to have it?!

[webdevilopers]

Finally after using dev-master I get some logs in Symfony:

[2018-03-10 10:18:27] app.DEBUG: Executing: supervisorctl --configuration=/var/www/vhosts/***/htdocs/releases/20180310091756/app/../supervisor/sps_intranet/supervisord.conf status [] []
[2018-03-10 10:18:27] app.CRITICAL: supervisorctl returns code: Misuse of shell builtins [] []
[2018-03-10 10:18:27] app.DEBUG: Output: error: <class 'socket.error'>, AF_UNIX path too long: file: /usr/lib64/python2.7/socket.py line: 224  [] []
[2018-03-10 10:18:27] console.ERROR: Exception thrown while running command "rabbitmq-supervisor:rebuild". Message: "Misuse of shell builtins" {"exception":"[object] (Phobetor\\RabbitMqSupervisorBundle\\Exception\\ProcessException(code: 2): Misuse of shell builtins at /var/www/vhosts/***/htdocs/releases/20180310091756/vendor/phobetor/rabbitmq-supervisor-bundle/Phobetor/RabbitMqSupervisorBundle/Services/Supervisor.php:67)","command":"rabbitmq-supervisor:rebuild","message":"Misuse of shell builtins"} []
[2018-03-10 10:18:27] console.ERROR: Command "rabbitmq-supervisor:rebuild" exited with code "2" {"command":"rabbitmq-supervisor:rebuild","code":2} []

The command:

supervisorctl --configuration=/var/www/vhosts/***/htdocs/releases/20180310100334/app/../supervisor/sps_intranet/supervisord.conf status

The error does not occur because of using "../". Since it does work when using "../../../".

The absolute path will fail too:

supervisorctl --configuration=/var/www/vhosts/***/htdocs/releases/20180310100334/supervisor/sps_intranet/supervisord.conf status

And again I want to note: The supervisor files are ALWAYS created. The path is correct. That means the permissions should be correct too.

Maybe it is not the path itself. But that the way it is configured causes this bundle to crash when trying to start supervisor.

[webdevilopers]

Misuse of shell builtins is thrown by:

• https://github.com/symfony/process/blob/master/Process.php#L91

Similar error messages here:

• Tests do not pass. Error code 2. "Misuse of shell builtins". STDERR infection/infection#61
• Misuse of shell builtins. spatie/laravel-backup#92
• Misuse of shell builtin spatie/laravel-backup#285

And here where it looks like it also happens while deployment to @anacona16, @The-Don-Himself or @Turtlesfr?

• Exit Code: 2 (Misuse of shell builtins) EasyCorp/easy-deploy-bundle#33

[webdevilopers]

At the bottom line it looks like the error occurs if the configured directories are none-permanent.
While a deployment with Capistrano a new folder is added under "releases". After deployment finished it is symlinked to "current".

Maybe the link does not exist for a short period of time - thought the absolute path inside the error looks correct and should work.

What I did not is that I moved the "supervisor" built files through other directories.
Event directories that would have worked as you can see in my original configuration above.

The error would still occur!

That makes me think that there is a problem inside the built files. I will post them here:

[webdevilopers]

; supervisor config file

[unix_http_server]
file=/var/www/vhosts/***/htdocs/releases/20180310101225/app/../supervisor/sps_intranet/supervisord.sock   ; (the path to the socket file)
chmod=0700             ; sockef file mode (default 0700)

[supervisord]
logfile=/var/www/vhosts/***/20180310101225/app/../supervisor/sps_intranet/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/www/vhosts/***/htdocs/releases/20180310101225/app/../supervisor/sps_intranet/supervisord.pid ; (supervisord pidfile;default supervisord.pid)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/www/vhosts/***/htdocs/releases/20180310101225/app/../supervisor/sps_intranet/supervisord.sock ; use a unix:// URL  for a unix socket

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

[include]
files = /var/www/vhosts/***/htdocs/releases/20180310101225/app/../supervisor/sps_intranet/worker/*.conf

Anything strange so far?

[webdevilopers]

Now i finally recognized that onky the .conf file was created. log, pid and .sock were missing.
That will cause the exception.

What I cannot tell is what causes them not to be created. Well I know that it does not work in newly created and later symlinked folders.

Maybe a permission problem while deployment?

[Phobetor]

Hi @webdevilopers

Thanks for all the information and first of all sorry for the confusion with 2.0.0 vs. dev-master. There is a new up-to-date version 3.0.0 out now.
You can use that version and console rabbitmq-supervisor:rebuild --wait-for-supervisord to get error output.

The .conf file is created by the commands of this bundle. The .log, .pid and .sock files are created by supervisord on start. If the latter do not exist the start of supervisord may have failed. It is hard to tell why that happened.

You write "it does work when using ../../../". That sounds like the other path is invalid for some reason. Maybe the directory does not exist or is not writable for the user that runs the command?

[Phobetor]

As there is no new information on this issue and it should be fixed in 3.0.0 I will close it. Feel free to open a new one if there are still problems with the current version.

[webdevilopers]

I tried to deploy with the new --wait-for-supervisord flag but get no new error messages.

The .conf file is created by the commands of this bundle. The .log, .pid and .sock files are created by supervisord on start.
If the latter do not exist the start of supervisord may have failed. It is hard to tell why that happened.

I agree. It really looks like a problem with the server process. The bundle looks fine actually.

Thanks for the help so far.

[webdevilopers]

I could now break down and re-produce our issue.

When our capistrano deployment executed the rebuild command it gives no error and supervisor is not restarted on the remote server.
When I use your new flag "wait-for-supervisord" supervisor IS restarted. But capistrano freezes waiting for a response infinitely.

However similar happens if I execute the commands outside of the deployment directly on the server.
Without the flag supervisor is not started. With the flag it is start and it DOES NOT HANG UP.

The conflict seems to be running the rebuild while being in the capistrano deployment process.

Ever stumbled upon this with other deployment tools?

Maybe these consumer options could help?

• Implement an ability to configure worker options #12 (comment)

What really disturbs me is the fact the the rebuild command doesn't even work on the server and supervisor is never restarted though the files are correctly rebuild.
The error logs are empty!

[vvasiloi]

Just had the same issue: seems like it fails when it cannot read or write the configuration files.
I was running rabbitmq-supervisor:rebuild --wait-for-supervisord command with www-data user in a Docker container. Executing chown -R www-data:www-data on the configuration folder solved the issue.

[webdevilopers]

Thanks @vvasiloi for your feedback. I no longer feel alone now! :)
TLDR I think we moved the configuration files to a different location where indeed no permission problems can occur. It worked out so far. Since then we never touched it again.

[vvasiloi]

If the issue is caused by not being able to read/write configuration files then it should throw a specific error, not one like "Misuse of shell builtins".
The bundle should check if it can read from/write to the configuration folder before running the supporvisor commands.
Will try to reproduce the issue and come with a PR later.