Skip to content

EADDRINUSE during a reload #1596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
julien6387 opened this issue Jul 28, 2023 · 1 comment
Closed

EADDRINUSE during a reload #1596

julien6387 opened this issue Jul 28, 2023 · 1 comment

Comments

@julien6387
Copy link
Contributor

I'm using Supervisor 4.2.4 and I have a use case where I can repeat the following failure almost 100% of the time when restarting Supervisor:

Error: Another program is already listening on a port that one of our HTTP servers is configured to use.  Shut this program down first before starting supervisord.
For help, use /usr/local/bin/supervisord -h
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)

I suspect it happens when the HTTP socket is still under use when the close() is called on the socket.
As long as there is a handle on the socket, the operating system will not actually deallocate the "closed" socket.

Indeed, I am using a lot of REMOTE_COMMUNICATION_EVENT that make the HTTP socket a bit busy.
When the Supervisor restart is called, the socket is most of the time not deallocated, although it has been closed.
The restart being swift, the socket bind cannot reuse the port (still in TIME_WAIT) and fails.

The socket has to be shutdown before it's closed, so that the FIN event is sent to the peers.
Then the socket close triggers the deallocation quite immediately.
Once applied, I never had the issue again and it's good practice anyway.

@julien6387
Copy link
Contributor Author

There is a comment in ServerOptions.close_httpservers stating that:

            # For unknown reasons, sometimes an http_channel
            # dispatcher in the socket map related to servers
            # remains open *during a reload*.

It might be related somehow.

mnaberez added a commit that referenced this issue Aug 9, 2023
mnaberez added a commit that referenced this issue Mar 2, 2024
```
error: uncaptured python exception, closing channel <supervisor.http.deferring_http_channel connected 127.0.0.1:58792 at 0x102d3a310 channel#: 0 requests:4> (<class 'OSError'>:[Errno 57] Socket is not connected [/Users/username/git/supervisor/supervisor/medusa/asynchat_25.py|handle_read|89] [/Users/username/git/supervisor/supervisor/medusa/http_server.py|recv|528] [/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py|recv|354] [/Users/username/git/supervisor/supervisor/medusa/asynchat_25.py|handle_close|156] [/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py|close|361])
error: uncaptured python exception, closing channel <supervisor.http.deferring_http_channel connected 127.0.0.1:58792 at 0x102d3a310 channel#: 0 requests:4> (<class 'OSError'>:[Errno 57] Socket is not connected [/Users/username/git/supervisor/supervisor/supervisord.py|runforever|218] [/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py|handle_read_event|392] [/Users/username/git/supervisor/supervisor/medusa/asynchat_25.py|handle_read|91] [/Users/username/git/supervisor/supervisor/medusa/http_server.py|handle_error|546] [/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py|handle_error|422] [/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py|close|361])
Traceback (most recent call last):
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 347, in recv
    self.handle_close()
  File "/Users/username/git/supervisor/supervisor/medusa/asynchat_25.py", line 156, in handle_close
    self.close()
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 361, in close
    self.socket.shutdown(socket.SHUT_RDWR)
OSError: [Errno 57] Socket is not connected

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/username/git/supervisor/supervisor/medusa/asynchat_25.py", line 89, in handle_read
    data = self.recv (self.ac_in_buffer_size)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/git/supervisor/supervisor/medusa/http_server.py", line 528, in recv
    result = asynchat.async_chat.recv (self, buffer_size)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 354, in recv
    self.handle_close()
  File "/Users/username/git/supervisor/supervisor/medusa/asynchat_25.py", line 156, in handle_close
    self.close()
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 361, in close
    self.socket.shutdown(socket.SHUT_RDWR)
OSError: [Errno 57] Socket is not connected

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/username/git/supervisor/supervisor/supervisord.py", line 218, in runforever
    dispatcher.handle_read_event()
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 392, in handle_read_event
    self.handle_read()
  File "/Users/username/git/supervisor/supervisor/medusa/asynchat_25.py", line 91, in handle_read
    self.handle_error()
  File "/Users/username/git/supervisor/supervisor/medusa/http_server.py", line 546, in handle_error
    asynchat.async_chat.handle_error (self)
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 422, in handle_error
    self.close()
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 361, in close
    self.socket.shutdown(socket.SHUT_RDWR)
OSError: [Errno 57] Socket is not connected

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/bin/supervisord", line 33, in <module>
    sys.exit(load_entry_point('supervisor', 'console_scripts', 'supervisord')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/git/supervisor/supervisor/supervisord.py", line 373, in main
    go(options)
  File "/Users/username/git/supervisor/supervisor/supervisord.py", line 383, in go
    d.main()
  File "/Users/username/git/supervisor/supervisor/supervisord.py", line 78, in main
    self.run()
  File "/Users/username/git/supervisor/supervisor/supervisord.py", line 94, in run
    self.runforever()
  File "/Users/username/git/supervisor/supervisor/supervisord.py", line 224, in runforever
    combined_map[fd].handle_error()
  File "/Users/username/git/supervisor/supervisor/medusa/http_server.py", line 546, in handle_error
    asynchat.async_chat.handle_error (self)
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 422, in handle_error
    self.close()
  File "/Users/username/git/supervisor/supervisor/medusa/asyncore_25.py", line 361, in close
    self.socket.shutdown(socket.SHUT_RDWR)
OSError: [Errno 57] Socket is not connected
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant