Socket error when starting Synapse - Cannot bind - No such file or directory

[MadLittleMods]

When trying to run Sytest against Synapse, I am running into the following error socket bind error, Error starting server-0: Cannot bind - No such file or directory @details=bind IO::Socket::UNIX=GLOB(0x7feab5d1df20) #sytest-JXOAUKDFIBVQMRPNAJEO.sock No such file or directory at tests/05homeserver.pl line 128..

Which probably corresponds to lib/SyTest/Homeserver/ProcessManager.pm#L355-L361

$ ./run-tests.pl tests/31sync/01filter.pl
Running tests/31sync/01filter.pl...
  Testing if: Can create filter...
Tests: 0 / 2 | OKSignature ok
subject=/CN=localhost
 # Started test HTTPS Server at localhost:56633
 # Started test SMTP Server at localhost:56634
 # Starting Homeserver using SyTest::HomeserverFactory::Synapse=HASH(0x7fdf0680a990)
 # Starting server-0
 # Using DB config from /Users/eric/Documents/github/element/sytest/server-0/database.yaml
 # Clearing SQLite database at :memory:
  Testing if: Can create filter...
Tests: 0 / 2 | OKSignature ok
subject=/CN=localhost
 # Generating config for port 8800
 # Creating config for server 0 with command python -m synapse.app.homeserver --config-path /Users/eric/Documents/github/element/sytest/server-0/config.yaml --generate-config --report-stats=no --server-name localhost:8800
 # [python]: Config file '/Users/eric/Documents/github/element/sytest/server-0/config.yaml' already exists. Generating any missing config files.
 # Starting server 0 for port 8800
  Testing if: Can create filter...
Tests: 0 / 2 | OK
 ** Error starting server-0: Cannot bind - No such file or directory @details=bind IO::Socket::UNIX=GLOB(0x7feab5d1df20) #sytest-JXOAUKDFIBVQMRPNAJEO.sock No such file or directory at tests/05homeserver.pl line 128.
  Testing if: Can create filter... FAIL:
  Testing if: Can download filter... SKIP due to lack of can_create_filter
 # Killing process server
 ** server process failed - code=2
 
1 tests FAILED

I think I have run into this problem in the past because I have debug logging around the sockets code in this PR, but I can't remember how I solved it. I know for sure I've run local Sytest against local Dendrite before.

Full run with my random debugs

 ./run-tests.pl tests/31sync/01filter.pl
Running tests/31sync/01filter.pl...
  Testing if: Can create filter...
Tests: 0 / 2 | OKSignature ok
subject=/CN=localhost
 # Started test SMTP Server at localhost:61574
 # Started test HTTPS Server at localhost:61575
 # Starting Homeserver using SyTest::HomeserverFactory::Synapse=HASH(0x7feab480af90)
 # Starting server-0
 # Using DB config from /Users/eric/Documents/github/element/sytest/server-0/database.yaml
 # Clearing SQLite database at :memory:
  Testing if: Can create filter...
Tests: 0 / 2 | OKSignature ok
subject=/CN=localhost
 # Generating config for port 8800
 # Creating config for server 0 with command python -m synapse.app.homeserver --config-path /Users/eric/Documents/github/element/sytest/server-0/config.yaml --generate-config --report-stats=no --server-name localhost:8800
 # [python]: Config file '/Users/eric/Documents/github/element/sytest/server-0/config.yaml' already exists. Generating any missing config files.
 # Starting server 0 for port 8800
  Testing if: Can create filter...
Tests: 0 / 2 | OKStarting synapse with command: python -m synapse.app.homeserver --config-path /Users/eric/Documents/github/element/sytest/server-0/config.yaml
New unix socket sytest-JXOAUKDFIBVQMRPNAJEO.sock
feawfefe
rdtnb
kuykuyyyukyukg
Binding unix socket sytest-JXOAUKDFIBVQMRPNAJEO.sock
jytjuytyjyjjy
Binding unix socket sytest-JXOAUKDFIBVQMRPNAJEO.sock
 ** Error starting server-0: Cannot bind - No such file or directory @details=bind IO::Socket::UNIX=GLOB(0x7feab5d1df20) #sytest-JXOAUKDFIBVQMRPNAJEO.sock No such file or directory at tests/05homeserver.pl line 128.
  Testing if: Can create filter... FAIL:
 | asdf at tests/05homeserver.pl line 130.
 | 	main::__ANON__("Cannot bind - No such file or directory", "bind", IO::Socket::UNIX=GLOB(0x7feab5d1df20), "#\x{1}\x{0}sytest-JXOAUKDFIBVQMRPNAJEO.sock", "No such file or directory") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 480
 | 	Future::_mark_ready(IO::Async::Future=HASH(0x7feab6ff0c18), "failed") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 687
 | 	Future::fail(IO::Async::Future=HASH(0x7feab6ff0c18), "Cannot bind - No such file or directory", "bind", IO::Socket::UNIX=GLOB(0x7feab5d1df20), "#\x{1}\x{0}sytest-JXOAUKDFIBVQMRPNAJEO.sock", "No such file or directory") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 853
 | 	Future::on_ready(IO::Async::Future=HASH(0x7feab6ff06c0), IO::Async::Future=HASH(0x7feab6ff0c18)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 471
 | 	Future::_mark_ready(IO::Async::Future=HASH(0x7feab6ff06c0), "wait_any") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 1905
 | 	Future::__ANON__(IO::Async::Future=HASH(0x7feab5e5fd38)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 480
 | 	Future::_mark_ready(IO::Async::Future=HASH(0x7feab5e5fd38), "failed") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 687
 | 	Future::fail(IO::Async::Future=HASH(0x7feab5e5fd38), "Cannot bind - No such file or directory", "bind", IO::Socket::UNIX=GLOB(0x7feab5d1df20), "#\x{1}\x{0}sytest-JXOAUKDFIBVQMRPNAJEO.sock", "No such file or directory") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 853
 | 	Future::on_ready(IO::Async::Future=HASH(0x7feab5d5e7a0), IO::Async::Future=HASH(0x7feab5e5fd38)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 471
 | 	Future::_mark_ready(IO::Async::Future=HASH(0x7feab5e08db0), "done") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 625
 | 	Future::done(IO::Async::Future=HASH(0x7feab5e08db0)) called at lib/SyTest/Homeserver.pm line 577
 | 	SyTest::Homeserver::__ANON__(IO::Async::Process=HASH(0x7feab5e32558), 0) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Notifier.pm line 837
 | 	IO::Async::Notifier::invoke_event(IO::Async::Process=HASH(0x7feab5e32558), "on_finish", 0) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Process.pm line 564
 | 	IO::Async::Process::__ANON__(undef) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 480
 | 	Future::_mark_ready(IO::Async::Future=HASH(0x7feab5e60578), "needs_all") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 2003
 | 	Future::__ANON__(IO::Async::Future=HASH(0x7feab5e32210)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 480
 | 	Future::_mark_ready(IO::Async::Future=HASH(0x7feab5e32210), "done") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 625
 | 	Future::done(IO::Async::Future=HASH(0x7feab5e32210)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Process.pm line 537
 | 	IO::Async::Process::__ANON__(undef, undef, undef, IO::Async::Process=HASH(0x7feab5e32558), 23116, 0, "", "") called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Internals/ChildManager.pm line 290
 | 	IO::Async::Internals::ChildManager::__ANON__(23116, 0) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Loop.pm line 2896
 | 	IO::Async::Loop::_reap_children(HASH(0x7feab4a05ea8)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Loop.pm line 2961
 | 	IO::Async::Loop::__ANON__() called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Loop.pm line 818
 | 	IO::Async::Loop::__ANON__() called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/OS.pm line 571
 | 	IO::Async::OS::_Base::__ANON__(IO::Async::Handle=HASH(0x7feab53ed8e0)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Loop/Poll.pm line 172
 | 	IO::Async::Loop::Poll::post_poll(IO::Async::Loop::Poll=HASH(0x7feab4a239c0)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Loop/Poll.pm line 292
 | 	IO::Async::Loop::Poll::loop_once(IO::Async::Loop::Poll=HASH(0x7feab4a239c0)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Loop.pm line 654
 | 	IO::Async::Loop::await(IO::Async::Loop::Poll=HASH(0x7feab4a239c0), IO::Async::Future=HASH(0x7feab5d1d9b0)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/IO/Async/Future.pm line 101
 | 	IO::Async::Future::await(IO::Async::Future=HASH(0x7feab5d1d9b0)) called at /Users/eric/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/Future.pm line 929
 | 	Future::get(IO::Async::Future=HASH(0x7feab5d1d9b0)) called at ./run-tests.pl line 763
 | 	eval {...} called at ./run-tests.pl line 710
 | 	main::_run_test0(SyTest::Output::Term::Test=HASH(0x7feab69f1d20), main::Test=ARRAY(0x7feab6e6b888), ARRAY(0x7feab70dfb40)) called at ./run-tests.pl line 674
 | 	main::_run_test(SyTest::Output::Term::Test=HASH(0x7feab69f1d20), main::Test=ARRAY(0x7feab6e6b888)) called at ./run-tests.pl line 897
 +----------------------
  Testing if: Can download filter... SKIP due to lack of can_create_filter
 # Killing process server
 ** server process failed - code=2

1 tests FAILED

Just creating this issue so I can track the solution and reference it in the future if I figure it out.

---

Environment:

• macOS 11.1
• Running inside my Synapse virtualenv
• Python 3.8.6
• perl 5, version 32

[richvdh]

this is probably something to do with unix sockets not working on mac os?

[richvdh]

or, specifically, the "abstract" namespace for unix sockets. Try changing my $path = "\0sytest-$random_id.sock"; in ProcessManager.pm to omit the \0.

[MadLittleMods]

@richvdh Woot! That's it! Thank you 🐳

And now looking back at my history, I can see that I searched for all of this sort of stuff back on 2020-12-30:

• Cannot bind - No such file or directory at unix sock macos
• abstract socket cannot bind perl
• perl socket connect bind
• perldoc "IO::Async::Socket" bind
• perl socket bind abstract namespace
• perl socket bind no such file or directory
• perl socket bind null byte "\0"
• perl socket bind null byte
• perl socket bind null terminator
• perl abstract socket bind no such file or directory
• "IO::Socket" "Cannot bind"
• "IO::Socket::UNIX" "Cannot bind"
• What does the @ symbol denote in the beginning of a unix domain socket path in Linux?, https://unix.stackexchange.com/a/206395/52320
• mac os x - UNIX domain sockets in abstract namespace, https://apple.stackexchange.com/a/157509/164173
• https://stackoverflow.com/q/16183794/796832

Now running successfully:

$ ./run-tests.pl tests/31sync/01filter.pl
Running tests/31sync/01filter.pl...
  Testing if: Can create filter...
Tests: 0 / 2 | OKSignature ok
subject=/CN=localhost
 # Started test HTTPS Server at localhost:52469
 # Started test SMTP Server at localhost:52474
 # Starting Homeserver using SyTest::HomeserverFactory::Synapse=HASH(0x7fb34580af90)
 # Starting server-0
 # Using DB config from /Users/eric/Documents/github/element/sytest/server-0/database.yaml
 # Clearing SQLite database at :memory:
  Testing if: Can create filter...
Tests: 0 / 2 | OKSignature ok
subject=/CN=localhost
 # Generating config for port 8800
 # Creating config for server 0 with command python -m synapse.app.homeserver --config-path /Users/eric/Documents/github/element/sytest/server-0/config.yaml --generate-config --report-stats=no --server-name localhost:8800
 # [python]: Config file '/Users/eric/Documents/github/element/sytest/server-0/config.yaml' already exists. Generating any missing config files.
 # Starting server 0 for port 8800
 # Started synapse 0
 # Started server-0
  Testing if: Can create filter... PASS
  Testing if: Can download filter... PASS
 # Killing process server

All tests PASSED (with 2 passes)

[MadLittleMods]

In order to make the codebase compatible with macOS out of the box, could we drop the abstract namespace?

Otherwise some platform specific code to fallback ($^O)?

[richvdh]

happy to drop the abstract namespace if you can find a good place for the socket file to live.