Alpine + Swoole + PgSQL on the host = could not send SSL negotiation packet: Resource temporarily unavailable

[flexchar]

Since the issue is so mysterious, I was able to and made a reproducible. Please try it yourself.

https://github.com/flexchar/pdo-pgsql-issue

---

Please answer these questions before submitting your issue.

1. What did you do? If possible, provide a simple script for reproducing the error.
   I attempt to connect to the pgsql running on my host mac machine from inside docker container running Laravel. It would fail with SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable error.

It does work if I would connect using psql cli terminal. It also works if I used pg_connect instead php-pdo.

I cannot test with php83 since the latest swoole pecl upstream has been broken for months now. I am not sure what's going on.

2. What did you expect to see?
   It should connect.

3. What did you see instead?
   SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable

4. What version of Swoole are you using (show your php --ri swoole)?

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 5.1.1
Built => Nov 28 2023 19:24:57
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.1.4 24 Oct 2023
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
c-ares => 1.27.0
zlib => 1.3
brotli => E16781312/D16781312
mutex_timedlock => enabled
pthread_barrier => enabled
mysqlnd => enabled
async_redis => enabled
coroutine_pgsql => enabled
coroutine_sqlite => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

as installed from https://pkgs.alpinelinux.org/packages?name=php82-pecl-swoole&branch=v3.19&repo=community&arch=x86_64&maintainer=

5. What is your machine environment used (show your uname -a & php -v & gcc -v) ?

$ uname -a
Linux 031104a4b74c 6.6.22-linuxkit #1 SMP Fri Mar 29 12:21:27 UTC 2024 aarch64 Linux

$ php -v
PHP 8.2.18 (cli) (built: Apr 11 2024 14:41:12) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.18, Copyright (c) Zend Technologies

$ gcc -v
ash: gcc: not found

---

It may also be that pdo-pgsql is broken but I am not sure where to report it. I would appreciate the guidance. Thank you. :)

[NathanFreeman]

It seems that I need to synchronize the code for php8.3 pdo_pgsql.

[flexchar]

Swoole unfortunately doesn't work with PHP 8.3 on Alpine because the most recent published version is the one with the bug 5.1.1. Alpine maintains only the most recent version and it's related to the PECL problem. Ref #5242

[myxingkong]

I encountered the same problem on Debian, and the issue persists after attempting to upgrade PHP to 8.3.

$ uname -a
Linux 2a9a37e4e7c4 6.7.11-orbstack-00143-ge6b82e26cd22 #1 SMP Sat Mar 30 12:20:36 UTC 2024 aarch64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

$ php -v
PHP 8.3.6 (cli) (built: Apr 24 2024 19:23:57) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.6, Copyright (c) Zend Technologies
    with Zend OPcache v8.3.6, Copyright (c), by Zend Technologies

$ php --ri swoole
Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 5.1.2
Built => May 14 2024 03:45:52
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.0.11 19 Sep 2023
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
c-ares => 1.18.1
zlib => 1.2.13
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled
coroutine_pgsql => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

[flexchar]

Same here on alpine with the new version. The thing that sucks is that alpine only has the latest versions which now makes everyone stuck with the broken one. I tried switching to FrankenPHP but Swoole outperforms it by 30x in my humble benchmarks. So it's also hard to say no.

[NathanFreeman]

@myxingkong Can you provide a script for reproduction?

[NathanFreeman]

@flexchar Do you have coroutines enabled?

[flexchar]

@NathanFreeman If this is the fair way to judge, then it seems so.

(Note this is the output from the working version)

laravel@:/var/www $ php --ri swoole

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 5.0.3
Built => May 11 2023 21:36:59
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.1.0 14 Mar 2023
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
c-ares => 1.19.1
zlib => 1.2.13
brotli => E16777225/D16777225
mutex_timedlock => enabled
pthread_barrier => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

I tried setting them off using the following but the issue remains the same.

[swoole]
swoole.enable_coroutine = 0

[myxingkong]

If coroutines are not used, connecting to external servers will not work properly. However, there is no problem connecting to the internal Docker server.

$dsn = 'pgsql:dbname=postgres host=192.168.199.201 port=5432';
try {
    $db = new PDO($dsn, 'postgres', '123456');
    echo "Connected successfully";
} catch (PDOException $e) {
    echo "Connection failed: " . $e->getMessage();
}

Output：

Connection failed: SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable

Moving the above code into a coroutine, it can work normally.

use function Swoole\Coroutine\run;

run(function () {
    $dsn = 'pgsql:dbname=postgres host=192.168.199.201 port=5432';
    try {
        $db = new PDO($dsn, 'postgres', '123456');
        echo "Connected successfully";
    } catch (PDOException $e) {
        echo "Connection failed: " . $e->getMessage();
    }
});

Output:

Connected successfully

After attempting to disable coroutines, the problem still exists. Only when I completely disable the Swoole extension, can the problem be solved.

[never615]

php -v

same problem on Debian

swoole 5.1.1 php8.2;
swoole 5.1.2 php8.2;
swoole 5.1.2 php8.3; I have tested these versions and have the same problems on debian. Test based on phpswoole/swoole docker image.

on Ubuntu,It is ok. (swoole 5.1.1 php8.3.6)

[vic-pic]

Hi,
I have the same problem with all versions starting from version 5.1, with debian and alpine, with php 8.2 and 8.3.
I'm using official swoole docker images.
With official docker images of swoole 5.0 everything works correctly.

[NathanFreeman]

I have tried many methods,

1. php, swoole, and postgresql are directly installed on the server.
2. php, swoole installed in a docker, postgresql on the server.
3. php, swoole in docker A, postgresql in docker B.

None of which can be reproduced.
Is there any other information that can be provided?
@never615 @vic-pic @myxingkong @flexchar

[NathanFreeman]

Or execute php -m to check if php has installed pdo_pgsql.

[flexchar]

@NathanFreeman,
it works if both are on the same level such as on the server or inside docker containers in the same stack. However not if swoole inside docker and pgsql on the host - your variant number 2.

Did you try the 2nd using the docker image I provided? I used Postrgres.App on mac as the database on the host and in this scenario it will cause the error.

[vic-pic]

@NathanFreeman ,
my case is that I'm running pgsql on the host machine, and it doesn't work. The same Dockerfile used to build container with swoole 5.1 that doesn't work, with swoole 5.0 works without problems.

[allsilaevex]

@NathanFreeman

If I run Swoole v5.1.2 and PostgreSQL v15.2 on the same network via docker compose, then everything works fine. But if I assign swoole the network "net1", and postgresql the network "net2", then I get an error "could not send SSL negotiation packet".

[lv2u]

@NathanFreeman I understand you cannot reproduce the issue, because it only happens sometimes. The bug doesn't always occur on every environment and I have absolutely no idea why.

I've spend the good part of the day trying to debug this issue, because we run into this issue on 2 of our 4 DTAP environments.

In our environments, everything is running great up until the acceptance and production environments, where the 'SSL negotiation packet' error is thrown. Each environment is rolled out using the same IaC codebase. I have confirmed they are all using the same software, tools, versions and OS's.

The same image - based on your Dockerfile - is being used on all environments during my tests. I've also tested it with a Debian based image, which gives the exact same results.

Connecting from within the image to the Postgres database using the psql client works as expected, which rules out potential firewall issues.

Disabling the SSL mode - for testing only obviously - results in a could not send startup packet error message.

All things together; it looks like the PHP process just can't seem to connect with the Postgres database, even though TCP packets are send to and from the database when these errors occur.

I'm currently out of ideas how to debug this issue and provide you with the information you need. If there is anything that I missed here or anything I can do to help you out, let me know.

[7oku]

Not sure if this is of help, otherwise just another +1 here.

We have the same with a Laravel application which we try to migrate from OpenSwoole to Swoole. Our phpinfo output looks similar to the mentions above.

Our image is built FROM alpine:3.20, packages are installed as needed via apk add php82-XXX, swoole via apk add php82-pecl-swoole. That's essentially the way to reproduce it.

On start a PostgreSQL db migration on an AWS RDS instance via php artisan migrate is necessary for our app. While OpenSwoole succeeds, Swoole fails at this stage with

   Illuminate\Database\QueryException

  SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable (Connection: pgsql, SQL: select * from information_schema.tables where table_catalog = <dbname> and table_schema = public and table_name = migrations and table_type = 'BASE TABLE')

  at vendor/laravel/framework/src/Illuminate/Database/Connection.php:829
    825▕                     $this->getName(), $query, $this->prepareBindings($bindings), $e
    826▕                 );
    827▕             }
    828▕
  ➜ 829▕             throw new QueryException(
    830▕                 $this->getName(), $query, $this->prepareBindings($bindings), $e
    831▕             );
    832▕         }
    833▕     }

      +35 vendor frames

  36  artisan:35
      Illuminate\Foundation\Console\Kernel::handle()

We hunt this problem for 3 months now and cannot find anything, so we're stuck with OpenSwoole currently.

Maybe you have an idea how we can debug this further from here in our app and provide you with infos?

[myxingkong]

After some debugging, I found the problem. Swoole sets the PostgreSQL connection to non-blocking mode and does not use functions like select to listen for the connection status in non-coroutine mode. This causes the PQconnectPoll function to return immediately.

It is not possible to reproduce this problem between containers because the latency between them is small. When the PQconnectPoll function returns, the socket has been sent. This problem only occurs when trying to connect to an external server with a delay of >2ms.

Try to rebuild Swoole using the following patches, and the problem is temporarily resolved.

diff --git a/ext-src/swoole_pgsql.cc b/ext-src/swoole_pgsql.cc
index 7c8ee1649..fade6827f 100644
--- a/ext-src/swoole_pgsql.cc
+++ b/ext-src/swoole_pgsql.cc
@@ -86,7 +86,28 @@ PGconn *swoole_pgsql_connectdb(const char *conninfo) {

     PQsetnonblocking(conn, 1);

+    fd_set readfds, writefds;
+    timeval timeout = {0, 10000};
+    int retval = 0;
+    if (swoole_pgsql_blocking) {
+        FD_ZERO(&readfds);
+        FD_ZERO(&writefds);
+        FD_SET(fd, &readfds);
+        FD_SET(fd, &writefds);
+    }
+
     SW_LOOP {
+        if (swoole_pgsql_blocking) {
+            retval = select(fd + 1, &readfds, &writefds, nullptr, &timeout);
+            if (retval == -1) {
+                if (errno == EINTR) {
+                    continue;
+                }
+                return nullptr;
+            } else if (retval == 0) {
+                continue;
+            }
+        }
         int r = PQconnectPoll(conn);
         if (r == PGRES_POLLING_OK || r == PGRES_POLLING_FAILED) {
             break;

[flexchar]

Wow, you're the hero @myxingkong!

I rooting this is enough for @NathanFreeman to pin a new version and unblocks as all to upgrade to PHP 8.3 :))

[korridor]

I think we have the same issue since we rebuild our image for production yesterday. Is there a way to prevent this issue from happening until a fix is released?

Does this only happen if I connect to the database with SSL? Can I set the ssl mode to disabled to circumvent the issue? Or is it possible to downgrade to a Swoole version where this issue does not happen and if so do you know which version?

[therobfonz]

For anyone running into this problem with a Laravel Forge provisioned server, here are the steps I took to disable Swoole. Forge, by default, installs the Swoole extension so users can easily run Laravel Octane, but its there whether or not you decide to use it.

vi /etc/php/8.3/cli/conf.d/25-swoole.ini

Then comment out the extension line like this

;extension=swoole.so

Restart PHP

sudo service php8.3-fpm restart

Obviously if not using 8.3, update the command to the particular version, but this fixed the could not send SSL negotiation packet error.

[NathanFreeman]

@flexchar @myxingkong Here, it should be necessary to distinguish between coroutine and non-coroutine situations. Let me make the modification.

[7oku]

@andreinocenti The latest release https://github.com/swoole/swoole-src/releases/tag/v5.1.3 is from June 6th, but #5397 has been merged July 5th into main. There simply has not been any release with the fix yet, which I would expect to be either >=v5.1.4 or v6.0.

You could build from main if you need it quicker. We are still on openswoole and wait with a migration until a new swoole version with fix is released.

[andreinocenti]

@7oku Thank you. I just found a "solution". I lowered my php version to 8.2.19 and the swoole to 5.0.3. Now it is working using these old versions.

[stupid-programmer]

For anyone running into this problem with a Laravel Forge provisioned server, here are the steps I took to disable Swoole. Forge, by default, installs the Swoole extension so users can easily run Laravel Octane, but its there whether or not you decide to use it.

vi /etc/php/8.3/cli/conf.d/25-swoole.ini

Then comment out the extension line like this

;extension=swoole.so

Restart PHP

sudo service php8.3-fpm restart

Obviously if not using 8.3, update the command to the particular version, but this fixed the could not send SSL negotiation packet error.

Thank you @therobfonz this worked for me, with the small change of the file being a symlink so the actual file I needed to edit was /etc/php/8.3/mods-available/swoole.ini

[EliasElimah]

Hello @NathanFreeman
When will the fix be released?
We've been waiting for a while.

[NathanFreeman]

@EliasElimah see https://github.com/swoole/swoole-src/releases/tag/v5.1.4

[flexchar]

For alpine users.

Waiting for Andy Postnikov @andypost to release on Alpine: https://pkgs.alpinelinux.org/packages?name=php83-*swoole*&branch=v3.20

If anyone knows how to let him know of the new version, it'd be greatly appreciated!

[andypost]

Send to build alpinelinux/aports@ee4d00d alpinelinux/aports@d89ba00

[jhm-ciberman]

@matyhtf @NathanFreeman Hi! We have been getting this same error in our production environment after upgrading to v5.1.4

Chart: Number of occurrences of could not send SSL negotiation packet: Resource temporarily unavailable in our live production server.

Last sunday we upgraded from Swoole v5.1.1 to Swoole v5.1.3 to fix this issue related to cookies. Then on monday we discovered this bug (SQLSTATE[08006] [7] could not send SSL negotiation packet: Resource temporarily unavailable) and proceeded to upgrade from swoole v5.1.3 to swoole v5.1.4.

The number of occurrences of this error went down from 22,000 per day to "only" 200 per day. But still, 200 lost connections to our database is a lot and we are losing important customer data so I am still worried.

We are using an Ubuntu server hosted on AWS, Laravel Forge for deployment and nginx.

Any idea what can it be the cause?

Edit reason: I mistyped the version numbers. The correct version numbers are good now.

Edit2: Swoole is installed from PECL.

[NathanFreeman]

@jhm-ciberman You can try disabling --enable-swoole-pgsql when compiling Swoole and directly use the native pdo-pgsql. I am still looking for ways to fix this issue.

[NathanFreeman]

@jhm-ciberman I'm very sorry; I missed fixing one more thing in the previous update. I will likely submit a new PR tonight or tomorrow.

[jhm-ciberman]

@NathanFreeman Thanks for your answer. Sorry my ignorance, how should I tackle this?

• Should I try to disable the --enable-swoole-pgsql when compiling Swoole from PECL? Is there anything else that need to be done in order to use the native pdo-pgsql? (What differences are between Swoole pgsql and pdo-pgsql?)
• Or should I wait for your PR and a later release?

I am new to the world of installing PECL extensions so I want to minimize the downtime of our application.

Thank you for your help 😁

[NathanFreeman]

@jhm-ciberman If your project does not use coroutines, then it doesn't matter if you disable --enable-swoole-pgsql. Swoole pgsql is primarily designed for coroutine-based usage with PostgreSQL, and it hooks into some functions of pdo-pgsql.

[jhm-ciberman]

@NathanFreeman Mh... Nope, I don't think I am using Coroutines. I am using Laravel + Laravel Octane with the Swoole driver to serve my application. I use Nginx as reverse proxy. I just checked Laravel Octane source code and I don't think it's making use of Swoole coroutines in any way. I could be wrong.

I will try to recompile with --enable-swoole-pgsql disabled. Thanks!

[jhm-ciberman]

@NathanFreeman Sorry to bother. But as far as I understand, my version of Swoole in my production server is compiled without --enable-swoole-pgsql, am I right?

php --ri swoole

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 5.1.4
Built => Sep  9 2024 20:41:27
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.0.2 15 Mar 2022
dtls => enabled
http2 => enabled
json => enabled
pcre => enabled
zlib => 1.2.11
brotli => E16777225/D16777225
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

I compiled swoole in my staging server with and without pgsql support and the difference is that coroutine_pgsql => enabled is shown. But here in my production server (the one that is experimenting the error) it is not showing this.

Am I missing something?

[NathanFreeman]

@jhm-ciberman Yes, according to the configuration above, --enable-swoole-pgsql is indeed disabled. If you have disabled this option and are still encountering the issue, it could be that you have multiple PHP versions or multiple Swoole versions in your production environment. The version of Swoole running your code might be the one that has --enable-swoole-pgsql enabled.

[NathanFreeman]

If you previously had --enable-swoole-pgsql enabled in your production environment server and then recompiled to remove this configuration, did you restart your service afterwards?

[jhm-ciberman]

@NathanFreeman Hey, thanks for your quick answer!

If you have disabled this option and are still encountering the issue, it could be that you have multiple PHP versions or multiple Swoole versions in your production environment.

Mh. I am not sure how to check that

• apt list --installed | grep swoole gives nothing, meaning swoole is not installed via apt.
• php --ri swoole shows the output I shown before. (Swoole 5.1.4 running without coroutine_pgsql).
• pecl list | grep swoole shows swoole 5.1.4 stable.
• I never installed swoole directly from the source code.
• I know Laravel Forge initially installs swoole from apt when provisioning the server but I uninstalled it to be able to manually install swoole from pecl.

If you previously had --enable-swoole-pgsql enabled in your production environment server and then recompiled to remove this configuration, did you restart your service afterwards?

Yes! I just checked my ~/.bash_history. I did restart the server after recompiling PHP.

sudo pecl install swoole
# ... more commands, like re-adding swoole to php.ini
sudo systemctl restart nginx
# ... more commands
php artisan octane:reload

I just re restarted everything again just to be sure. I also sudo systemctl restart php8.3-fpm although as far as I understand that's not necessary.

I will wait a few hours and keep you updated just in case that second restart somehow solves the problem.

[jhm-ciberman]

Nope. Even with the restart, it keeps happening.

[flexchar]

@jhm-ciberman I could totally recommend hosting on your own server, with docker & cloudflare tunnels. Not that I have anything against Forge but this setup lets you roll out your own docker image with your own packages. Even though I opened this issue because I used to connect to the host during development, thanks to this deployment my production never had an issue. :)

That being said, if you're feeling lucky, you could also try Swoole 6.0.0. beta. Don't run it if you risk loosing revenue but do try. I tried and found no issues whatsoever, even though it is a beta version.

Also, where is your database hosted at?

[NathanFreeman]

@jhm-ciberman It may be that the PHP pgsql extension is dependent on an older version of the PostgreSQL library. If the PostgreSQL library version is <= 16.1 and SSL connections are enabled, there may be issues.

[jhm-ciberman]

@NathanFreeman Is this the version of PostgreSQL that I should look for?

$ php -i | grep PostgreSQL
PDO Driver for PostgreSQL => enabled
PostgreSQL(libpq) Version => 14.13
PostgreSQL Support => enabled
PostgreSQL (libpq) Version => 14.13

$ apt-cache policy php8.3-pgsql
php8.3-pgsql:
  Installed: 8.3.3-1+ubuntu22.04.1+deb.sury.org+1
  Candidate: 8.3.11-1+ubuntu22.04.1+deb.sury.org+1
  Version table:
     8.3.11-1+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy/main arm64 Packages
 *** 8.3.3-1+ubuntu22.04.1+deb.sury.org+1 100
        100 /var/lib/dpkg/status
        ```

[jhm-ciberman]

@jhm-ciberman I could totally recommend hosting on your own server, with docker & cloudflare tunnels.

@flexchar I am more of a VPS server guy. I like docker, but I prefer the simplicity of working with our own machine. Maybe in a future I change my mind, so thanks for the suggestion.

That being said, if you're feeling lucky, you could also try Swoole 6.0.0. beta. Don't run it if you risk loosing revenue but do try. I tried and found no issues whatsoever, even though it is a beta version.

Definitely no. Our server handles 1700 requests per minute. We cannot afford losing (more) customer data.

Also, where is your database hosted at?

Server is a AWS EC2 instance. Database is a AWS Postgres RDS instance.

[NathanFreeman]

@jhm-ciberman Can you show me your octane.php configuration?

[jhm-ciberman]

@NathanFreeman You are not gonna believe me. Yesterday we upgraded our server from t4g.large to t4g.2xlarge and the bug suddenly disappeared:

The disk image was exactly the same, so it wasn't something related to software. Maybe the server was under a heavy load and that caused the bug? I am not sure.

I will be keeping you up to date in case the bug appears again.

But thank you a lot for your help, and thanks for making Swoole. Thanks to Swoole our app runs super fast. Thanks!