Unclear "Use of Closed Network Connection" Error

[abraithwaite]

Describe the bug

When using clickhouse-go with clickhouse-cloud, we observe these errors happening during what we suspect to be auto-scaling events on the cloud side.

{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n    github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n        /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n  - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}

I believe it's because clickhouse proactively closes network connections that have encountered some kind of exception? It would be nice to have either a clearer error message here, or even better automatic reconnection / retries in the event of a failure here.

Getting this particular error bubbled up to the application isn't particularly useful. The only thing we can do is kill the client and recreate it before re-issuing the query because we can no longer trust the connection.

To be clear: we are not closing the connection on our end, so for whatever reason clickhouse server has to close the connection proactively, I think this library should be able to catch and handle that situation for a cleaner shutdown with a sentinel error value at the very least so we can make a more informed decision about how to handle it.

Steps to reproduce

1. Create a clickhouse client connection to a clickhouse-cloud instance in Go.
2. Trigger some scaling events while attempting to insert large batches.
3. Observe the error

Expected behaviour

We get a better error returned or have options to configure retries, or both.

Error log

{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n    github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n        /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n  - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}

Configuration

Environment

• Client version: github.com/ClickHouse/clickhouse-go/v2 v2.23.2
• ch-go version: v0.61.5
• Language version: 1.22.3
• OS: Ubuntu Linux 22.04

ClickHouse server

• ClickHouse Server version: 24.2.2
• ClickHouse Server non-default settings, if any: clickhouse-cloud default settings.
• CREATE TABLE statements for tables involved:
• Sample data for all these tables, use clickhouse-obfuscator if necessary

[charredlot]

not sure if it's the same issue, but ran into a similar problem with the chpool client getting stuck in a bad state when the server-side disconnects...made a PR here #409

[cwegener]

FWIW, I can see these errors when using a local ClickHouse server standalone instance.