Unexpected quick connection retries to the same host: Outgoing DDoS Detection by Hetzner

[SmaugPool]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.tech

Version

ipfs version --all
Kubo version: 0.24.0
Repo version: 15
System version: amd64/linux
Golang version: go1.21.3

Config

👉️ Click to expand

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": null,
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
  ],
  "DNS": {
    "Resolvers": null
  },
  "Datastore": {
    "BloomFilterSize": 8388608,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/3",
            "sync": false,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "20T"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false,
      "Interval": 10
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "DeserializedResponses": null,
    "DisableHTMLErrors": null,
    "ExposeRoutingAPI": null,
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "..."
  },
  "Internal": {
    "Bitswap": {
      "EngineBlockstoreWorkerCount": 16,
      "EngineTaskWorkerCount": 8,
      "MaxOutstandingBytesPerPeer": 1048576,
      "ProviderSearchDelay": null,
      "TaskWorkerCount": 8
    }
  },
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": null,
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": [
      ...
    ]
  },
  "Pinning": {},
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "0s",
    "Strategy": "roots"
  },
  "Routing": {
    "AcceleratedDHTClient": false,
    "Methods": null,
    "Routers": null,
    "Type": "autoclient"
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {
      "GracePeriod": "20s",
      "HighWater": 128,
      "LowWater": 64,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Enabled": true,
      "MaxMemory": "8 GB"
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

Hetzner detected an outgoing DDOS from running kubo:

Abuse Message [AbuseID:D5244A:2C]: DDoSOutLevel: Outgoing DDoS Detection; SRC: [xx.xx.xx.xx], DST: [86.84.231.48]

We have indications that there was an attack from your server.
Please take all necessary measures to avoid this in the future and to solve the issue.

##############################################################################    
#      DDoS-Attack detected from host xx.xx.xx.xx                           #    
##############################################################################    
    
    
TIME                                 SRC              ->  DST              SIZE  PROT  SRC-PORT  DST-PORT    
----------------------------------------------------------------------------------------------------------    
2023-12-09 21:55:48.62946843  +0000  xx.xx.xx.xx     ->  86.84.231.48      131   TCP      4001      4001    
2023-12-09 21:55:48.636624923 +0000  xx.xx.xx.xx     ->  86.84.231.48      125   TCP      4001      4001    
2023-12-09 21:55:48.65691599  +0000  xx.xx.xx.xx     ->  86.84.231.48      125   TCP      4001      4001    
2023-12-09 21:55:48.676951656 +0000  xx.xx.xx.xx     ->  86.84.231.48       82   TCP      4001      4001    
2023-12-09 21:55:48.68339217  +0000  xx.xx.xx.xx     ->  86.84.231.48      125   TCP      4001      4001    
2023-12-09 21:55:48.689840641 +0000  xx.xx.xx.xx     ->  86.84.231.48     1537   TCP      4001      4001    
2023-12-09 21:55:48.697183818 +0000  xx.xx.xx.xx     ->  86.84.231.48       82   TCP      4001      4001    
2023-12-09 21:55:48.704837033 +0000  xx.xx.xx.xx     ->  86.84.231.48      131   TCP      4001      4001    
2023-12-09 21:55:48.720577474 +0000  xx.xx.xx.xx     ->  86.84.231.48      125   TCP      4001      4001    
2023-12-09 21:55:48.737214402 +0000  xx.xx.xx.xx     ->  86.84.231.48      307   TCP      4001      4001    
2023-12-09 21:55:48.777546548 +0000  xx.xx.xx.xx     ->  86.84.231.48       52   TCP      4001      4001    
2023-12-09 21:55:51.482935619 +0000  xx.xx.xx.xx     ->  86.84.231.48       40   TCP      4001      4001    
2023-12-09 21:55:51.690973777 +0000  xx.xx.xx.xx     ->  86.84.231.48       40   TCP      4001      4001    
2023-12-09 21:55:51.894931229 +0000  xx.xx.xx.xx     ->  86.84.231.48       40   TCP      4001      4001    
2023-12-09 21:55:52.313679904 +0000  xx.xx.xx.xx     ->  86.84.231.48       40   TCP      4001      4001    
2023-12-09 21:55:53.165124965 +0000  xx.xx.xx.xx     ->  86.84.231.48       40   TCP      4001      4001    
2023-12-09 21:55:54.969556665 +0000  xx.xx.xx.xx     ->  86.84.231.48       40   TCP      4001      4001  

Note: I redacted my node source IP

I was told in IPFS Discord that was unexepected because Kubo is supposed to use some backoff retry strategy.

[moshemalawach]

Same here, many of our node operators have been warned by their providers.

[hoh]

Looks like this was already an issue in 0.23.0

[lidel]

Hm.. I was not able to reproduce, the reported "DST" ("DDoS" receiver) node is a valid Kubo daemon running in Docker, and it is online now:

$ ipfs swarm connect /ip4/86.84.231.48/tcp/4001/p2p/12D3KooWDpvkix77iXcsxzbiV5aJuJC9oCcZ8o55QzA7bq8t7zAA
connect 12D3KooWDpvkix77iXcsxzbiV5aJuJC9oCcZ8o55QzA7bq8t7zAA success

My working theory is that 12D3KooWDpvkix77iXcsxzbiV5aJuJC9oCcZ8o55QzA7bq8t7zAA went offline, and the ISP (https://kpn.com) interpreted failed connection attempts to this node as DDoS (random IPs from all over the world sending weird packets), then reported it to ISPs of every IPFS node that tried to connect to the above peer, including nodes running on Hetzner.

And/or it is also possible the ISP started blackholing the P2P traffic for some reason, which would also made the peer look offline, and cause retries (we don't know if there were any response packets).

To me, based on limited available information, this looks like a false-positive.

If I was a client of Hetzner or KPN, I would try to explain this is not a DDoS, but IPFS nodes trying to talk to a service exposed by one of KPN clients. Mind that KPN knows who is operating 86.84.231.48 and can reach out to them to confirm.

Alternative ideas, or information how to reproduce the 40 byte packets is welcome.

[SmaugPool]

@lidel I think the question is why is there 11 connection retries in 150 ms.
It seems it's not the expected Kubo behavior because of supposed backoff retry strategy.