Flowlogs: make calico-node to fetch flow logs from a node #10144

mazdakn · 2025-04-04T00:37:40Z

Description

Add a new command to calico-node binary to list or watch flow logs from a specific node. This is very useful for troubleshooting. The command line will be:

calico-node -flows [n]

if n is positive then it will watch for n logs and exits. If n is negative, then it watch for flowlogs for ever until interrupted.

In a cluster, it can be used as the following to watch for flowlogs for ever:

kubectl exec -ti [node-name] -n [node-namespace] -- calico-node -flows -1

Sample output:

- Time=2025-04-11 21:50:58 +0000 UTC Reporter=Src Action=Allow
  Src={wep(default/loadgenerator-78cb8f69c8-*)} Dst={wep(kube-system/coredns-6d4b75cb6d-*) Svc:kube-dns/kube-system} Proto={udp(53) Svc:dns/53}
  Counts={Ingress: 12Pkts/1818Bytes Egress:12Pkts/960Bytes} Connections={Started:6 Completed:0 Live:6}
  Enforced:
  - kind:CalicoNetworkPolicy namespace:"default" name:"allow-dns-egress" tier:"default" action:Allow
  Pending:
  - kind:CalicoNetworkPolicy namespace:"default" name:"allow-dns-egress" tier:"default" action:Allow
- Time=2025-04-11 21:50:58 +0000 UTC Reporter=Src Action=Deny
  Src={wep(default/loadgenerator-78cb8f69c8-*)} Dst={net(-/pvt) Svc:-/-} Proto={tcp(80) Svc:-/0}
  Counts={Ingress: 0Pkts/0Bytes Egress:27Pkts/1620Bytes} Connections={Started:6 Completed:0 Live:6}
  Enforced:
  - kind:EndOfTier tier:"default" action:Deny rule_index:-1 trigger:{kind:CalicoNetworkPolicy namespace:"default" name:"loadgenerator-frontend-policy" tier:"default"}
  Pending:
  - kind:EndOfTier tier:"default" action:Deny rule_index:-1 trigger:{kind:CalicoNetworkPolicy namespace:"default" name:"allow-dns-egress" tier:"default"}

To enable reporting flow logs to local node socket, a new Felix config, named flowLogsLocalReporter is introduced, which can be set to either Disabled (default) or Enabled.

Related issues/PRs

Todos

Tests
Documentation
Release note

Release Note

TBD

Reminder for the reviewer

Make sure that this PR has the correct labels and milestone set.

Every PR needs one docs-* label.

docs-pr-required: This change requires a change to the documentation that has not been completed yet.
docs-completed: This change has all necessary documentation completed.
docs-not-required: This change has no user-facing impact and requires no docs.

Every PR needs one release-note-* label.

release-note-required: This PR has user-facing changes. Most PRs should have this label.
release-note-not-required: This PR has no user-facing changes.

Other optional labels:

cherry-pick-candidate: This PR should be cherry-picked to an earlier release. For bug fixes only.
needs-operator-pr: This PR is related to install and requires a corresponding change to the operator.

felix/collector/goldmane/client.go

caseydavenport

First pass, want to take a closer look yet

felix/collector/goldmane/client.go

felix/collector/goldmane/node_server.go

node/cmd/calico-node/main.go

caseydavenport · 2025-04-11T22:20:07Z

felix/collector/goldmane/client.go

@@ -48,6 +53,9 @@ func NewReporter(addr, cert, key, ca string) (*GoldmaneReporter, error) {
 	return &GoldmaneReporter{
 		address: addr,
 		client:  cli,
+
+		// Do not send flowlogs to node socket, if goldmane address set via FelixConfig is equal to node socket
+		mayReportToNodeSocket: addr != NodeSocketAddress,


How much work would it be to lift this out of the Goldmane reporter and into its own Reporter, and then adding the ability to run multiple reporters + dynamically add / remove them?

It seems to me that we are going to want the ability to have multiple Reporters running at once - it would be cool to be able to use this e.g., even if Goldmane wasn't configured.

not much! Collector already supports multiple reporters.
But if we want to be able to run this without goldmane enabled, we should introduce a config option to enabled/disable it. Something like FlowLogsGoldmaneNodeServer with Enabled and Disabled values. WDYT?

That seems like a preferrable route to me, if it's not too much work. A new LocalFlowSocket: Enabled or something would make sense as the config option name probably?

felix/collector/goldmane/node_server.go

Co-authored-by: Casey Davenport <[email protected]>

goldmane/pkg/client/client.go

node/cmd/calico-node/main.go

node/pkg/flowlogs/flowlogs.go

tomastigera · 2025-04-15T22:17:21Z

calico-node --flows I though that calico-node used a single dash. All the other options have a single dash.

mazdakn · 2025-04-16T00:17:14Z

@tomastigera you are right. It's with one dash.

mazdakn · 2025-04-16T18:48:22Z

api/pkg/apis/projectcalico/v3/felixconfig.go

@@ -835,6 +835,10 @@ type FelixConfigurationSpec struct {
 	// FlowLogGoldmaneServer is the flow server endpoint to which flow data should be published.
 	FlowLogsGoldmaneServer *string `json:"flowLogsGoldmaneServer,omitempty"`

+	// FlowLogsLocalSocket configures local unix socket for reporting flow data from each node. [Default: Disabled]
+	// +kubebuilder:validation:Enum=Disabled;Enabled
+	FlowLogsLocalSocket *string `json:"flowLogsLocalSocket,omitempty"`


maybe we should name this FlowLogsLocalReporter or FlowLogsNodeReporter.

Yeah, that makes sense to me - I like the Local terminology rather than Node myself.

caseydavenport

@mazdakn I think this looks good, but I'd like to clean up some terminology and structure a bit - namely there are references to both NodeReporter and LocalReporter and I think we should standardize on the latter.

Also, its referred to as goldmane in a few places but I don't think it is tied to Goldmane any more?

caseydavenport · 2025-04-16T21:06:28Z

felix/collector/dpstatshelper.go

+		log.Info("Adding Flow Logs Aggregator (allowed) for node socket")
+		fr.AddAggregator(gaa, []string{FlowLogsLocalReporterName})
+		log.Info("Creating node socket Aggregator for denied")
+		gad := flowAggregatorForGoldmane(rules.RuleActionDeny, configParams.FlowLogsCollectorDebugTrace)


I still find this a bit confusing - this reporter isn't really for goldmane, rather it's for the node socket, right? Is this just a case of an awkward function name?

changed it to defaultFlowAggregator

felix/collector/goldmane/node_reporter.go

mazdakn · 2025-04-17T22:17:27Z

CI passed in the last run.

…ico#10144)

first commit

d98f9ed

mazdakn added docs-not-required Docs not required for this change release-note-not-required Change has no user-facing impact labels Apr 4, 2025

mazdakn requested a review from a team as a code owner April 4, 2025 00:37

marvin-tigera added this to the Calico v3.31.0 milestone Apr 4, 2025

mazdakn added 10 commits April 4, 2025 11:35

first prototype

914c030

udapte

f6675fa

Add more

813a16d

update

1b4462b

more

0a91e70

Cleanup

71437b5

Fix linter

71e3bd2

Merge remote-tracking branch 'open-source/master' into tool-flowlog

c20ee50

remove goldmane server mock

57cae89

Fix FVs

2013ced

mazdakn commented Apr 11, 2025

View reviewed changes

felix/collector/goldmane/client.go Outdated Show resolved Hide resolved

mazdakn added 11 commits April 11, 2025 10:36

clean up

7c21315

Merge remote-tracking branch 'open-source/master' into tool-flowlog

8524f88

some more change

744bc25

more

fd1c83d

Merge remote-tracking branch 'open-source/master' into tool-flowlog

e6e3ebf

Add FV

d6d9c71

Final cleanup

f4931da

Update Fv

5b8fc24

update comment

95af41d

Fix FVs

f931e41

improve FV

c479545

caseydavenport reviewed Apr 14, 2025

View reviewed changes

mazdakn and others added 2 commits April 14, 2025 09:47

Update node/cmd/calico-node/main.go

dfbd71c

Co-authored-by: Casey Davenport <[email protected]>

Update felix/collector/goldmane/node_server.go

a2c0168

Co-authored-by: Casey Davenport <[email protected]>

mazdakn requested a review from caseydavenport April 14, 2025 18:20

mazdakn commented Apr 14, 2025

View reviewed changes

goldmane/pkg/client/client.go Outdated Show resolved Hide resolved

mazdakn added 2 commits April 14, 2025 13:58

Fix node

fe90de6

Merge remote-tracking branch 'open-source/master' into tool-flowlog

fc08717

caseydavenport reviewed Apr 15, 2025

View reviewed changes

node/cmd/calico-node/main.go Outdated Show resolved Hide resolved

caseydavenport reviewed Apr 15, 2025

View reviewed changes

node/pkg/flowlogs/flowlogs.go Outdated Show resolved Hide resolved

mazdakn added 2 commits April 15, 2025 16:33

use a felixconfig

4b71e40

few changes

2ae5a5e

mazdakn requested a review from caseydavenport April 16, 2025 00:12

mazdakn added 2 commits April 16, 2025 10:44

update

37376a6

update

f5cb375

mazdakn commented Apr 16, 2025

View reviewed changes

mazdakn added 4 commits April 16, 2025 13:38

change config option

0cc4e5a

Merge remote-tracking branch 'open-source/master' into tool-flowlog

a87bd9a

change names

9f3e845

Merge remote-tracking branch 'open-source/master' into tool-flowlog

0ce48d9

caseydavenport approved these changes Apr 17, 2025

View reviewed changes

mazdakn added 2 commits April 17, 2025 11:16

rename to local socket

d361089

Merge remote-tracking branch 'open-source/master' into tool-flowlog

19c43e1

mazdakn merged commit 155dc4e into projectcalico:master Apr 17, 2025
3 of 4 checks passed

mazdakn deleted the tool-flowlog branch April 17, 2025 22:17

mazdakn added the cherry-pick-candidate label Apr 17, 2025

mazdakn mentioned this pull request Apr 22, 2025

flowlogs: cleanup socket when creating local reporter #10276

Merged

3 tasks

mazdakn added a commit to mazdakn/calico that referenced this pull request May 7, 2025

Flowlogs: make calico-node to fetch flow logs from a node (projectcal…

06ae429

…ico#10144)

mazdakn mentioned this pull request May 7, 2025

[Release v3.30] Picks from master #10374

Merged

3 tasks

mazdakn added cherry-pick-completed PR has been cherry-picked and removed cherry-pick-candidate labels May 16, 2025

Flowlogs: make calico-node to fetch flow logs from a node #10144

Flowlogs: make calico-node to fetch flow logs from a node #10144

Uh oh!

Conversation

mazdakn commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues/PRs

Todos

Release Note

Reminder for the reviewer

Uh oh!

Uh oh!

caseydavenport left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

caseydavenport Apr 11, 2025

Choose a reason for hiding this comment

Uh oh!

mazdakn Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caseydavenport Apr 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomastigera commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mazdakn commented Apr 16, 2025

Uh oh!

mazdakn Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

caseydavenport Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

caseydavenport left a comment

Choose a reason for hiding this comment

Uh oh!

caseydavenport Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

mazdakn Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mazdakn commented Apr 17, 2025

Uh oh!

Uh oh!

Uh oh!

mazdakn commented Apr 4, 2025 •

edited

Loading

mazdakn Apr 14, 2025 •

edited

Loading

tomastigera commented Apr 15, 2025 •

edited

Loading