Skip to content

Improve Akka.Cluster.Metrics collected values #6203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Oct 28, 2022

Conversation

Arkatufus
Copy link
Contributor

@Arkatufus Arkatufus commented Oct 19, 2022

close #4142

This improves the values collected by the metrics plugin, it does not fix the heap selector since it is still missing informations.
Heap metric based routing can only be implemented after we implement the new EventCounters API.

Copy link
Member

@Aaronontheweb Aaronontheweb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but needs API approvals

// VirtualMemorySize64 is not best idea here...
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryAvailable, process.VirtualMemorySize64).Value,
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryUsed, GC.GetTotalMemory(false)).Value,
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryAvailable, process.WorkingSet64).Value,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aaronontheweb
Copy link
Member

We will also need to backport this to v1.5 @Arkatufus

@Aaronontheweb
Copy link
Member

@Arkatufus looks like some of the MNTR specs for Akka.Cluster.Metrics are not passing. Could you investigate?

@Arkatufus
Copy link
Contributor Author

Arkatufus commented Oct 20, 2022

Need a re-review.
Changed the code, available memory is the virtual size, used memory is the working set

@Aaronontheweb
Copy link
Member

@Arkatufus DocFx warning check here

// Forcing garbage collection to keep metrics more resilent to occasional allocations
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryUsed, GC.GetTotalMemory(true)).Value,
// VirtualMemorySize64 is not best idea here...
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryUsed, process.WorkingSet64).Value,
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryAvailable, process.VirtualMemorySize64).Value,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be working set

var metrics = new List<NodeMetrics.Types.Metric>()
{
// Memory
// Forcing garbage collection to keep metrics more resilent to occasional allocations
NodeMetrics.Types.Metric.Create(StandardMetrics.MemoryUsed, GC.GetTotalMemory(true)).Value,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be GC

@@ -55,12 +55,11 @@ public NodeMetrics Sample()
{
using (var process = Process.GetCurrentProcess())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this is wrong - shouldn't we just cache this value somewhere?

@Aaronontheweb
Copy link
Member

The right algorithm, which we can't implement on .NET Standard 2.0 without a lot of hacks:

used memory = gc.totalsize OR process.workingset64
available memory = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes

We should be measuring how close we're getting to hitting the total amount of available memory on the system as a percentage. In the meantime, we can do the following on .NET Standard 2.0:

used memory = gc.totalsize
available memory = process.workingset64 + process.pagedmemorysize64

This will tell us how much memory is being consumed by the .NET application itself - it won't be able to factor in issues like system memory and some unmanaged memory without the user manually calling GC.AddMemoryPressure but that's probably the best compromise we can make.

@Aaronontheweb
Copy link
Member

Testing out the latter now - we'll go back to the drawing board if we keep running into problems getting the MNTR to pass.

@Aaronontheweb
Copy link
Member

@Arkatufus needs a port to v1.5

@Aaronontheweb Aaronontheweb merged commit dca908b into akkadotnet:v1.4 Oct 28, 2022
Arkatufus added a commit to Arkatufus/akka.net that referenced this pull request Nov 2, 2022
* Improve collected values

* Update API Verify list

* Fix metrics collection

* Fix documentation

* Update API Verify list

* Update DefaultCollector.cs

* Update DefaultCollector.cs

Co-authored-by: Aaron Stannard <[email protected]>
(cherry picked from commit dca908b)
Aaronontheweb added a commit that referenced this pull request Nov 3, 2022
* Improve collected values

* Update API Verify list

* Fix metrics collection

* Fix documentation

* Update API Verify list

* Update DefaultCollector.cs

* Update DefaultCollector.cs

Co-authored-by: Aaron Stannard <[email protected]>
(cherry picked from commit dca908b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants