Fix circular mean by always storing and using the weighted one

[edenhaus]

Breaking change

Proposed change

• Store the mean_weight for the circular mean stats in the short term table
• Always use the mean_weight to calculate the new circular mean

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:
• Link to developer documentation pull request:
• Link to frontend pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Ruff (ruff format homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (sensor) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of sensor can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign sensor Removes the current integration label and assignees on the pull request, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

[emontnemery]

The code change looks good, OK to merge unless the statistic.get("mean_weight") or 0.0 is there to handle old rows where the weight is NULL.

If that's the case, I think it's better to bump the schema and remove all rows for circular mean so we don't need to handle the NULL.

Also, I think we should improve the tests:

• Instead of calculating the circular averages in the same way in tests as we do in the implementation, the tests should calculate the circular averages only based on the states
• Update tests to check that what we insert in the weight column is as expected

[emontnemery]

Is the get here needed to handle rows created before this patch was merged? If yes, I think it's better to just wipe the rows so we don't need to deal with this forever.

[edenhaus]

As the column mean_weight is defined as float | None, we also need to handle the case where it's None. This case is when the mean_type is arithmetic.

An alternative could be:

diff --git a/homeassistant/components/recorder/statistics.py b/homeassistant/components/recorder/statistics.py
index 80c0028ef7a..e168cbd92d6 100644
--- a/homeassistant/components/recorder/statistics.py
+++ b/homeassistant/components/recorder/statistics.py
@@ -1025,7 +1025,8 @@ def _reduce_statistics(
     _want_sum = "sum" in types
     for statistic_id, stat_list in stats.items():
         max_values: list[float] = []
-        mean_values: list[tuple[float, float]] = []
+        arithmetic_mean_values: list[float] = []
+        circular_mean_values: list[tuple[float, float]] = []
         min_values: list[float] = []
         prev_stat: StatisticsRow = stat_list[0]
         fake_entry: StatisticsRow = {"start": stat_list[-1]["start"] + period_seconds}
@@ -1042,15 +1043,14 @@ def _reduce_statistics(
                 if _want_mean:
                     row["mean"] = None
                     row["mean_weight"] = None
-                    if mean_values:
-                        match metadata[statistic_id][1]["mean_type"]:
-                            case StatisticMeanType.ARITHMETIC:
-                                row["mean"] = mean([x[0] for x in mean_values])
-                            case StatisticMeanType.CIRCULAR:
-                                row["mean"], row["mean_weight"] = (
-                                    weighted_circular_mean(mean_values)
-                                )
-                    mean_values.clear()
+                    if arithmetic_mean_values:
+                        row["mean"] = mean(arithmetic_mean_values)
+                    if circular_mean_values:
+                        row["mean"], row["mean_weight"] = weighted_circular_mean(
+                            circular_mean_values
+                        )
+                    arithmetic_mean_values.clear()
+                    circular_mean_values.clear()
                 if _want_min:
                     row["min"] = min(min_values) if min_values else None
                     min_values.clear()
@@ -1068,8 +1068,14 @@ def _reduce_statistics(
                 max_values.append(_max)
             if _want_mean:
                 if (_mean := statistic.get("mean")) is not None:
-                    _mean_weight = statistic.get("mean_weight") or 0.0
-                    mean_values.append((_mean, _mean_weight))
+                    match metadata[statistic_id][1]["mean_type"]:
+                        case StatisticMeanType.ARITHMETIC:
+                            arithmetic_mean_values.append(_mean)
+                        case StatisticMeanType.CIRCULAR:
+                            if (
+                                _mean_weight := statistic.get("mean_weight")
+                            ) is not None:
+                                circular_mean_values.append((_mean, _mean_weight))
             if _want_min and (_min := statistic.get("min")) is not None:
                 min_values.append(_min)
             prev_stat = statistic

[emontnemery]

As the column mean_weight is defined as float | None

But we ourselves insert the rows, and for circular mean we know we always insert the mean and the weight.
But I guess it makes sense to not blow up if there's invalid data in the database?

I like the proposed new version better, then we don't convert an invalid None to a valid 0.0

[emontnemery]

The test tests/components/tibber/test_statistics.py::test_async_setup_entry is failing

[edenhaus]

Update tests to check that what we insert in the weight column is as expected

This is done implicit as the expect output is calculated weighted and if we would not store the weight in the db, these row will be skipped and than the expected output will not match anymore

[emontnemery]

Update tests to check that what we insert in the weight column is as expected

This is done implicit as the expect output is calculated weighted and if we would not store the weight in the db, these row will be skipped and than the expected output will not match anymore

That kind of implicit testing is not good enough IMO. But OK to merge this PR without it.

[frenck]

I'm going to cut the release, so getting it in–tests can be improved later on. The CI failure isn't related.