Skip to content

perf(v2): initialize dataset segment head lazily #4274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 1, 2025

Conversation

kolesnikovae
Copy link
Collaborator

@kolesnikovae kolesnikovae commented Jun 27, 2025

It's been observed that under certain conditions, segment-writer latency might have spikes, even when there are enough CPU resources. The mutex delay profile shows an unusual pattern that fully correlates with the observed latency increase:

image

I see three potential issues here:

  1. An RW lock on a write-intensive path.
  2. An expensive defer.
  3. A costly allocation under a wide (shard-wide) lock – NewHead is responsible for about 1% of all allocations.

That said, I find it hard to believe these are causing what the profile shows. Even with artificial load tests, I wasn't able to reproduce the problem (the branch version is at least twice as fast in all scenarios). I've seen much more heavily loaded instances that don't show any issues like this. Therefore, I believe it may not fully address the problem.

@kolesnikovae kolesnikovae marked this pull request as ready for review June 27, 2025 10:10
@alsoba13
Copy link
Contributor

LGTM

@kolesnikovae kolesnikovae merged commit ffc2b91 into main Jul 1, 2025
24 checks passed
@kolesnikovae kolesnikovae deleted the perf/lazy-segment-head-init branch July 1, 2025 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants