Add ThroughputMonitor for enhanced data transfer tracking #49720
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces the ThroughputMonitor class to monitor and calculate throughput during data transfer operations. Updates - JobBuilder, - TransferJobInternal, - TransferManager, and - TransferProgressTracker to integrate throughput monitoring. Adds unit tests for ThroughputMonitor functionality and updates existing tests to mock and verify its integration in transfer processes.
API Change CheckAPIView identified API level changes in this PR and created the following API reviews |
added a new parameterless constructor that initializes the instance with a default processor.
|
API change check APIView has identified API level changes in this PR and created following API reviews. |
t-davidbrown-msft
requested review from
seanmcc-msft,
amnguye,
jaschrep-msft,
schaabs,
tg-msft,
jalauzon-msft and
nickliu-msft
as code owners
| /// <summary> | ||
| /// Gets the current throughput in bytes per second. | ||
| /// </summary> | ||
| internal virtual decimal Throughput { get; set; } |
There was a problem hiding this comment.
We should probably stick with float for decimal types.
There was a problem hiding this comment.
@jalauzon-msft oops. I thought we said double, but I can change them to float.
| if (_totalBytesTransferred == 0) | ||
| return Task.CompletedTask; | ||
|
|
||
| if (!_isStopwatchRunning) |
There was a problem hiding this comment.
Is there a way we can get rid of stopwatch?
| /// Monitors the throughput of data transfer operations by tracking the total bytes transferred | ||
| /// and calculating the throughput in bytes per second. | ||
| /// </summary> | ||
| internal class ThroughputMonitor : IAsyncDisposable |
amnguye
reviewed
May 7, 2025
sdk/storage/Azure.Storage.DataMovement/src/ConcurrencyTuner/ThroughputMonitor.cs
Show resolved
Hide resolved
amnguye
reviewed
May 7, 2025
sdk/storage/Azure.Storage.DataMovement/src/ConcurrencyTuner/ThroughputMonitor.cs
Outdated
Show resolved
Hide resolved
amnguye
reviewed
May 7, 2025
added 6 commits
May 8, 2025 00:12
Significantly modify the TransferManager class to replace the previous TransferOptions property with a parameterized approach using transferOptions in various methods. Update ResumeAllTransfersAsync and ResumeTransferAsync to utilize the transferOptions parameter, ensuring robustness by initializing transferOptions when null. Enhance TransferProgressTracker to guarantee proper initialization of ThroughputMonitor and improve progress tracking consistency.
The `TransferOptions` property and its associated private field `_transferOptions` have been removed from the `TransferManager` class. This change eliminates the property implementation that managed the assignment and validation of transfer options, including the handling of `ProgressHandlerOptions`. This simplification may reflect a shift in how transfer options are managed or a refactoring of the codebase.
- Modify `AvgThroughput` to return `double.MaxValue` for zero elapsed time with bytes transferred. - Update `ProcessBytesTransferredAsync` to handle zero elapsed time correctly. - Change `DisposeAsync` to a synchronous `Dispose` method. - Refactor and rename tests in `ThroughputMonitorTests.cs` for clarity. - Add new tests for `AvgThroughput` under various conditions. - Enhance tests to verify internal state using reflection. - Clarify cancellation token handling in `QueueBytesTransferredAsync`.
Modified the `ThroughputInMb_And_AvgThroughputInMb_CalculatedCorrectly` test in `ThroughputMonitorTests.cs` to use a tolerance of `0.05` Mbps for assertions. This change enhances the flexibility of comparisons for calculated throughput values, allowing for minor variations in results.
Updated the ThroughputMonitor class to change properties and calculations from double to float for improved performance and reduced memory usage. Changed the access modifier of QueueBytesTransferredAsync from public to internal. Added a SetStartTime method for direct initialization of _startTime. Updated tests in ThroughputMonitorTests to reflect these changes, including using float comparisons, renaming tests for clarity, and adding assertions for throughput calculations.
amnguye
approved these changes
May 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces the ThroughputMonitor class to monitor and calculate throughput during data transfer operations.
Updates
Adds unit tests for ThroughputMonitor functionality and updates existing tests to mock and verify its integration in transfer processes.
Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.