[BACKPORT][v1.6.2][IMPROVEMENT] Investigate performance bottleneck in v1 data path

[github-actions]

backport #8436

[PhanLe1010]

Ref PR: longhorn/longhorn-engine#1104

[roger-ryao]

Verified on v1.6.x-head 20240508

• longhorn v1.6.x-head 46706be
• longhorn-engine v1.6.x-head longhorn/longhorn-engine@4f53092

The test steps
#8436 (comment)

Result Passed

1. In the AWS EC2 t2.xlarge environment, I did not observe significant differences in IOPS.
   Therefore, I have decided to build up a cluster in a local virtual machine to conduct this test.
   The test result is as follows.

v1.6.1

=========================
FIO Benchmark Summary
For: test_device
CPU Idleness Profiling: disabled
Size: 30G
Quick Mode: disabled
=========================
IOPS (Read/Write)
        Random:           10,329 / 4,684
    Sequential:          17,677 / 10,204

Bandwidth in KiB/sec (Read/Write)
        Random:        413,252 / 141,958
    Sequential:        461,723 / 170,358
                                        

Latency in ns (Read/Write)
        Random:        431,773 / 553,568
    Sequential:        357,682 / 547,949

v1.6.x-head

=========================
FIO Benchmark Summary
For: test_device
CPU Idleness Profiling: disabled
Size: 30G
Quick Mode: disabled
=========================
IOPS (Read/Write)
        Random:           14,329 / 6,573
    Sequential:          22,396 / 10,423

Bandwidth in KiB/sec (Read/Write)
        Random:        513,981 / 153,945
    Sequential:        536,158 / 169,366
                                        

Latency in ns (Read/Write)
        Random:        392,363 / 527,421
    Sequential:        343,545 / 515,456

Here is the summary table

	IOPS (Random Read/Write)	IOPS (Sequential Read/Write)
v1.6.1-1st	10,329 / 4,684	17,677 / 10,204
v1.6.x-head-1st	14,329 / 6,573	22,396 / 10,423

	Bandwidth KiB/sec (Random Read/Write)	Bandwidth KiB/sec (Sequential Read/Write)
v1.6.1-1st	413,252 / 141,958	461,723 / 170,358
v1.6.x-head-1st	513,981 / 153,945	536,158 / 169,366

	Latency ns (Random Read/Write)	Latency ns (Sequential Read/Write)
v1.6.1-1st	431,773 / 553,568	357,682 / 547,949
v1.6.x-head-1st	392,363 / 527,421	343,545 / 515,456

Hi @PhanLe1010
I think we can close this ticket. If there are any concerns regarding the test results, please let me know.

[PhanLe1010]

Hi @roger-ryao

Thank you for testing!

In the AWS EC2 t2.xlarge environment, I did not observe significant differences in IOPS.

t2.xlarge is using EBS with burst performance to 3,000 IOPS. This is too slow and will not expose the improvement. Please use the ec2 with instance store (attached nvme) like c5d.2xlarge

Therefore, I have decided to build up a cluster in a local virtual machine to conduct this test.

Testing inside virtual machine might be ok too. But looks like the seq write inside virtual machine is capped at 10k somehow. Is it the max that VM can achieve natively?

[roger-ryao]

Testing inside virtual machine might be ok too. But looks like the seq write inside virtual machine is capped at 10k somehow. Is it the max that VM can achieve natively?

I believe it may be limited by my laptop's capabilities. I set up the cluster using Multipass, and there wasn't a way to configure that.