Skip to content

[catpowder] Enhancement: XDP refactor and bug fixes #1522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

[catpowder] Enhancement: XDP refactor and bug fixes #1522

wants to merge 1 commit into from
+930 −558

Conversation

kyleholohan
Copy link
Contributor

This PR fixes several bugs in the XDP backend related to BufferPool population, specifically w.r.t. XDP-for-Windows' rules for truncation of buffers at the end of the pool. It also adds the ability to send out backend on the VF by default, rather than the default interface. Finally, it refactors and componentizes parts of the backend. The last point should be especially useful for the cohosting component, as it can be reused for the catnip backend.

@kyleholohan kyleholohan added bug Something Isn't Working enhancement Enhancement Request on an Existing Feature labels Apr 10, 2025
@kyleholohan kyleholohan self-assigned this Apr 10, 2025
Copilot
Copilot AI reviewed Apr 10, 2025
View reviewed changes
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

@github-actions GitHub Actions
Copy link

libos = catpowder
commit id = c19faef

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::inetstack::poll;inetstack::layer4::poll_once 17051380 685 275 10871420790 46 10864832243 85
bgc::inetstack::poll 8525690 1596 642 12689790561 54 1818369771 14
ioc::network::libos::push 16145 2989 1230 48691650 0 48691650 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch 5381 2695 1060 6700218 0 6698402 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11151 421 170 5770581 0 5770581 0
ioc::network::libos::pop 9055 449 181 3719368 0 3719368 0
bgc::inetstack::tcp::established::background 11165 1903 769 13466818 0 3439579 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11165 726 302 3326169 0 3326169 0
ioc::network::libos::pushto 666 4392 1801 2876968 0 2876968 0
ioc::network::libos::connect 42 39558 15927 1726145 0 1726145 0
bgc::inetstack::tcp::passiveopen::background 42 31354 12694 1045411 0 1045411 0
demikernel::sgaalloc 6504 160 65 1026007 0 1026007 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11108 885 341 930489 0 930489 0
demikernel::sgafree 8748 94 38 786265 0 786265 0
ioc::network::libos::close 46 6257 2501 292563 0 292563 0
ioc::network::libos::accept 38 2347 961 56106 0 56106 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;udp::receive 181 206 87 35705 0 35705 0
bgc::inetstack::arp::background 7 2990 1203 12259 0 12259 0
bgc::inetstack::icmp::background 1 1859 746 1976 0 1976 0

@github-actions GitHub Actions
Copy link

libos = catnap
commit id = c19faef

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::catnap::transport::epoll 10852501 4727 1981 5803381058 100 5803381058 100
ioc::network::libos::push 17600 512 210 7912202 0 7912202 0
ioc::network::libos::pop 15063 480 200 5721507 0 5721507 0
ioc::network::libos::connect 50 177710 72998 2988175 0 2988175 0
ioc::network::libos::accept 26 111289 44902 1379978 0 1379978 0
demikernel::sgaalloc 11280 135 56 1218726 0 1218726 0
demikernel::sgafree 11534 75 31 753816 0 753816 0
ioc::network::libos::pushto 1049 397 160 389172 0 389172 0
ioc::network::libos::close 18 23344 9593 342758 0 342758 0

@github-actions GitHub Actions
Copy link

libos = catnip
commit id = c19faef

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::inetstack::poll;inetstack::layer4::poll_once;catnip::runtime::receive 23693096 216 86 4956639135 19 4956639135 43
bgc::inetstack::poll;inetstack::layer4::poll_once 23693096 404 161 8981136904 35 4019897860 35
bgc::inetstack::poll 11846548 1015 403 11463831263 45 2482694359 22
ioc::network::libos::push 16145 1275 512 18144310 0 15979815 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11060 358 143 4713728 0 4712591 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch 2462 2920 1155 4677873 0 4659537 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11072 418 169 3034776 0 3031478 0
bgc::inetstack::tcp::established::background 11072 1217 486 11382230 0 2924193 0
ioc::network::libos::pop 9043 400 159 2672539 0 2672539 0
ioc::network::libos::push;catnip::runtime::transmit 8763 359 144 2350023 0 2350023 0
ioc::network::libos::pushto 1330 1428 589 1904483 0 1499594 0
ioc::network::libos::connect 42 34675 14032 1362395 0 1302838 0
demikernel::sgaalloc 6542 154 62 785982 0 785982 0
demikernel::sgafree 8785 97 39 725110 0 725110 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11018 316 123 709534 0 699300 0
bgc::inetstack::tcp::passiveopen::background 42 24481 9653 555239 0 505789 0
ioc::network::libos::pushto;catnip::runtime::transmit 1330 301 124 404889 0 404889 0
ioc::network::libos::close 49 5727 2279 259283 0 259283 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;udp::receive 572 233 95 128071 0 128071 0
ioc::network::libos::connect;catnip::runtime::transmit 42 6641 2642 59557 0 59557 0
bgc::inetstack::tcp::passiveopen::background;catnip::runtime::transmit 21 15287 6019 49450 0 49450 0
ioc::network::libos::accept 37 2241 889 46476 0 46476 0
bgc::inetstack::arp::background 7 4220 1664 27333 0 27333 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger;catnip::runtime::transmit 20 1573 662 20522 0 20522 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;catnip::runtime::transmit 24 575 227 12457 0 12457 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender;catnip::runtime::transmit 21 564 224 10234 0 10234 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter;catnip::runtime::transmit 14 818 345 6365 0 6365 0
bgc::inetstack::icmp::background 1 1662 660 1834 0 1834 0

@kyleholohan kyleholohan force-pushed the enhancement-xdp-refactor branch from c19faef to af863da Compare April 10, 2025 22:04
@github-actions GitHub Actions
Copy link

libos = catpowder
commit id = af863da

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::inetstack::poll;inetstack::layer4::poll_once 17879449 687 276 11250852122 46 11244034861 84
bgc::inetstack::poll 8939725 1604 645 13237155416 54 1986303295 15
ioc::network::libos::push 16145 3125 1270 49590469 0 49590469 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch 5481 2732 1084 6932808 0 6930955 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11241 481 192 5705258 0 5705258 0
ioc::network::libos::pop 9114 511 209 3709795 0 3709795 0
bgc::inetstack::tcp::established::background 11254 2115 912 13242406 0 3369116 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11254 856 412 3177441 0 3177441 0
ioc::network::libos::pushto 661 4347 1773 2841517 0 2841517 0
ioc::network::libos::connect 42 39980 16330 1719004 0 1719004 0
demikernel::sgaalloc 6503 166 67 1079208 0 1079208 0
bgc::inetstack::tcp::passiveopen::background 42 31553 12750 1059153 0 1059153 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11197 990 383 990592 0 990592 0
demikernel::sgafree 8748 103 42 737642 0 737642 0
ioc::network::libos::close 46 6593 2636 306633 0 306633 0
ioc::network::libos::accept 38 2443 1008 57297 0 57297 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;udp::receive 187 198 83 36445 0 36445 0
bgc::inetstack::arp::background 7 3088 1232 15164 0 15164 0
bgc::inetstack::icmp::background 1 1932 778 2086 0 2086 0

@github-actions GitHub Actions
Copy link

libos = catnap
commit id = af863da

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::catnap::transport::epoll 10851571 3635 1526 5813240568 100 5813240568 100
ioc::network::libos::push 17600 570 237 8272316 0 8272316 0
ioc::network::libos::pop 15177 524 222 5860037 0 5860037 0
ioc::network::libos::connect 50 194189 78613 3278011 0 3278011 0
ioc::network::libos::accept 26 125963 53193 1569775 0 1569775 0
demikernel::sgaalloc 11289 142 60 1218849 0 1218849 0
demikernel::sgafree 11538 84 36 770354 0 770354 0
ioc::network::libos::pushto 1369 425 183 643168 0 643168 0
ioc::network::libos::close 18 28296 11792 379706 0 379706 0

@github-actions GitHub Actions
Copy link

libos = catnip
commit id = af863da

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::inetstack::poll;inetstack::layer4::poll_once;catnip::runtime::receive 22788520 217 87 5035360213 20 5035360213 44
bgc::inetstack::poll;inetstack::layer4::poll_once 22788520 405 163 9040334601 35 4000149985 35
bgc::inetstack::poll 11394260 1022 410 11382074455 45 2341739854 21
ioc::network::libos::push 16145 1268 512 18058008 0 15864406 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch 2523 3539 1400 4906172 0 4888569 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11062 379 151 4751560 0 4749826 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11073 451 180 3029202 0 3025146 0
bgc::inetstack::tcp::established::background 11073 1290 514 11496501 0 2996827 0
ioc::network::libos::pop 9041 408 164 2762148 0 2762148 0
ioc::network::libos::push;catnip::runtime::transmit 8763 356 143 2381625 0 2381625 0
ioc::network::libos::pushto 1234 1437 615 1770918 0 1431885 0
ioc::network::libos::connect 42 34380 14166 1407643 0 1344687 0
demikernel::sgaalloc 6535 161 64 791389 0 791389 0
demikernel::sgafree 8780 97 39 725644 0 725644 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11020 333 129 718913 0 707706 0
bgc::inetstack::tcp::passiveopen::background 42 25254 10008 594451 0 544993 0
ioc::network::libos::pushto;catnip::runtime::transmit 1234 275 118 339033 0 339033 0
ioc::network::libos::close 49 5749 2318 261577 0 261577 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;udp::receive 523 244 103 115359 0 115359 0
ioc::network::libos::connect;catnip::runtime::transmit 42 6681 2737 62957 0 62957 0
bgc::inetstack::tcp::passiveopen::background;catnip::runtime::transmit 21 14651 5792 49458 0 49458 0
ioc::network::libos::accept 37 2308 917 47635 0 47635 0
bgc::inetstack::arp::background 7 3312 1328 26264 0 26264 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger;catnip::runtime::transmit 15 2125 878 20647 0 20647 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;catnip::runtime::transmit 24 598 238 12366 0 12366 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender;catnip::runtime::transmit 21 655 266 11207 0 11207 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter;catnip::runtime::transmit 16 1397 563 9710 0 9710 0
bgc::inetstack::icmp::background 1 1674 672 1836 0 1836 0

@kyleholohan kyleholohan force-pushed the enhancement-xdp-refactor branch from af863da to b20e1da Compare April 15, 2025 17:45
@github-actions GitHub Actions
Copy link

libos = catpowder
commit id = b20e1da

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::inetstack::poll;inetstack::layer4::poll_once 17198775 696 314 10945028331 46 10939298990 85
bgc::inetstack::poll 8599388 1626 734 12859853189 54 1914824858 15
ioc::network::libos::push 16145 3148 1511 48075624 0 48075624 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch 5295 2877 1188 5826448 0 5824150 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11066 446 206 5740804 0 5740804 0
ioc::network::libos::pop 9030 631 276 3666803 0 3666803 0
bgc::inetstack::tcp::established::background 11077 1911 831 13245809 0 3368554 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11077 708 301 3188016 0 3188016 0
ioc::network::libos::pushto 745 4156 1642 3042024 0 3042024 0
ioc::network::libos::connect 42 39520 16798 1693281 0 1693281 0
demikernel::sgaalloc 6507 207 84 1025947 0 1025947 0
bgc::inetstack::tcp::passiveopen::background 42 30141 13556 993994 0 993994 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11023 856 344 948434 0 948434 0
demikernel::sgafree 8752 91 41 711830 0 711830 0
ioc::network::libos::close 46 6495 2709 290732 0 290732 0
ioc::network::libos::accept 38 2355 1171 56192 0 56192 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;udp::receive 242 198 81 45195 0 45195 0
bgc::inetstack::arp::background 7 2854 1322 12859 0 12859 0
bgc::inetstack::icmp::background 1 1823 808 1917 0 1917 0

@github-actions GitHub Actions
Copy link

libos = catnap
commit id = b20e1da

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::catnap::transport::epoll 10793524 3841 1603 5776770987 100 5776770987 100
ioc::network::libos::push 17600 538 223 7974252 0 7974252 0
ioc::network::libos::pop 15055 519 217 5751885 0 5751885 0
ioc::network::libos::connect 50 175438 73728 2996226 0 2996226 0
ioc::network::libos::accept 26 111716 46807 1395168 0 1395168 0
demikernel::sgaalloc 11289 126 53 1171088 0 1171088 0
demikernel::sgafree 11537 75 32 761796 0 761796 0
ioc::network::libos::pushto 1315 343 142 438711 0 438711 0
ioc::network::libos::close 18 23375 9703 330781 0 330781 0

@github-actions GitHub Actions
Copy link

libos = catnip
commit id = b20e1da

collapsed_stack num_calls cycles_per_call nanoseconds_per_call total_duration percent_total_duration total_duration_exclusive percent_total_duration_exclusive
bgc::inetstack::poll;inetstack::layer4::poll_once;catnip::runtime::receive 22894410 214 86 4873024940 19 4873024940 43
bgc::inetstack::poll;inetstack::layer4::poll_once 22894410 399 160 8944128541 35 4066278391 36
bgc::inetstack::poll 11447205 1011 405 11345984181 45 2401855640 21
ioc::network::libos::push 16145 1291 517 17809364 0 15603607 0
ioc::network::libos::pushto 4585 1423 624 6481501 0 5303119 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch 2528 4160 1627 4906994 0 4868763 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11059 349 139 4753134 0 4752151 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11071 390 153 3004049 0 3001011 0
bgc::inetstack::tcp::established::background 11071 1184 468 11416618 0 2949681 0
ioc::network::libos::pop 9139 405 160 2789474 0 2789474 0
ioc::network::libos::push;catnip::runtime::transmit 8763 370 148 2394822 0 2394822 0
ioc::network::libos::connect 42 33448 13525 1347332 0 1288728 0
ioc::network::libos::pushto;catnip::runtime::transmit 4585 264 116 1178382 0 1178382 0
demikernel::sgaalloc 6745 147 59 792563 0 792563 0
demikernel::sgafree 8970 96 38 748730 0 748730 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11016 311 121 709754 0 699368 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;udp::receive 2194 252 102 535243 0 535243 0
bgc::inetstack::tcp::passiveopen::background 42 24946 9742 551968 0 499300 0
ioc::network::libos::close 49 5926 2371 271276 0 271276 0
ioc::network::libos::connect;catnip::runtime::transmit 42 6749 2683 58604 0 58604 0
bgc::inetstack::tcp::passiveopen::background;catnip::runtime::transmit 21 15377 5952 52668 0 52668 0
ioc::network::libos::accept 37 2115 829 46275 0 46275 0
bgc::inetstack::arp::background 7 3300 1313 24976 0 24976 0
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger;catnip::runtime::transmit 19 1352 550 18907 0 18907 0
bgc::inetstack::poll;inetstack::layer4::poll_once;inetstack::layer4::receive_batch;catnip::runtime::transmit 24 619 244 11605 0 11605 0
bgc::inetstack::tcp::established::background;tcp::established::background::sender;catnip::runtime::transmit 21 586 233 10386 0 10386 0
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter;catnip::runtime::transmit 13 653 270 6114 0 6114 0
bgc::inetstack::icmp::background 1 1674 668 1831 0 1831 0

iyzhang
iyzhang approved these changes Apr 15, 2025
View reviewed changes
Copy link
Contributor

@iyzhang iyzhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me. I mostly looked at the changes outside of the catpowder/win directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anandbonde anandbonde anandbonde left review comments

Copilot code review Copilot Copilot left review comments

@iyzhang iyzhang iyzhang approved these changes

Assignees

@kyleholohan kyleholohan

Labels
bug Something Isn't Working enhancement Enhancement Request on an Existing Feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kyleholohan @anandbonde @iyzhang