[v24.2.x] storage: work around segment index overflows

[vbotbuildovich]

Backport #25979 to branch v24.2.x. This issue was created as a placeholder because the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v24.2.x
git checkout -b manual-backport-25979-v24.2.x-146
git cherry-pick -x 0ed8d64254 fc1e8bfe72

git push origin manual-backport-25979-v24.2.x-146
gh pr create \
  --title "[v24.2.x] storage: work around segment index overflows" \
  --base "v24.2.x" \
  --label "kind/backport" \
  --head "manual-backport-25979-v24.2.x-146" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v24.2.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/25979 "

JIRA Link: [CORE-9998](https://redpandadata.atlassian.net/browse/CORE-9998)

[CORE-9998]: https://redpandadata.atlassian.net/browse/CORE-9998?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[WillemKauf]

#26024