[Remote Store] Decouple refresh intervals from upload rates

[Bukhtawar]

Is your feature request related to a problem? Please describe

With remote store and segrep, data once refreshed on the primary has to be replicated to all replica copies via a remote store. The replication requires upload from the primary and then hydration back to all the replica copies. The network round-trip delay prevents keeping lower refresh intervals, which might be critical for users who might have lower tolerance to data staleness.

Describe the solution you'd like

We can improve data freshness on primaries by supporting lower refresh intervals while keeping uploads at a higher interval to optimise, too many small segment transfers happening between the primaries and the replicas, which eventually gets merged and needs to be re-transmitted.
Decoupling refresh rates and upload rates provides a nice way to independently control, data freshness while also controlling rate of segment replication. Queries that need higher data freshness can use the primary routing to fetch data, while other queries can use the eventually consistent replica copies to fetch data.

Related component

Storage:Performance

Describe alternatives you've considered

Keep as is.

Additional context

No response

[peternied]

[Triage - attendees 1 2 3 4 5]
@Bukhtawar Thanks for filing, looking forward to a pull request for this area