StateMarketDeals performance optimization

[dumikau]

Checklist

• This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
• I have a specific, actionable, and well motivated feature request to propose.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

What is the motivation behind this feature request? Is your feature request related to a problem? Please describe.

We at Glif Nodes are producing a StateMarketDeals JSON dump for the ecosystem to use, and it takes a lot of time (40 min - 1 h) and resources (r6g.8xlarge AWS EC2 instance, which sometimes OOMs still) for lotus to produce it.

Describe the solution you'd like

Would it be possible for lotus core to adopt an approach similar to https://github.com/aschmahmann/filexp/commits/f05dump/, which seems to produce the same response in a fraction of the time? The code is there and is ready to take

Describe alternatives you've considered

No response

Additional context

No response

[rvagg]

Specifically https://github.com/aschmahmann/filexp/blob/f05dump/internal/state/f05.go#L34; which I think benefits from 2 major differences: parallelising the inner part of the ForEach and streaming write of the output rather than accumulating (an ongoing frustration of mine about Lotus APIs, typical Go "build it then spew it" style). We could certainly parallelise the inner loop, but streaming would break the API and I think only make it available over websockets. Given how expensive it is maybe breaking it is not such a bad thing. We're looking at /v2 APIs at the moment and I'd suggested that this is one that shouldn't be migrated, at least as it is, as it's too expensive.

[masih]

@dumikau We are currently working on the v2 APIs that aims to take this kind of pain away for users.

What I would love to learn is: How popular is the service Glif deal snapshotting provides?
I am curious to understand if the functionality offered by StateMarketDeals is less used because it is cumbersome to use ( i.e. all or nothing which is effectively demands excessive amount of hardware, which in turn gives birth to the need for periodic snapshots), or because it is not that useful to begin with.

As an organisation working with this specific API extensively, I'd love your input to understand what changes could make the most impact. Thanks