experiment: Evaluate recent research on optimizing LSM-trees using AI/ML techniques #5058
Description
Recent academic research explores AI and machine-learning-based approaches to optimize the performance and efficiency of LSM-tree storage engines. We should allocate time to carefully evaluate these approaches, prototype promising ideas, and determine their potential application to our storage engine.
Relevant Papers:
- CAMAL: Optimizing LSM-trees via Active Learning
- Leaper: A Learned Prefetcher for Cache Invalidation in LSM-tree based Storage Engines
- Learning to Optimize LSM-trees: Towards A Reinforcement Learning based Key-Value Store for Dynamic Workloads
Goals:
- Summarize key concepts and proposed benefits.
- Prototype and evaluate promising strategies from these papers.
- Provide recommendations on potential integration.
Jira issue: PEBBLE-1086