You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we do lots of unnecessary file IO when writing a new batch to the BucketList. Specifically, we create a fresh bucket from a vector of entries, write it to disk, and open the file to index the bucket. Then we open the new bucket file and the current level 0 curr bucket file and kick of a very low priority background thread. This background thread does a merge via file IO, then again reindexes the merge output. During all this, we are blocking the main thread, waiting on this low priority background work.
Instead of launching a background thread, we should do all this work on the main thread. We should also only index buckets once, and since we have all the entries in memory already anyway, we should avoid file IO for level 0 merges.
The text was updated successfully, but these errors were encountered:
Currently, we do lots of unnecessary file IO when writing a new batch to the BucketList. Specifically, we create a fresh bucket from a vector of entries, write it to disk, and open the file to index the bucket. Then we open the new bucket file and the current level 0 curr bucket file and kick of a very low priority background thread. This background thread does a merge via file IO, then again reindexes the merge output. During all this, we are blocking the main thread, waiting on this low priority background work.
Instead of launching a background thread, we should do all this work on the main thread. We should also only index buckets once, and since we have all the entries in memory already anyway, we should avoid file IO for level 0 merges.
The text was updated successfully, but these errors were encountered: