Index fixes

[JubilantJerry]

Description

Fixes for indexing and @codebase issues #4239, #4252, #4254, #4255

[netlify]

✅ Deploy Preview for continuedev ready!

Name	Link
🔨 Latest commit	963713d
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/67c6692e56001200088ba99f
😎 Deploy Preview	https://deploy-preview-4322--continuedev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[Patrick-Erichsen]

Hey @JubilantJerry , apologies for the slow review time here, it's been a busy past few weeks prepping our 1.0 launch.

This all looks solid but mind sharing a bit of an overview of what these changes entail? Also, if there's any tests you could write in core/indexing/LanceDbIndex.test.ts, core/indexing/FullTextSearchCodebaseIndex.test.ts, core/indexing/refreshIndex.test.ts, and core/indexing/CodebaseIndexer.test.ts to help us validate these changes that would also be much appreciated. All of the issues you linked to are pretty big problems so definitely want to get these changes in but would like to make sure we have tests so we don't have regressions in the future.

[Patrick-Erichsen]

@JubilantJerry I went through some of the issues you linked actually and that provides plenty of context for the changes here.

I pushed up a quick merge conflict resolution just now, and once the tests pass here I'm going to merge to main. However, if there's any test you think would be valuable to add to help us prevent future regressions that would be a great followup.

Thanks again for all the huge improvements here, and for documenting the issues in the first place!