Comment by bob1029

I think BM25 is the right path for most use cases. I'd reach for a single lucene index here, making the commit hash one of the document's indexed fields. This would make FTS across history nearly instant. I'd store it in an ignored folder right next to .git.

For code search only, BM25 might be a bit overkill and not exactly what you want. FM indexes would be a simpler and faster way to implement pure substring search.

Maybe having both kinds of search at the same time could work better than either in isolation. You could frame them as "semantic" and "exact" search from the perspective of the LLM tool calls. The prompt could then say things like "for searching the codebase use FunctionA, for searching requirements or issues, use FunctionB."