[ENH] Add token bitmap FTS index

[Sicheng-Pan]

Summary

Second PR in the full-text index redesign series. Adds FullTextBitmapWriter and FullTextBitmapReader — the core index that hashes word tokens to 2^20 buckets and stores one RoaringBitmap per bucket in a blockfile.

High-Level Plan (for context)

1. Word-based tokenization (PR 1)
2. Hash bucket bitmap index (this PR)
3. Popular word dictionary for substring/partial match support
4. Two-stage query with brute-force document verification
5. Segment integration with dual dispatch
6. Regex support via dictionary scan + bitmap lookup

This PR

Writer (FullTextBitmapWriter)

• DashMap<u32, BucketDelta> for lock-free concurrent accumulation
• add_document / delete_document with mutual exclusion (add clears pending delete, delete clears pending add) — last operation wins per (bucket, doc_id)
• write_to_blockfiles writes buckets in sorted key order, merges with existing bitmaps when forking via old_reader
• Hash: murmur3_32(token, seed=0x5f3759df) % 2^20
• Blockfile schema: ("", bucket_id: u32) → RoaringBitmap

Reader (FullTextBitmapReader)

• search(query) tokenizes via WordAnalyzer::tokenize_query, hashes each token, loads bucket bitmaps, AND's them together
• Prefix/suffix tokens are hashed directly for now (TODO: dictionary-based partial matching in PR 3)
• get_bucket returns Result, errors propagate properly

Fork support

• Writer accepts Option<FullTextBitmapReader> as old_reader
• On write: result = (existing | adds) - deletes
• Tested end-to-end with arrow blockfile provider

Tests

• 13 tests: add, multi-doc, delete, update-via-delete-then-add, short token filtering, Unicode normalization, search (single word, multi-word, no match, Unicode, reject short query, after delete), fork with old reader

[Sicheng-Pan]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [ENH] Add token bitmap FTS index #7001 👈 (View in Graphite)
• [ENH] Add word based FTS tokenizer #7000
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Review found no issues across the examined changes, and the enhancement appears ready to merge.

Status: No Issues Found | Risk: Low

Review Details

📁 5 files reviewed | 💬 0 comments