Rabbit Hole Internet Indexer — Conceptual Blueprint (v2)

Intro This is a conceptual blueprint for an AI-driven file/content indexer inspired by old-school "FTP-style" directory indexing — the idea that you can crawl a tree of paths, catalog what's there, and let people search it. The v1 update here keeps that portability as the core design constraint: the system should work the same way whether it's pointed at a local drive, an old FTP server, a cloud bucket, or a web crawl target. No component should assume a specific vendor, cloud, or file format. Swap the storage layer, swap the index backend, swap the embedding model — the pipeline shape stays the same. What's changed from the original 2024 draft: - The indexing/search layer is described in **vendor-neutral terms** (any vector store, not "Elasticsearch specifically") so the design stays portable as tools change. - Hybrid search (keyword + semantic) replaces pure vector similarity, which is the standard approach now — vector-only search underperforms hybrid on exact terms like model numbers, case citations, or filenames. - Chunking is added — embedding a whole PDF as one vector loses precision; chunked embeddings retrieve much better on long documents. - A permissions/access-control note is added, since "search everything" is dangerous the moment this touches a real file system with private content. - The code sample is fixed to run (the original had smart quotes from a word processor, which breaks Python) and updated to reflect chunking + hybrid retrieval.