The embedding model (bge-m3 in this case) has a sequence length of 8192 tokens, i.e. rlama tries to embed the whole book, but Ollama can only put the first few pages into the embedding request.

Then when retrieving, it retrieves the entire document instead of the relevant passage (because there is no chunking), but truncates this to the first 1000 characters, i.e. the first half-page of Table of Contents.

As a result, when queried, the model says: "There is no direct mention of the Buddha in the provided documents." (The word Buddha appears 44,121 times in the documents I indexed.)

A better solution (and, as far as I can tell, what every other RAG does) is to split the document into chunks that can actually fit the context of the embedding model, and then retrieve those chunks -- ideally with metadata about which part of the document it's from.

---

I'd also recommend showing the search results to the user (I think just having a vector search engine is already an extremely useful feature, even without the AI summary / question answering), and altering the prompt to provide references (e.g. the based on the chunk metadata like page number).

- as an end user, some primary concerns re apps using the file system:

   - who will be able to read it? does the app share data?
     - I'm not thinking about a privacy policy, but a hard block that would not allow any internet access for the binary/app. Would rlama still work correctly ?

   - is the app able to modify/delete files?
      - it should be ensured that there is no "full file system" access, ie just read permission

- really crisp website: did you code it from scratch or is it template-based?

I like the fact that it is written in Go and small enough to skim over the weekend, but after repeatedly burning my time on dozens of llm ecosystem tools, I'm careful in choosing to even explore the code myself without seeing these basic disclosures upfront. I'm sure you'd see more people adopting your tool if you can provide a high-level overview of the project's architecture (ideally in a visual manner)

To me getting my data from my notes correctly is most important. I use AI tools for coding occasionally (which I can easily verify on my own), for anything else I can never bring myself to be doubtless about the output.