Semantic Search over a Flownix Project (RAG)

Every project has a semantic index built from its node content and comments, queried through three read-only MCP tools. There are no indexing tools — the index maintains itself. Prerequisite: flownix-basics.

RAG vs the structural tools

RAG is recall, not truth — ranked chunks, possibly seconds stale. Once a hit looks relevant, get_node it and work from the authoritative node. Never quote acceptance criteria or status straight out of a chunk.

rag_query — ranked semantic search

rag_query({
  project_id: "<project_id>",                 // required
  query: "rate limiting on the public API",   // required, natural language
  k: 5,                                       // default 5
  kind_filter: "task",                        // epic|feature|plan|task|doc
  status_filter: ["in_progress"],             // restrict to node statuses
  component: "AF-COMP-3",                     // restrict to one component's direct links
  min_score: -1,                              // relevance floor override, see below
  max_content_chars: 600                      // snippet length
})

Returns results[] (node_id, slug, kind, title, content, tags, status, score, content_truncated) plus query_time_ms, filtered_out, min_score_applied, and a note only when the floor removed everything.

Reading score, and what an empty result means

Scores live in a narrow band — e5 models compress cosine similarity. Measured on an index built with multilingual-e5-small:

These numbers belong to that model, not to the search. The floor is a property of whichever embedding model the index was built with, and a different model spreads cosine differently — so read the floor actually in force from min_score_applied in the response rather than assuming the figure above. A min_score_applied of 0 means no floor is in force: either you asked for none, or the active model has not been calibrated yet.

Because "perfect hit" and "utterly unrelated" are ~0.10 apart, a raw ranked list is not self-evidently meaningful, so the service applies a relevance floor (min_score). That makes an empty result informative:

• results: [], filtered_out > 0 → the project has content and none of it answers your query. A real answer: say so or rephrase. Don't reason from weak matches, and don't conclude the project is empty.
• results: [], filtered_out == 0 → nothing matched the filters, or nothing is indexed. Check rag_status before claiming anything.

min_score: -1 switches the floor off when you deliberately want to sift weak matches.

content is a snippet, not the ticket

Capped at 600 chars (max_content_chars), with content_truncated: true when cut. Deliberate: rag_query decides which node to open, and full bodies would burn context restating what get_node returns authoritatively.

Also worth knowing:

• Asking about current work? Use status_filter. An established index is overwhelmingly finished work, so "what is in flight" unfiltered returns done tickets that look like an answer. Use status_filter: ["in_progress", "planning", "backlog"] — or list_nodes, the right tool for a question about status rather than meaning.
• Ask in natural language, not keywords: "how do we authenticate MCP clients" beats "auth mcp token".
• Ask in the user's language. The index runs intfloat/multilingual-e5-small: Russian queries retrieve Russian content as well as English retrieves English. Cross-language retrieval usually works but was not specifically measured — if it comes back thin, retry in the other language before concluding nothing exists.
• Comments are indexed with their parent node, and ranked by their mark: [decision] and [done] rank as high as node bodies, [progress] is heavily discounted. Another reason to mark comments properly.
• component filters by direct links only, matching list_nodes — component progress is what counts recursively.

rag_context — one context block for a prompt

rag_context({ project_id, query: "what I need to know to implement SSE for sync status",
              max_chars: 10000 })

Returns markdown_context, sources_count, total_chars. The floor matters more here: a weak chunk reaches your prompt as background, with no score to discount it by. sources_count: 0 means "nothing relevant to inject", not "the tool failed". max_chars truncates — a smaller budget means fewer sources, not shorter ones.

rag_status — is the index worth trusting?

Returns total_chunks, status_counts, vector_dimension, last_indexed_at, needs_reindex, index_embedding_model, plus the sync breakdown pending / in_progress / synced / failed / skipped.

• pending/in_progress > 0 → recent edits are still catching up; fall back to get_node for anything you just wrote.
• failed > 0 → nodes are missing from the index. Say it is degraded rather than "nothing exists".
• skipped → nodes with nothing to index (no content, no non-empty comments). A finished state, not a backlog.
• total_chunks == 0 → never indexed (brand-new project, or rag-service was down).
• needs_reindex: true → stored vectors came from a different embedding model (index_embedding_model names it; "mixed" means the rows disagree). rag_query/ rag_context are then rejected, not degraded — FailedPrecondition … full reindex required. You cannot fix it from MCP: report that an operator must trigger a full reindex, and fall back to get_tree/list_nodes.

How sync works (why there are no indexing tools)

Automatic and eventually consistent: any node/comment mutation publishes a best-effort job onto NATS JetStream (a comment re-indexes its parent node); rag-service consumes it and runs fetch → chunk → embed (ONNX multilingual-e5-small, 384-dim) → upsert into the project's Qdrant collection, driving pending → in_progress → synced (or failed). Delivery is at-least-once and reindexing is idempotent, so retries are harmless; a reconciliation sweep (~10m) re-submits anything not cleanly synced.

Consequences: write-then-search is racy — never verify your own write with rag_query, use get_node. You cannot force a reindex over MCP. Index quality is a direct function of ticket quality.

Typical patterns

// Before planning — avoid duplicating existing work
rag_query({ project_id, query: "<the feature request in one sentence>", k: 8 })

// Before executing — load the knowledge the ticket doesn't repeat
rag_context({ project_id, query: "<task title> — prior decisions, conventions, related modules" })

// During review — the decisions the change must stay consistent with
rag_query({ project_id, query: "<subsystem> architecture decision", kind_filter: "doc" })

// "Why is it like this?" — RAG's strongest case: the [decision] comment usually comes first
rag_query({ project_id, query: "почему выбрали <технологию> для <подсистемы>", k: 5 })

// "What's in flight?" — semantic search alone answers this badly
rag_query({ project_id, query: "<topic>", status_filter: ["in_progress", "planning"] })
list_nodes({ project_id, status: "in_progress" })   // best when there is no topic

Write for retrieval

• Spell out the subject in the title and first paragraph — "Sync status SSE stream", not "Part 3".
• Record decisions as [decision] comments with the why, not just the what.
• Use the domain words a future searcher would use, including the alternatives you rejected.
• Keep content truthful (update-knowledge) — stale content ranks just as well as correct content, which makes it worse than none.

Rules

• RAG is read-only. It never changes nodes, status, or the index.
• Always pass project_id — results are scoped to one project; there is no cross-project search.
• Treat hits as leads: confirm with get_node before acting or quoting. content is truncated, so this is not optional.
• On rag-service unavailable, degrade to get_tree/list_nodes and say so — don't retry in a loop.
• Empty results ≠ "doesn't exist" — but with filtered_out > 0 it does mean "nothing indexed answers this". Distinguish the two.
• Never pad a thin result. One hit clearing the floor is one hit.
• Use status_filter (or list_nodes) for anything about current work.