I Built a Company Brain That Refuses to Guess, and It Caught Me Lying About My Own Test Suite

OrgMemory answers hard engineering questions by traversing a FalkorDB graph instead of guessing. It found an incident with no recorded cause, forced me to rewrite a false claim in my own README, and turned my test suite into the thing that caught two real bugs. This is what I built at Graph Hacks, and what I would do differently.

I Built a Company Brain That Refuses to Guess, and It Caught Me Lying About My Own Test Suite

Introduction

I went into Graph Hacks wanting to build an agent that could answer questions about a codebase. I shipped something else, because of a sentence I wrote into my own seed data.

The part I did not expect was how much of the work turned out to be about refusing to answer things. The agent's best moment in the demo is the one where it admits it has nothing.

The project is called OrgMemory. It connects people, services, projects, incidents, decisions and documents into one FalkorDB graph, then answers questions that need multi-hop reasoning. The whole demo hangs on one question: "Can we replace Payment Service A with Payment Service B? What could break, who should review it, and why do we use the current service?"

Here is the problem I kept running into, what the graph actually fixed, the two bugs my own tests caught, and the moment the project caught me.

The problem

Company knowledge is scattered across runbooks, ADRs, postmortems, tickets, and whoever happens to remember. Retrieval-augmented chat over that material returns isolated snippets.

It finds the postmortem. It does not find the fact that the service named in the postmortem has three other services depending on it.

Here is the shape of the problem I kept explaining to people:

"Can we replace Payment Service A with Payment Service B?"

There is no answer to that inside any single record. You need to know which services depend on A, who owns those, what has broken there before, and which decision got us to A in the first place. Those are joins over a topology. They are not sentences to rank.

And the second half of the problem is worse. When you ask a model to reason over documents like these, it will produce a cause for every incident you give it, because a postmortem that does not say why something broke is a gap, and models fill gaps.

Why a graph, specifically

I did not start with FalkorDB. I started with the question, and the question turned out to be relational in four specific ways.

Reverse impact traversal. Since (A)-[:DEPENDS_ON]->(B) means A breaks if B breaks, "what breaks if this changes?" runs backwards along the edge. That is one bounded variable-length query, not a loop:

MATCH p = (impacted:Service)-[:DEPENDS_ON*1..3]->(target:Service {id: $service_id})
WITH impacted, min(length(p)) AS depth
RETURN impacted.id AS id, impacted.name AS name,
       impacted.criticality AS criticality, depth
ORDER BY depth ASC, impacted.criticality ASC, id ASC
LIMIT $limit

The demo returns 8 downstream services across 3 hops. That is not a calculation I layered on top of a graph. That is the traversal.

Provenance as structure. When an investigation finishes, its citations get written back as (Investigation)-[:CITED]-> edges pointing at the services, people, incidents, decisions and documents it actually used. History ends up in the graph, so a follow-up question can traverse from a past answer back into exactly what it touched.

Ask "who should review the replacement?" right after the first question and the agent finds the earlier investigation instead of starting cold.

Supersession chains. SUPERSEDES answers "why do we use this?" with the decision that replaced the decision you asked about, and what superseded that in turn. In the demo, one of the four architecture decisions is already marked superseded, which means the organisation already decided to move off the processor someone is now asking about.

Structured absence. This is the one I care about most. One seeded incident has no CAUSED_BY edge. Because a graph can represent a missing edge, "we don't know why this happened" comes back as a result rather than as a gap the model fills with something plausible:

MATCH (incident:Incident)-[:AFFECTED]->(service:Service {id: $service_id})
OPTIONAL MATCH (incident)-[:CAUSED_BY]->(cause:Service)
OPTIONAL MATCH (resolver:Person)-[:RESOLVED]->(incident)
RETURN incident.id, incident.title, incident.severity,
       incident.root_cause_known, cause.name AS cause

OPTIONAL MATCH leaves cause null and root_cause_known false, and the writer turns that into an explicit unknown. The absence of an edge is the result.

The graph, in full

Seven node labels and eleven relationship types carry the entire demo. Every node has a stable string id like svc-checkout, which is what makes citation edges possible.

That is the whole thing. No embeddings, no vector store, no similarity search anywhere. A question is answered by matching patterns in this table and following them.

The seeded company, "Northwind Logistics," is 52 nodes and 124 relationships, all fictional. Small enough to reason about by hand, which is how I found most of the bugs below.

The agent picks from a fixed set of tools

The agent does not write Cypher. It chooses among 28 fixed query templates, each with a name and a description, defined in backend/app/agent/tools.py. Every one is a real bounded traversal.

This matters more than it sounds. If the model can compose its own query, a hallucinated relationship type is one bad generation away. The allowlist makes the graph both the reasoning substrate and a hard boundary.

Two bugs my own test suite caught

This is the part I keep telling people about, because it is the least glamorous and most useful thing in the project.

I built an in-memory repository from the same seed data so the test suite could run without a database and without a model. 218 tests, no API keys. Then I marked some of them as integration and ran the same assertions against a live FalkorDB instance.

They did not agree. Twice.

The first one: .result_set returns positional rows, not dicts. My in-memory model returned dicts, every call site used row["id"], and against the real driver that silently indexed the wrong column. Passes in memory, wrong answers in production.

The second one, and this one is a FalkorDB behaviour I did not expect: a parameter inside a variable-length relationship range gets rejected. [:DEPENDS_ON*1..$max_depth] fails with:

Encountered unhandled type in inlined properties

A parameter is not allowed inside the range. The depth bound is a genuine safety limit the caller must control, so the fix is to clamp it to a Python int in build() before interpolating, so no caller-supplied string can ever reach the query text.

Both of these would have shipped. The in-memory tests were green the entire time.

The takeaway I keep coming back to: a test double that is more forgiving than the real thing is worse than no tests, because it converts unknown into false confidence.

The moment it caught me lying

I wrote a line into the README and forgot about it. It said the seeded graph was 52 nodes and 124 relationships.

Then I recorded the demo video. On screen, the UI reads 54 nodes, 138 edges.

Both numbers were right, and I still went looking for the bug, because a demo that disagrees with its own documentation is exactly the kind of thing a judge notices. The answer was that the demo's own investigations create Investigation nodes and CITED edges as they run. The graph was growing while I filmed it.

So I left it in and added the explanation. But the honest version of this story is that I had written an unverified number into a README, believed it, and only found out because I put a video next to it.

That is a small thing. It is also the exact failure mode this whole project is about.

What I learned

Five things I would tell someone building an agent over a graph:

1. Put the absence of a fact in your data model. The single highest-value decision in this project was seeding an incident with no CAUSED_BY edge so the agent would have to report an unknown. It turns "the model hallucinated" into "the graph had nothing to give."

2. Make your test double as strict as the real system. My in-memory repository was friendlier than FalkorDB, and that friendliness hid two real bugs for the entire project. If the double is easier, it is not testing the thing.

3. A tool allowlist beats a tool-writing model. Constraining the agent to 28 named traversals is what makes the grounding enforceable at all.

4. Traversals are fast enough to run per step. The reason this architecture works and the batch-then-LLM pattern does not is that a graph query is cheap enough to sit inside the agent loop, once per reasoning step, instead of as a precomputation.

5. Deterministic fallback is a feature, not an apology. Every LLM node has a non-model path that writes an answer straight from graph evidence. With LLM_ENABLED=false the whole app runs, and the test suite runs exactly that way. A demo should never depend on a metered endpoint being reachable.

Outro

I set out to build an agent that answers questions about a codebase, and ended up with an agent that mostly tells you what it does not know, a graph that records its own citations, and a test suite that caught two bugs my mocks were hiding.

The demo video is 51 seconds. The question is one sentence. The graph underneath is 52 nodes and 124 relationships, and by the end of the recording it is two nodes heavier because the agent was writing down where its answer came from.

I still owe this project an MCP server. That is next.

If this clicked, you will probably enjoy How LLMs Actually Generate Text or Build Your Own MCP Server from Scratch.

Credible Sources

1. FalkorDB documentation. Graph database built for AI agents, with Cypher support and multigraph topology.
2. FalkorDB: Cypher. Query syntax, including the variable-length relationship ranges used for impact traversal.
3. LangGraph documentation. Bounded agent execution with explicit recursion limits.
4. Property-based testing with Hypothesis. The approach behind the fuzzer that checks hostile values cannot survive into query text.
5. Neo4j: Graph Data Modelling. Why relationship-first modelling suits questions whose answers are paths, and where a property graph earns its keep over rows.

Subscribe to Our Newsletter