Pourya Shahroudi
  1. Home
  2. Notes
  3. Quoted evidence

An LLM that had to quote its evidence

An order pipeline for a bakery, where the model could propose a status but code decided what counted.

La Bouchée takes orders through a quote form on its website, and each request lands in the business inbox. In the summer of 2026 I built a pipeline around that inbox. Each new request became an event on the bakery's calendar, marked tentative. The harder part was knowing what happened next, whether the customer had confirmed, declined or cancelled.

That answer usually sits in the email thread, in a customer's reply saying they want to go ahead or have changed their plans. Reading those replies is a fair job for an LLM. Letting an LLM write the result straight onto a business calendar is not, because a model that misreads one thread produces a confident, wrong order.

The model wrote a proposal, never a change

Each night a job gave the model the recent threads for orders still marked tentative. For each one it had to answer confirmed, declined, cancelled or tentative, and the prompt defined tentative as everything else, including a customer who sounds keen but has not committed. Any answer other than tentative had to come with a quote, between 15 and 200 characters, copied word for word from the thread. When in doubt, the instruction was to answer tentative.

The model had no way to act on its answer. It could not send email, draft email or touch the calendar. It wrote a file of decisions, and that was all.

Plain code decided what counted

A separate script, with no model in it, read that file afterwards. For every decision it:

  • refused the whole file if it was more than three hours old, so a stale run could never be applied late;
  • fetched the email thread again, read-only, and looked for the quote in it, ignoring case and spacing;
  • applied a decision only if the quote was there and the order was still tentative;
  • archived the file once it was done, so the same decisions could never be applied twice.

Even then, the only change it made on its own was tentative to confirmed, through the same code path a person's manual update used. A decline or a cancellation became a proposal for a person, with the exact reply line they could send to make it official.

The quote check is cheap and certain. A string is in the thread or it is not. It also has a limit worth stating plainly. It proves the quote exists. It cannot prove the quote means what the model said it means. Every decision other than a confirmation still went to a person.

Why it stopped

The weak point turned out to be the data source. Customers also confirm by text message and on Instagram, and a customer who starts by email often confirms by text. The inbox only ever held part of each order's story, so a status built from it was a guess that looked like a fact on the calendar. The bakery chose to keep order status by hand, and the pipeline came out in August, tests and all.

What I kept from it

The pattern carries over to most places I would put a model near real records. The model writes a proposal to a file and has no write access to anything that matters. Code checks whatever part of the proposal can be checked mechanically, and a quote is a good one to ask for because it turns part of a judgment into something a script can verify. A person decides whatever is left.

Before any of that comes a plainer question, which is where the truth actually lives. If it is spread across channels the system cannot see, the most an automation can honestly do is draft.

Notes

More notes

All notes