For architects and squad leads deciding how far to let an AI assistant into a large Salesforce programme. This is the design story behind ai-ready-sf-repo v0.1.0, and it puts the ideas from Intent → Spec → Test and Archaeological Specs into a repo you can clone.
TL;DR
- The build is the only evidence. Every claim cites
path:lineand carries a label: Confirmed, Inferred or Unverifiable. Counts come from a query over a metadata index, never from the model's memory. - The assistant recommends; a named person confirms. Stories, designs and impacts become records only when a person promotes them, and open questions block promotion.
- Rules live in code, not prompts. A hook blocks deploys, pushes and out-of-scope writes on the laptop. CI and branch protection decide what merges.
- One real run, honestly reported. It found a question that shrank the change, a criterion no test could prove, and a conflict the builder flagged instead of hiding.
What You'll Learn
- Why a longer system prompt is the wrong place for an AI assistant's rules
- How evidence labels and a metadata index stop confident guessing
- How the chain from story to code keeps a person in charge of every decision
- What one end-to-end run caught, and what the framework does not do yet
The Problem
In a large Salesforce org, writing the trigger is the easy part. The hard parts are knowing what a change will touch, knowing which other squad is changing the same class, and knowing that someone with the authority to say yes actually said yes.
An AI assistant makes the easy part cheaper and the hard parts worse. Left to itself, it fails in predictable ways:
- It answers confidently about metadata it never opened.
- Its counts are guesses.
- Ask it to "check in UAT" and it will try.
- Two squads ask it about the same
CaseTriggerHandleron the same day, and neither hears about the other. - Nobody records who approved its output, or whether anyone did.
- A story that says "saves quickly" gets built as written.
- The code moves on after a change is sized, and the analysis still reads as current.
- Asked to fix one story, it tidies three other files while it is there.
- Code arrives that no test ties back to a criterion.
- A record ID or a customer email slips from an export into a commit.
- When two requirements clash, it quietly picks one.
The usual fix is a longer prompt: "only cite real files", "never deploy", "ask before changing scope". A prompt is a request, though, and a model can misread a request. We wanted rules that hold even when the model gets it wrong.
Common Questions This Article Answers:
- How do I stop an AI assistant from making up facts about my org?
- Who should approve what an AI assistant produces, and how is that recorded?
- What actually stops an assistant from deploying or editing files it should not touch?
Quick Answer
Give the assistant a repo where every rule that matters is enforced by a script, a hook or a CI check. ai-ready-sf-repo indexes the checked-out metadata and requires every claim to cite a file and line with an evidence label. It splits work into a chain (refine the story, design options, size the impact, package the scope, build) where the assistant drafts each step and a named person confirms it with a command run in their own terminal. A guard hook enforces modes on the laptop, and the same checks run again on every pull request, where branch protection decides what merges.
Each Problem, and What Stops It
| Problem | What the repo does | Where it is checked |
|---|---|---|
| Claims about metadata it never opened | Every claim cites path:line and carries an evidence label |
The impact validator, at promotion |
| Counts it guessed | Counts come from query.py over the index, at a stated commit |
A written rule, not code: query.py supplies every number, so none needs typing |
| Reaching into the org | Org calls, deploys and pushes are blocked in every mode | The guard hook, on the laptop |
| Two squads on one class | The register flags overlaps when the impact is drafted | story-impact, then the impact check on every pull request, which reports overlaps but does not fail |
| No record of who said yes | A named person runs promote.py --confirmed-by, and open questions block it |
promote.py; the confirmed record is committed to git |
| Vague stories | Vague words such as "quickly" fail the criteria check | check_ac.py, at promotion and in CI |
| Stale analysis | The packager refuses when anything under the source root changed since the index stamp | package.py |
| Edits beyond the story | Build mode writes only the files the package lists; anything else becomes a written scope request | The guard's scope lock |
| Code no test proves | Code stays blocked until every criterion id appears in a test file in scope | The guard's tests-first rule; check_trace.py reports gaps on the pull request |
| Real data in commits | Record IDs, emails, credentialed URLs and session IDs are blocked | scan_data.py, at promotion and in CI, and at commit if you install the pre-commit hook |
The last problem has no row because no code stops it. In the run below, the builder reported a clash instead of picking a side, but that came from the skill's instructions. The pull request reviewer is the backstop.
A few words you will see
| Term | What it means here |
|---|---|
| Index | A metadata dependency index: a graph of the checked-out metadata, built by a script and stamped with a git commit |
| Evidence label | Confirmed (the build shows it), Inferred (reasoned from the build) or Unverifiable (org-only, managed package, missing input) |
| Promotion | A person running promote.py --confirmed-by "<role>" to turn a draft into a committed record |
| Register | The committed stories, designs and impacts in impacts/, which other squads' analyses read for overlaps |
| Build package | The exact files a story may modify, create and test, plus its criteria and done commands |
| Guard | A Claude Code hook that runs before every tool call and enforces the current mode |
Rule 1: The Build Is the Only Source of Truth
The checked-out metadata is the only evidence the assistant may cite. Wikis, knowledge files and its own memory are leads to re-check, never proof.
That rule needs tooling, or it is just another prompt. build_index.py parses the source into nodes and reference edges, and query.py answers questions such as "what reaches this class within three hops". Anything countable comes from that query at a stated commit. The assistant never types a number.
Every claim in a report then carries a path:line or a negative finding ("zero hits for pattern X"), and one of three labels. The validators reject a report without them.
The labels earn their keep when they are uncomfortable. "Assignment rules in the org may overwrite this owner: Unverifiable, Salesforce Admin to confirm" is more useful than a confident guess, because it names the person who can actually answer.
Rule 2: The Assistant Recommends, a Named Person Confirms
Every step keeps a human in the loop. The assistant may analyse, draft and recommend, then carry on with its draft. It never approves, deploys, pushes or makes a binding decision. A draft becomes a record only when a person runs the promotion command, and promotion refuses on purpose:
- an open question nobody answered or explicitly deferred blocks it;
- vague acceptance criteria block it ("saves quickly" fails
check_ac.py); - evidence that does not open at the cited line blocks it;
- anything that looks like real data (record IDs, emails, credentials) blocks it.
Confirmed records live in git, in impacts/, and together they form a register. When a second squad drafts an impact that touches a component already on the register, the report says so, and so does the pull request check. Overlaps surface when the analysis is drafted, not as a merge conflict three weeks later.
Rule 3: Guardrails Live in Code, Not in Prompts
A hook runs before every Claude Code tool call. It has four modes: advise (read-only), normal (the framework's analysis scripts), build (writes, but only to the exact files one story's package lists) and maintain (framework upkeep). In every mode it blocks org calls, deploys, pushes, curl, and writes to its own files, settings and CI workflows. It also blocks shell tricks it cannot see through, such as variable expansion and heredocs.
The assistant cannot change modes. A person runs start.py in a real terminal, pastes two lines into their personal settings and restarts the session. A build session starts only when someone chose to start it.
The laptop hook is best effort, and the docs say so. Someone determined can work around a local hook. The control that holds is on the pull request.
The Chain, From Story to Code
Each step has one owner and leaves a reviewed file behind.
- story-refine turns a rough story into Given/When/Then criteria, edge cases seeded from the index, and open questions, each with an owner role.
- solution-design writes two to four options, always including "do nothing", with evidence-backed trade-offs and one provisional recommendation.
- story-impact sizes the chosen option: every component touched, with evidence, a label and its overlaps.
- package.py folds the confirmed records into a build package. It refuses if the source changed since the analysis.
- start.py opens a build session scoped to that package.
- implement-story writes a test for every criterion first, then the code, only inside the package. A file outside scope gets a written scope request, not a quiet edit, and
check_trace.pyproves every criterion has a test that names it.
None of these steps deploys anything. Apex compiles and runs in your own org-connected CI, after a named reviewer approves the pull request.
What One Real Run Taught Us
We ran the whole chain once on a demo story, DEMO-202: assign new cases to agents in turn. Four things came out of it.
An open question shrank the change. The first impact flagged two existing test classes that would break, because they create cases with an explicit owner. Refinement had not asked whether an explicit owner should be kept. The Product Owner answered "keep it", which became a new criterion, and both test classes dropped out of the impact. One extra question saved two unplanned test rewrites.
The packager caught a criterion no test could prove. Refinement raises an "org-only" edge case for each object: sharing settings and real data volumes are org configuration the repo cannot show. The packager made it a criterion that needed a test, and no Apex test in a repo can prove an org's sharing settings. We moved it to the open question that already covered it, and queued a framework fix so adopters do not hit the same wall.
The builder reported a conflict instead of hiding it. Because the code respects user-mode access, anyone without access to the new rotation object fails to save a case once the agent pool has members. That includes integration users and Web-to-Case. The Product Owner had said no case is excluded. The builder did not quietly pick a fix; it named the conflict for the Tech Lead and Salesforce Admin.
The score is honest, including the unflattering part. Against an answer key written before the design existed, the impact found every expected component (recall 1.0) and listed seven the key does not have (precision 0.46). Those seven are the new object, its field, two permission sets and three review items the chosen design needs. We left the key alone: a key edited after seeing a run is no longer a blind test.
What It Does Not Do Yet
- No Apex has compiled or run, the demo build included. The repo is offline by design; Apex is checked by your org-connected CI.
- No live model evals are scored yet. The harness and cases exist; there are no golden outputs or consistency figures.
- Walked through on a Mac only. Every deterministic check passes in CI on Ubuntu and Windows. Windows took six runs to go green: line endings, a null device that passed for a terminal, and paths on different drives. The Claude Code walkthrough itself has only been followed on macOS.
- One end-to-end run. DEMO-202 is a single story with a person answering along the way. It is an example, not a benchmark.
Frequently Asked Questions
Q: Why not just write a better system prompt?
A: A prompt asks; it does not enforce. The model can misread it, a long session can drift from it, and nothing records that it was followed. Each rule here is a script, a hook or a CI check, and the prompts only explain those rules to the assistant.
Q: Does the assistant ever touch a real org?
A: No. The guard blocks org calls, deploys and the old sfdx CLI in every mode, and the analysis runs on the checked-out metadata. Org exports and user inputs go in git-ignored folders, and a deterministic scanner blocks anything that looks like real data before a commit or a promotion.
Q: Is the guard a security boundary?
A: Not on a laptop. It catches mistakes early and makes the safe path the easy one. The control is CI with branch protection and code owners: required checks re-run the data scan, the Apex rules and the impact checks on every pull request, and the trace check reports any criterion without a test.
Q: Can I use it with GitHub Copilot?
A: Partly. The framework renders the same instructions for Copilot, but there is no Copilot guard yet, so Copilot users get the rules as text and rely on the CI checks for enforcement.
Key Takeaways
- Make evidence mandatory. A claim without a file and line, or without a Confirmed, Inferred or Unverifiable label, does not get through the validator.
- Put a named person on every decision, and let open questions block promotion until someone answers or defers them.
- Enforce rules in code, and treat the laptop hook as a convenience and CI as the control.
- Report the unflattering numbers. A precision of 0.46 against a stale key is more useful than a perfect score nobody believes.
What's Next?
Recommended Reading:
- Intent → Spec → Test: Linking Must-Nevers to Held-Out Tests, and When Not To, the traceability idea behind the criterion-to-test check
- Archaeological Specs: Using AI Agents Safely on a Salesforce Org You Didn't Build, why the assistant reads files instead of the org
- AI-Assisted Delivery Methodologies Compared, where this approach sits among the others
Action Items:
- Clone the repo and run the deterministic walkthrough check:
python3 tools/demo/walkthrough_check.py. - Follow
demo/WALKTHROUGH.mdin Claude Code on the synthetic org, one story from refinement to a build package. - Read
docs/ADOPTING.mdbefore copying anything into your own repo, and turn on the required checks it lists.
Responses
Checking your session.
Loading responses.