Intent → Spec → Test: Linking Must-Nevers to Held-Out Tests, and When Not To

For tech leads, developers and QA engineers who want agents to write code without shipping what nobody checked. This is the deep dive behind one of the eight approaches in AI-Assisted Delivery Methodologies Compared. It is a model I have been developing, so its costs are covered as plainly as its benefits.

TL;DR

• Three layers, written by people before the agent starts: intent (the outcome and a must-never list), spec (concrete rules in when/then form), and tests (visible ones the agent works against, and held-out ones it never sees).
• Every item carries the ID of the one above it, and CI fails the build if a must-never has no rule, a rule has no test, or a must-never has no held-out test.
• Held-out tests only work if the agent cannot read them. Ignore files and deny rules are a soft guardrail; keeping the tests in a repository only CI checks out is the hard one.
• It is too heavy for most changes. It is a modern requirements traceability matrix, and applied everywhere it produces hollow specs written to turn the check green. Use it for high-risk changes and route everything else to lighter approaches.

What You'll Learn

• How must-nevers, rules and tests link by ID, with a working Salesforce example
• How to keep held-out tests away from an agent, and why most guardrails are soft
• How to report a failed held-out test without teaching the agent to game it
• When not to use this at all, and how to route changes by risk instead

The Problem

AI coding agents are very good at making tests pass. That is the problem: if the agent can see the tests, it can shape the code to those exact tests, or write tests after the code that pass for the wrong reason. Code review does not catch it reliably once agents open more pull requests than people can read.

The usual fixes help but leave gaps. Specs drift from the code unless something checks them. Tests prove behaviour but not that the behaviour was the one the business wanted. This model joins the two and lets a machine check the join.

Common Questions This Article Answers:

• How do I stop an agent from writing code that only passes the tests I showed it?
• Where do I put tests the agent must not see, if it can read my whole repository?
• Do I really need specs and traceability for every change?

Quick Answer

Write a short must-never list for each high-risk change ("a guest user must never read Contact records"), turn each must-never into a rule with real values, and prove each rule with a test. Keep at least one test per must-never held out: in a repository that only CI checks out, so it never appears in the agent's workspace. Give every item the ID of the one above and make CI fail when a link is missing. When a held-out test fails, show the agent which must-never broke, not the test code. Do all of this only for changes where a mistake is expensive, such as authentication, permissions, billing, guest access and data retention, and use a fast test loop for everything else.

A few words you will see

Term	What it means here
Must-never	A plain statement of something that must never happen, with an ID such as MN-001
Rule	A concrete when/then statement that makes a must-never testable, with an ID such as R-014
Held-out test	A test the agent never sees, run only in CI, so it cannot shape its work to it
Traceability	The chain from must-never to rule to test, checked by machine
Property-based test	A test that generates random inputs on every run, so answers cannot be hard-coded

How the Three Layers Link

Layer	Answers	Written by	Fails on its own because
Intent	Why, and what must never happen	Product owner	The agent guesses the implementation
Spec	Exactly what correct looks like	BA and architect	It turns into waterfall in Markdown, or code written in YAML
Test	Machine proof that the code meets the spec	QA	The agent games the tests, or they check code paths instead of business meaning

The traceability rule ties them together. CI fails the build when:

• a must-never has no rule (the spec does not say what it means in practice);
• a rule has no test (nothing proves it);
• a must-never has no held-out test (the agent could have shaped the code to the visible tests alone);
• a new test also passes on the old code (it never failed, so it proves nothing about the change).

If a must-never cannot be tested, the design usually needs a better seam. Weakening the must-never is not the fix.

The Links on Disk: A Salesforce Example

The must-never, the rule that covers it and the held-out test that proves it each carry the ID of the one above:

<!-- intent/portal-security.md -->
### MN-001: Guest users never see Contacts
A guest (unauthenticated) site user must never read Contact records,
under any site configuration.

# spec/portal-security.spec.yaml
- id: R-014
  covers: [MN-001]
  when: A deployment changes the guest user profile, sharing or site settings
  then: The guest user has no read access to Contact
  tests: [GuestContactIsolationTest]   # held out: lives in the CI-only repository

// CI-only test repository, deployed to a scratch org or sandbox with the change
// MustNever: MN-001 | Rule: R-014
@IsTest
private class GuestContactIsolationTest {
    @IsTest
    static void guestUserCannotReadContacts() {
        insert new Contact(LastName = 'Isolation check');
        User guest = [SELECT Id FROM User WHERE UserType = 'Guest' AND IsActive = true LIMIT 1];
        Integer visible = 0;
        System.runAs(guest) {
            try {
                visible = [SELECT COUNT() FROM Contact WITH USER_MODE];
            } catch (QueryException e) {
                visible = 0; // no object access: the query is refused
            }
        }
        Assert.areEqual(0, visible, 'MN-001 violated: guest user can read Contact records');
    }
}

The test is a sketch to adapt to your own site. Note where it lives: not in force-app with everything else. Apex tests have to be deployed to an org to run, so a held-out Apex test sits in a separate repository that only CI fetches and deploys alongside the change. In CI that is two extra steps after the main deploy:

sf project deploy start --source-dir held-out-tests/force-app --target-org ci-scratch
sf apex run test --class-names GuestContactIsolationTest --target-org ci-scratch --wait 10

The check that every must-never has a rule, and every rule a test, can be a short script in the same pipeline that reads the intent and spec files and looks up each test name.

Keeping Held-Out Tests Out of the Agent's Reach

A held-out test only works if the agent cannot read it. There are soft and hard ways to do that:

• Soft: ignore and deny rules. Cursor reads a .cursorignore file. Claude Code uses permissions.deny rules in .claude/settings.json, such as Read(./held-out/**). Its documentation says these rules cover its own file tools and shell commands like cat, but not a grep -r run from a parent folder or a script that opens files itself, and points to OS-level sandboxing for real enforcement. Treat them as a guardrail, not a boundary.
• Hard: keep the tests out of the workspace. Put held-out tests in a separate repository that only CI checks out. The agent's working copy never contains them, so nothing has to block them.
• Control who can change them. Changes to the held-out repository need QA approval, so a held-out test cannot be quietly weakened to make a change pass.

Report the symptom, not the source

When a held-out test fails, the message the agent sees should name the broken invariant, not the test code:

✗ MN-001 violated: guest user can read Contact records (rule R-014)
  Full test output is in the CI log.

That tells the agent, or the developer, what to fix without showing the assertion it could aim at. The full output stays in the CI log, where the people debugging can read it.

Make some tests impossible to game

Property-based tests generate random inputs on every run, so an agent cannot hard-code answers even when it can see the test. Hypothesis (Python) and fast-check (JavaScript) are the common libraries. Apex has no mainstream equivalent, so on Salesforce this applies to code outside Apex, such as Lightning Web Components and integration services.

When Not to Use It

Linking must-nevers to rules to tests by ID and checking the links in CI is a modern version of a requirements traceability matrix, familiar from regulated industries. Applied to every change, it turns into bureaucracy. Worse, people (or agents) start generating hollow specs to turn the check green, which is less useful than having no spec at all.

Signs it has become too heavy:

• specs written after the code;
• must-nevers that only restate the rule beneath them;
• a green traceability check on specs nobody reads.

It also costs a BA or product owner willing to write must-nevers, and it is still unproven on platforms where most change is configuration rather than code. On Salesforce, Flow tests and queries against what was deployed can stand in for some unit tests, but the tooling is thinner than for Apex.

Route Changes by Risk Instead

Most teams should not pick one approach for everything. Route each change by how much damage a mistake would do:

Tier	Example changes	Approach	What it costs
1. Low risk	Layout changes, copy, internal tooling	A fast test loop	Fast tests, no spec
2. Medium risk	New endpoints, integrations other teams use, data migrations	Spec-first development plus test-first	Write the contract first, then build and test
3. High risk	Authentication, permissions, billing, guest access, data retention	Intent → Spec → Test	Must-nevers, linked rules, held-out tests in CI

Route by path where you can, so nobody has to remember. CI rules or code owners on the folders that hold permission sets, sharing rules, authentication classes and site settings put those changes in tier 3 automatically. Everything else defaults to tier 1 unless the story says otherwise.

Frequently Asked Questions

Q: Is this just a requirements traceability matrix?

A: It is the same idea, kept small and checked by machine. A traditional matrix is a document someone maintains by hand; here the links are IDs in files next to the code, and CI fails the build when one is missing. The risk is the same too: applied to everything, it becomes paperwork, which is why it belongs only in the high-risk tier.

Q: Can an agent write the held-out tests?

A: Not the agent that writes the code, or the tests stop being independent. A separate agent working only from the spec can draft them, but a QA engineer should review and own them, and they should live in the repository the coding agent cannot see.

Q: Does this work for Flows and configuration?

A: Partly. Intent and spec work for any change, and they pay off immediately for admin-led teams. The test layer is weaker: Flow tests and post-deploy queries can check some rules, but configuration has nothing as strong as Apex unit tests plus held-out tests yet.

Q: Do I need it for everything?

A: No, and you should not use it everywhere. Applied to every change it becomes a traceability bureaucracy. Use it for authentication, permissions, billing, guest access and data retention, and a fast test loop for the rest.

Key Takeaways

• Must-never, rule, test, linked by ID and checked in CI, makes gaps visible before any code is written.
• Held-out tests belong outside the agent's workspace. Ignore and deny rules are a guardrail, not a boundary.
• Report which must-never broke, not the test code, and keep the full output for people.
• Use it only where a mistake is expensive. Route by risk, and keep everything else light.

What's Next?

Recommended Reading:

• AI-Assisted Delivery Methodologies Compared, the eight approaches and which team each fits
• Guest user exposure, graded by real reachability, the kind of risk a must-never like MN-001 protects against
• Deployable permission sets for a delivery team, a least-privilege CI deploy user for the pipeline above

Action Items:

1. Pick one high-risk area (guest access, permissions or billing) and write three must-nevers for it.
2. Turn each into a rule with real values, and write one held-out test per must-never in a repository only CI checks out.
3. Add the traceability check to CI, and route only that area through it at first.

Resources & References

• Claude Code: Configure permissions
• Claude Code: Sandboxing
• Cursor: Ignore files
• Hypothesis (property-based testing for Python)
• fast-check (property-based testing for JavaScript)
• Birgitta Böckeler: Understanding Spec-Driven Development