Apex Test Data and CI Tooling: What to Use and What It Costs

This article is part of the Apex Testing series, three articles on writing Apex tests that catch regressions instead of chasing a number. Articles are standalone; no need to read in order.

TL;DR

• Most Apex test suites fail on data, not logic. A new required field or validation rule breaks two hundred tests that have nothing to do with it.
• A hand-rolled TestDataFactory is the right place to start and the thing that rots. It survives if it exposes overridable defaults rather than fixed records.
• Builder and fluent fixture patterns cost more up front and absorb org change far better, because each test states only what it cares about.
• Apex Test Kit generates whole object graphs declaratively, including relationships and unique field values. Strong for deep hierarchies, another unmanaged dependency to own.
• Test.loadData with a static resource keeps large fixtures out of Apex, at the cost of a CSV that silently drifts from the schema.
• In the pipeline: sf apex run test with Apex test suites, --code-coverage and a JUnit result format. Add Code Analyzer's PMD Apex rules, because they are what stops an assertion-free suite passing the coverage gate.

What You'll Learn

• Why test data, not test logic, is what breaks your suite
• Four ways to build Apex test data, with the failure mode of each
• How to run and filter tests from the sf CLI, and which flags matter in CI
• The deployment test levels, and when each is the right one
• Which static analysis rules actually enforce test quality, and how to gate on them

The Problem

Ask a Salesforce team what broke their pipeline last month and the answer is rarely a logic bug in a test. It is that someone added a required field to Contact, or a validation rule on Opportunity, or a new mandatory picklist value, and four hundred test methods that each insert a Contact somewhere in their setup all failed at once.

This is structural. Apex tests cannot see org data by default, which is correct, so every test builds the world it needs. Multiply that by a few hundred test methods and the org's schema is now hard-coded across your entire test suite. Every schema change is a schema migration of the tests.

The second half of the problem is the pipeline. A coverage gate at 75% is easy to configure and easy to satisfy without improving anything, which means a green build tells you far less than it appears to. The tooling that would tell you something, static analysis over the test code itself, is usually the piece nobody wired up.

Common questions this article answers:

• How do I stop a schema change breaking every test at once?
• Should I use a test data factory, a builder, or a library like Apex Test Kit?
• What should a Salesforce CI job actually run, beyond checking coverage?

Quick Answer

Centralise test data creation so a schema change is one edit rather than four hundred, and make the central helper return objects a test can modify rather than records it has already inserted. A TestDataFactory with static methods is the simplest version and is fine until tests start needing variations, at which point a builder (new AccountBuilder().withIndustry('Technology').build()) absorbs change much better because each test states only the fields it cares about and inherits sensible defaults for the rest. Apex Test Kit earns its place when you routinely need deep object graphs with hundreds of related records, and Test.loadData with a CSV in a static resource suits large reference data sets, with the caveat that a CSV has no compiler to tell you when it drifts from the schema. In the pipeline, run sf apex run test --test-level RunLocalTests --code-coverage --result-format junit --wait 60 so results land in a format your CI can read, group related tests into Apex test suites so a developer can run a relevant subset in seconds, and add Salesforce Code Analyzer with the PMD Apex ruleset so that assertion-free tests and SeeAllData=true fail the build rather than quietly inflating the coverage number.

Why the factory rots

The first version is always this, and it is the right first version:

@IsTest
public class TestDataFactory {

    public static Account createAccount() {
        Account a = new Account(Name = 'Test Account');
        insert a;
        return a;
    }
}

It rots for two reasons, and both are predictable.

It inserts. Once the method commits the record, a test that needs a variation cannot get one. So it queries the record back and updates it, which doubles the DML, or a second method appears called createAccountWithIndustry, then createAccountWithIndustryAndOwner, and within a year the factory has forty near-identical methods nobody can safely delete.

Its defaults are invisible. A test asserting on a total has no idea the factory set AnnualRevenue to 1,000,000, so when someone changes that default to satisfy a different test, this one fails for reasons the failure message cannot explain.

The minimal fix is to separate building from inserting, and to accept overrides:

@IsTest
public class TestDataFactory {

    // Build, do not insert. The caller decides when, and can change anything first.
    public static Account account(Map<SObjectField, Object> overrides) {
        Account a = new Account(
            Name = 'Northwind Trading',
            BillingCountry = 'New Zealand'
        );
        for (SObjectField field : overrides.keySet()) {
            a.put(field, overrides.get(field));
        }
        return a;
    }

    public static List<Account> accounts(Integer count, Map<SObjectField, Object> overrides) {
        List<Account> accounts = new List<Account>();
        for (Integer i = 0; i < count; i++) {
            Account a = account(overrides);
            a.Name = a.Name + ' ' + i;   // keep unique where the org requires it
            accounts.add(a);
        }
        return accounts;
    }
}

That one change kills most of the rot. A new required field is one line here, not four hundred lines across the suite, and a test needing a variation writes it inline instead of adding a method.

Builders, when variation is the norm

A builder is the same idea with a fluent surface and per-object defaults. It costs more to write and reads considerably better at the call site:

@IsTest
public class AccountBuilder {

    private Account record = new Account(
        Name = 'Northwind Trading',
        BillingCountry = 'New Zealand',
        Industry = 'Retail'
    );

    public AccountBuilder named(String name) {
        record.Name = name;
        return this;
    }

    public AccountBuilder inIndustry(String industry) {
        record.Industry = industry;
        return this;
    }

    public AccountBuilder ownedBy(Id userId) {
        record.OwnerId = userId;
        return this;
    }

    public Account build() {
        return record;
    }

    public Account insertRecord() {
        insert record;
        return record;
    }
}

At the call site:

Account customer = new AccountBuilder()
    .named('Southern Cross Logistics')
    .inIndustry('Transportation')
    .insertRecord();

The value is in what the test does not say. This test declares that the industry matters to it and the billing country does not. When someone changes the default country next year, this test is unaffected, and when a test breaks because of a default, you know that default was load bearing for it.

The cost is one builder class per object you build often, which is real but bounded. Most orgs need four or five.

Add a buildWithFakeId() method that stamps a synthetic Id without inserting. Combined with the Stub API, it lets whole classes of test run with no DML at all.

Apex Test Kit

Apex Test Kit is an open-source library for generating object graphs declaratively. Instead of writing loops, you describe the shape you want:

ATK.prepare(Account.SObjectType, 10)
    .field(Account.Name).index('Account-{0000}')
    .field(Account.AnnualRevenue).repeat(1000000)
    .withChildren(Contact.SObjectType, Contact.AccountId, 40)
        .field(Contact.LastName).index('Contact-{0000}')
        .field(Contact.Email).index('contact{0000}@example.com')
    .save();

Ten Accounts, forty Contacts distributed across them, unique names and emails generated automatically, relationships wired, one statement. The DSL moves between major versions, so check the repository for the current syntax rather than copying the snippet above verbatim.

What it is good at: deep hierarchies, uniqueness constraints, and the drudgery of populating required fields you do not care about. It can build in memory without committing, which keeps it useful alongside mocking. It solves the "sixty lines of setup before the first assertion" problem convincingly.

What it costs: another unmanaged open-source dependency deployed into your org, which you upgrade yourself and reconcile with your own naming. A DSL every new developer has to learn before they can read a test. And a layer of indirection between the test and the records, which means a failure in generation reads as a stack trace inside the library rather than a line you can point at.

Evaluate it if setup is dominating your test code. Skip it if a builder would do, because a builder is code your team already understands.

Test.loadData and static resources

The platform's own answer for bulk fixtures. Put a CSV in a static resource, then:

@TestSetup
static void makeData() {
    List<SObject> products = Test.loadData(Product2.SObjectType, 'TestProducts');
    Assert.areEqual(250, products.size(), 'Product fixture should load 250 rows');
}

Good for large reference data sets: a product catalogue, a pricing matrix, a list of regions. Non-developers can maintain the CSV. It keeps a 250-row fixture out of Apex entirely.

The trap is that a CSV has no compiler. Rename a field, tighten a picklist, add a validation rule, and nothing tells you the fixture is now wrong until every test using it fails with a DML exception pointing at the static resource. The assertion on products.size() above is the cheap defence: it turns a partial load into an immediate, readable failure.

Test.loadData inserts immediately and returns the records, so it does not fit the build-then-modify pattern. Use it for data that really is reference data and does not vary per test.

Comparison

	Static factory	Builder	Apex Test Kit	Test.loadData
Cost to start	Minutes	An hour per object	Deploy and learn a library	Minutes, plus a static resource
Absorbs schema change	Well, if defaults are central	Well	Very well	Poorly, no compile-time check
Handles variation	Poorly, method explosion	Very well	Well	Not at all
Deep relationships	Manual	Manual	Its main strength	Manual, one resource per object
Can build without DML	Yes, if it does not insert	Yes	Yes	No, it inserts
New developer can read it	Immediately	Immediately	After learning the DSL	Immediately
Best for	Small orgs, first version	Most teams, most objects	Deep graphs, heavy setup	Large static reference data

Running tests from the sf CLI

The command most CI jobs should be running:

sf apex run test \
  --test-level RunLocalTests \
  --code-coverage \
  --result-format junit \
  --output-dir ./test-results \
  --wait 60 \
  --target-org ci-sandbox

The flags that matter:

• --test-level RunLocalTests runs all Apex in the org except managed package tests. This is what you want in CI. RunSpecifiedTests narrows to named classes, RunAllTestsInOrg includes managed packages and is almost never what you want because you cannot fix their failures.
• --code-coverage returns per-class coverage. Without it you get pass or fail and no numbers.
• --result-format junit writes a format your CI can turn into a test report with per-test failures, rather than a wall of console text. json and tap are also available.
• --output-dir is where those results land, and is required for the report to survive the job.
• --wait 60 makes the command block until results are in. Without it the CLI returns immediately with a job Id and your pipeline marks the step green before a single test has run. This misconfiguration is common.

For local work, the useful variants are narrower:

# One class, synchronously, with detailed per-line coverage
sf apex run test --class-names InvoiceTotalCalculatorTest --synchronous --code-coverage --detailed-coverage

# One method
sf apex run test --tests InvoiceTotalCalculatorTest.totalExcludesCancelledLines --synchronous

# A named suite
sf apex run test --suite-names BillingRegression --synchronous

--synchronous runs in the foreground and returns results directly, so it suits iterating. It is limited to a single class or a small set, so it does not replace the CI invocation.

Apex test suites are underused. An ApexTestSuite is metadata listing test classes, deployable like anything else and therefore reviewable. Defining a BillingRegression suite means a developer touching billing runs thirty relevant tests in twenty seconds instead of the whole org in eleven minutes, which is the difference between running tests before pushing and not.

Deployment test levels

A separate setting from the CLI test run, applied when metadata is deployed. Four options:

Level	Runs	Use when
NoTestRun	Nothing	Sandbox deployments only. Not permitted to production.
RunSpecifiedTests	Only named classes	A targeted production hotfix, where you accept the risk in exchange for minutes. Every class in the deployment still needs 75% coverage from those named tests alone.
RunLocalTests	All non-managed Apex	The default for a production deployment, and the right answer nearly always.
RunAllTestsInOrg	Everything including managed packages	Rarely. Package tests fail for reasons you cannot fix.

The RunSpecifiedTests trap catches people during incidents: the coverage requirement is evaluated against only the tests you named, so a deployment that would pass comfortably under RunLocalTests gets rejected for insufficient coverage. If you plan to use it under pressure, check it works during a calm deployment first.

Static analysis catches what coverage cannot

A coverage gate cannot see assertions, which is the whole argument of the first article in this series. Static analysis can.

Salesforce Code Analyzer wraps PMD, ESLint and other engines behind one command. In version 5 the commands are under sf code-analyzer; older installations use sf scanner. Either way the useful move is the same: run it over your test classes, not only your production classes.

sf code-analyzer run --workspace ./force-app --view detail

The PMD Apex rules that enforce test quality specifically:

• ApexUnitTestClassShouldHaveAsserts flags a test method with no assertion at all. This is the highest-value rule on the list, because it is exactly the failure a coverage gate cannot see.
• ApexUnitTestShouldNotUseSeeAllDataTrue flags tests depending on org data.
• ApexUnitTestMethodShouldHaveIsTestAnnotation catches methods that look like tests but never run.
• ApexAssertionsShouldIncludeMessage requires the third argument on assertions, the difference between a readable failure at three in the morning and a guessing game.

A realistic adoption path is the same ratchet pattern used for any inherited lint debt: record today's violation count as a baseline, fail the build when the count increases, and reduce the baseline as classes get touched. That gets you a gate that bites immediately without a three-week cleanup before anything can merge.

When a test fails and the log is not enough

Two tools are worth trying before you resort to adding debug statements.

Apex Replay Debugger. Run the failing test with logging at FINEST, download the debug log, and step through it in VS Code with real breakpoints and variable inspection. It is a recorded replay rather than a live session, so you cannot change values, but for a test that fails in CI and not locally it is much faster than reading a 40,000 line log.

Check for parallelism before debugging logic. A test that fails intermittently, particularly with UNABLE_TO_LOCK_ROW, is usually contending with another test rather than wrong. Test classes marked @IsTest(isParallel=true) run concurrently, and two of them touching the same parent record will collide. Row lock contention behaves the same way in tests as in production, so the diagnosis is the same: find the shared record. Tests that touch setup objects such as users or permission sets cause most of these, and they are also the ones most likely to hit mixed DML restrictions.

Frequently Asked Questions

Q: Should my test data factory insert records or just build them?

A: Build, and let the caller insert. A factory that inserts forces every variation into a new method, which produces the forty-method factory nobody can maintain. Returning an uninserted sObject lets a test change whatever it needs and insert once, and it lets tests that are mocking their dependencies skip the insert entirely. Offering both, a build() and an insertRecord(), costs one extra method and covers every case.

Q: Is Apex Test Kit worth installing?

A: Only if setup code is the problem. Look at your three largest test classes and count the ratio of setup lines to assertion lines. If setup is two or three times the assertions and most of it is building related records, ATK will remove a lot of code and probably justifies the dependency. If your setup is short but repetitive, a builder gets you most of the benefit with no library to own. Do not install it because the sample code reads nicely; install it against a measured problem.

Q: Why does my CI job pass when tests are failing?

A: Almost always a missing --wait. Without it, sf apex run test enqueues an asynchronous run, prints a job Id, and exits zero. The pipeline sees a successful command and moves on while the tests are still queued. Add --wait 60 (or whatever exceeds your suite's runtime) so the command blocks and returns a non-zero exit code on failure. The second most common cause is a step that captures the result file but does not fail the job when the file contains failures.

Q: Can I enforce a higher coverage threshold than 75% in my pipeline?

A: Yes, and it is easy: parse the coverage from the --code-coverage --result-format json output and fail the job below your own number. Whether you should is a different question. Raising the threshold pushes people to write tests for the code that is easiest to cover, which is usually not the code that most needs testing. A threshold at 75% combined with ApexUnitTestClassShouldHaveAsserts and a review habit of asking what each test would catch is a stronger gate than 90% on its own.

Q: How do I speed up a suite that takes twenty minutes?

A: In order of return: mark classes @IsTest(isParallel=true) where they do not touch setup objects or shared records, which is usually the largest win. Move @TestSetup data creation out of individual methods. Replace DML-heavy tests of pure logic with mocked ones. Delete tests that assert nothing, since they cost runtime and return nothing. Then define test suites so developers rarely need the full run locally, and keep the full run in CI where its duration matters less.

Key Takeaways

• Data breaks suites, not logic: centralise construction so a schema change is one edit.
• Build, then insert: a factory that commits records forces variation into method explosion.
• Builders are the default answer: per-object defaults plus a fluent surface, with each test declaring only what matters to it.
• Apex Test Kit against a measured problem: deep graphs justify the dependency, repetitive one-liners do not.
• --wait or your pipeline is lying: without it the CLI returns before any test has run.
• PMD's assertion rules are the gate that bites: coverage cannot see an assertion, and ApexUnitTestClassShouldHaveAsserts can.

What's Next?

Recommended reading:

• Apex Test Quality: Why 75% Code Coverage Tells You Nothing, the first article in this series
• Mocking in Apex: Stub API, HttpCalloutMock and ApexMocks Compared
• Deploying metadata with the sf CLI
• UNABLE_TO_LOCK_ROW and what it really means

Action items:

1. Count setup lines against assertion lines in your three largest test classes. That ratio decides whether you need builders, a library, or neither.
2. Change your factory methods to return uninserted records, and fix the call sites. It is a mechanical change and it stops the method explosion permanently.
3. Check your CI command for --wait. If it is missing, your last green build proved nothing.
4. Add Code Analyzer with the PMD Apex ruleset, baseline today's violations, and fail the build when the count goes up.
5. Define one Apex test suite for your busiest area of the org and see whether people start running tests before they push.

Resources & References

• Apex Developer Guide: Create Test Data for Apex Tests
• Test.loadData reference
• sf apex run test command reference
• Deployment test levels in the Metadata API
• Salesforce Code Analyzer documentation
• PMD Apex rules reference
• Apex Test Kit on GitHub
• Apex Replay Debugger setup