DeepSeek Harness is a large pnpm monorepo (49 groups, 219 package-tier packages) that drives real LLMs, subprocesses, filesystems, and a browser GUI. Testing it well is a first-class engineering concern: the repository organises verification into several tiers, each with its own Vitest configuration, test location convention, and environment rules. This page maps the full matrix — from per-file 100% unit coverage to keyless snapshot replay and real-API e2e smokes — and dissects the shared test-support packages and the DSH_SNAPSHOT record/refresh workflow.

The test tiers at a glance

Tier	Command (package.json)	Config	Tests live in	Key required?
Unit	pnpm run test	vitest.config.ts	packages/*/*/tests/**/*.spec.ts, apps/*/tests, examples/*/tests, scripts/**/*.spec.ts	no
Coverage gate	pnpm run test:coverage	vitest.config.ts (+coverage)	same as unit	no
Real-API e2e	pnpm run test:e2e	vitest.e2e.config.ts	**/*.e2e.ts	yes (self-skips without)
Keyless snapshot	pnpm run test:snapshot	vitest.snapshot.config.ts	**/*.snapshot.ts	no
Web browser snapshot	pnpm run test:web	vitest.web.config.ts	apps/web/tests/*.{e2e,snapshot}.ts	no (replay)
Web perf / stress	test:web:perf, test:web:stress	vitest.web.{perf,web-stress}.config.ts	apps/web/tests/*.perf.ts, apps/web/stress-tests/*.stress.ts	no

The root package.json scripts block is the authoritative inventory. pnpm run test runs plain vitest run; each tier is a separate config invoked with --config. This is deliberately not one big runner — real-API tests need keys and long timeouts, snapshot tests must never accidentally write goldens in CI, and browser tests boot a whole Chromium instance.

The test-support packages

The shared machinery lives in packages/test-support/. Each package also ships a ./invariant companion (an npm subpath export to lib/invariant.js) so its package ownership is registered in the invariant registry — test-support packages typically carry an empty installer with a No runtime invariant: comment.

Package	Role
@deepseek-ai/dsh-agent-loop-testkit	Mounts the prerequisite services before an agent-loop test
@deepseek-ai/dsh-llm-mock-server	Scriptable OpenAI-compatible HTTP/SSE fault server
@deepseek-ai/dsh-llm-replay	Replay plugin: reconstructs model chunks from recorded session JSONL
@deepseek-ai/dsh-client-test-runtime	jsdom "slot" runtime: real Cordis Context + SlotRegistry + web-react renderer
@deepseek-ai/dsh-loader-smoke	Subprocess and direct-agent harness for keyless example smokes
@deepseek-ai/dsh-acp-snapshot	ACP snapshot suite factory, subprocess launcher, normalizers

agent-loop-testkit

The testkit does not mount the agent loop itself — it only mounts the services the concrete loop needs, so the test owns load order and topology:

// packages/test-support/agent-loop-testkit/src/index.ts
import AgentRegistry from '@deepseek-ai/dsh-agent'
import LlmRuntime from '@deepseek-ai/dsh-llm'
import SessionStore from '@deepseek-ai/dsh-session'
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
import ToolRuntime from '@deepseek-ai/dsh-tools'

export async function mountAgentLoopTestDependencies(
  ctx: Context,
  options: AgentLoopTestDependenciesOptions = {},
): Promise<void> {
  await ctx.plugin(LlmRuntime)
  await ctx.plugin(SessionStore)
  await ctx.plugin(SystemPrompt, options.systemPrompt ?? {})
  await ctx.plugin(ToolRuntime, options.tools ?? {})
  await ctx.plugin(AgentRegistry)
}

llm-mock-server

@deepseek-ai/dsh-llm-mock-server is a scriptable fault server for LLM recovery tests. Because it is an HTTP/SSE server, it exercises the real transport and hardens against real-world failures — connection_reset, stream_disconnect, partial_eof, stall, malformed_event, rate_limit, auth_error, context_overflow, quota_exceeded and more:

// packages/test-support/llm-mock-server/src/index.ts
export const MOCK_LLM_BEHAVIORS = [
  'connection_reset', 'stream_disconnect', 'empty', 'empty_body',
  'stream_eof', 'partial_eof', 'partial_disconnect', 'stall',
  'malformed_json', 'malformed_event', 'wrong_content_type', 'rate_limit',
  'server_error', 'service_unavailable', 'auth_error', 'invalid_request',
  'context_overflow', 'quota_exceeded', 'success', 'reasoning_success',
  'tool_call_success', 'max_tokens', 'slow_success', 'random',
] as const

llm-replay

@deepseek-ai/dsh-llm-replay is the keyless heart of snapshot testing. It reads a recorded session JSONL, derives one model-call script per session from assistant/chunk events, and binds fresh live sessions to those scripts by first-call order — so tests boot the real agent loop against replayed model output:

export type ReplayEntry =
  | { kind: 'chunks'; chunks: StreamChunk[] }
  | { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string }
  | { kind: 'hang'; readyFile?: string }

acp-snapshot

@deepseek-ai/dsh-acp-snapshot builds keyless ACP scenarios. Each scenario drives the real subprocess (booting the acp-agent example) and compares normalized stdout; session fixtures double as both replay input and expected output.

client-runtime & loader-smoke

• @deepseek-ai/dsh-client-test-runtime provides a jsdom "slot" runtime — a real Cordis Context + SlotRegistry + the web-react renderer — with test-owned session/workspace doubles, so client feature specs run without a browser process.
• @deepseek-ai/dsh-loader-smoke is the shared subprocess/direct-agent harness for keyless real-Loader example smokes. Every example ships a keyless smoke that boots the real cordis.yml through the Loader, drives it, and asserts output and clean exit.

The DSH_SNAPSHOT workflow

The snapshot config selects its mode from the DSH_SNAPSHOT environment variable. package.json:

"test:snapshot":           "vitest run --config vitest.snapshot.config.ts",
"test:snapshot:record":    "DSH_SNAPSHOT=record vitest run --config vitest.snapshot.config.ts --update",
"test:snapshot:refresh":   "DSH_SNAPSHOT=refresh vitest run --config vitest.snapshot.config.ts"

The three modes (from vitest.snapshot.config.ts):

Mode	Effect	Parallel?
replay (default)	Boots real subprocess paths from recorded responses; diffs assembled requests, normalized output, and persisted-log goldens. Never writes committed outputs.	parallel
record	Calls the real API (loads .env for a key) and updates fixtures + expected outputs.	serial
refresh	Replays committed scripts and rewrites derived expected outputs only (no key).	serial

Use record when a model transcript changes, refresh when replay input is still valid but expected output shifted. Review every JSONL and expected-output diff. refresh deliberately stays serial because concurrent write-back would corrupt the goldens it harvests from on-disk fixtures (--update implies a write). CI forces DSH_SNAPSHOT=replay (read-only) for the web lane so a commit can never silently alter expected output.

Source-plane resolution

Every Vitest config points vite-tsconfig-paths at tsconfig.base.json (no include → match-all). Bare workspace imports therefore resolve to src, never through package exports to built lib/ — a stale artifact there would load a second copy of module singletons. Built artifacts are consumed explicitly only in lib-mode subprocesses and the built-artifact smokes (packages/examples/*/tests/built-bin.e2e.ts).

A decorator pre-transform (standardDecoratorPlugin in vitest.shared.ts) compiles standard TypeScript decorators before Vite parses, and vitestExecArgv passes --no-webstorage so process-wide Web Storage cannot shadow jsdom storage.

Coverage: per-file 100%

vitest.config.ts sets thresholds: { perFile: true, statements: 100, branches: 100, functions: 100, lines: 100 } over packages/*/*/src/**. An uncovered line is usually dead code the gate is correctly flagging for deletion — not a missing test. The coverage exclude list exempts types-only files, self-booting bin.ts/worker.ts entries, and a documented set of client/UI files under a TODO(gui) debt label. pwsh-local files are only exempted when no real pwsh is present (probed by spawnSync(resolvePwshPath(), …)).

Real-API e2e and the with-key policy

The repo's stated philosophy in docs/testing.md: "We are DeepSeek — do not ration real-API tests." A *Key*-less test proves plumbing; only a with-key run proves the agent works against a real model. Highest-value tests are smokes that boot the real example, send one prompt, and check the world (external state), not the model's claim. Each e2e suite uses DEEPSEEK_API_KEY or provider-specific keys (EXA_API_KEY, PERPLEXITY_API_KEY, …) and self-skips without them, so keyless CI and keyless contributors stay green. vitest.e2e.config.ts sets testTimeout: 120_000, retry: 2, and a bounded worker pool (DSH_E2E_MAX_WORKERS, default 4).

The test-invariants setup and HMR safety

The main configs register setupFiles: ['./scripts/test-invariants.ts'] (the three web/browser configs do not) and every registry test gets an HMR-safety assertion (dispose the contributing fiber, assert cleanup). The invariant philosophy is covered in depth on the runtime invariants page.

Further reading

• Runtime invariants — the assertion philosophy your tests rely on.
• CI & release — how the tiers become a green PR gate (check:ci:*).
• Official docs/testing.md — the tier policy and with-key/real-entry-path rules, all in the repo.
• Official docs/defensive-patterns.md — the bug-class rules the tests must pin.
• packages/test-support/README.md — the umbrella over all six test-support packages.
• scripts/run-gates.ts — the check:ci:* scripts that orchestrate tiers in CI.