23 — Failure modes and negative paths

Every component in this platform will fail, and tenants will send bad input, on purpose or by accident. This document is the catalogue of those cases: what fails, who detects it, what the tenant sees, what the platform does, and the test that proves the behaviour. It is the source for the "negative path checklist" issue that blocks every phase gate (22-development-plan.md, section 5).

Rules:

1. A feature is not done until its rows here have a passing test. A row without a test is listed as TODO with the issue that owns it.
2. The tenant-visible outcome is always one of the error codes in docs/contracts/errors.md; never a stack trace, a file path, a hostname of ours, or a secret.
3. A failure of one tenant must never change the outcome for another tenant. Every row in sections 1–4 is also a noisy-neighbour test.
4. Detection must be automatic. "An operator notices" is not a detection mechanism.

Columns: Failure · Detected by · Tenant sees · Platform does · Proof (test or benchmark) · Phase.

1. Tenant code misbehaves (runtime)

Failure	Detected by	Tenant sees	Platform does	Proof	Phase
Infinite loop / CPU over limit	CPU watchdog	503 RENOX_CPU_LIMIT, logged with route and cpu_ms	TerminateExecution, isolate poisoned and replaced, meter status=cpu_limit	renox-runtime limits test; B12	F1
Unbounded allocation	V8 near-heap-limit callback	503 RENOX_OOM	Terminate, dispose isolate on its own thread, RSS returns to baseline	limits test (RSS delta ≤ 10 MB after)	F1
Handler never resolves	Wall-clock deadline	504 RENOX_DEADLINE	Cancel pending ops via token, meter status=timeout	limits test	F1
Handler throws	Dispatch wrapper	500 RENOX_INTERNAL, request id	Error with stack to tenant log sink only	dispatch test (body has no message/stack)	F1
Route module fails to evaluate on first use	Module loader at the route's first request (ops.md 1.4.0 §11 steps 4–5: lazy route modules)	500 RENOX_MODULE_EVAL for that route only; other routes answer	Error logged once per isolate with the module id; the failure is cached until the isolate is replaced; the isolate stays up. A missing export is refused earlier, by renox check and at prepare (artifact.md V23)	crates/renox-runtime/tests/lazy_routes.rs (#265)	F2
Watchdog fires while a DB transaction is open	Runtime op registry	Same as CPU limit	Rollback every open transaction of that request before the isolate is dropped; writer connection released	crates/renox-runtime/tests/termination.rs: cpu_limit_inside_a_transaction_rolls_back_and_releases_the_writer, oom_inside_a_transaction_rolls_back_and_releases_the_writer, deadline_inside_a_transaction_rolls_back_and_releases_the_writer, pending_fetch_is_aborted_when_the_request_is_terminated (#89)	F1
Too many concurrent requests	Semaphore per deployment	503 RENOX_BUSY + Retry-After	Queue up to the deadline, then reject	limits test	F1
Handler writes to global scope	renox check lint, dev warning	Lint error / dev overlay warning	Nothing at runtime (documented risk)	lint fixture	F2
Template nested loops / huge output	Render fuel, output size limit	500 RENOX_RENDER_LIMIT with template name and line	Render aborted; render time charged to tenant	B13; renderer limit tests	F2
fetch to blocked destination	net_policy	RenoxError RENOX_FETCH_BLOCKED {host, reason} thrown in tenant code	Request never leaves the node; counter in meter	fetch tests (metadata, loopback, private, redirect hop, v4-mapped v6)	F1
fetch response too large / too slow	Egress counter, deadline	RENOX_FETCH_EGRESS_LIMIT / RENOX_DEADLINE	Abort stream	fetch tests	F1
Secret printed or thrown	Log redaction	[redacted] in own logs	Redaction on every log and error path	ops test	F1
SQL touching another module's table	SQLite authorizer at prepare	RENOX_DB_AUTH {table}	Statement refused; no partial execution	authorizer tests (JOIN, CTE, ATTACH, PRAGMA, trigger)	F1
Runaway query	progress_handler	RENOX_DB_TIMEOUT	Statement interrupted, transaction rolled back	limits test (WITH RECURSIVE)	F1
Database file over quota	max_page_count	RENOX_DB_QUOTA	Writes fail, reads continue; dashboard warning	quota test	F1
Unparameterised SQL	renox check lint	Lint error	Rejected before deploy	lint fixture	F1

2. Bad input at the edge (gateway)

Failure	Detected by	Tenant sees	Platform does	Proof	Phase
Unknown Host	Route table miss	— (end user gets 404)	No cell touched	gateway test	F1
Malformed Host, header too large, too many headers	hyper limits (configured)	400 / 431	Connection closed, no cell touched	renox-gateway tests/limits.rs: header_over_16kb_is_431, two_hundred_headers_is_431; tests/gateway.rs: malformed_or_missing_host_is_400	F1
Request body over limit	Body limit per route kind (default 1 MB, uploads via storage module)	413 RENOX_BODY_TOO_LARGE	Stream aborted before reaching the isolate	renox-gateway tests/limits.rs: two_mb_body_is_413_and_never_dispatched, chunked_body_is_cut_at_the_limit	F1
Stored file over the per-file limit	op_storage_put against the tier's limit (ops.md 1.7.0 section 21)	StorageError RENOX_STORAGE_TOO_LARGE	Nothing stored	crates/renox-cell/tests/storage.rs: quota_size_type_svg_and_key_refusals (#321)	F7
Stored file of a disallowed or disguised type	Sniffing of the first 512 bytes against the declared type and [storage] allowed_types	RENOX_STORAGE_TYPE (details.reason: not_allowed, mismatch)	Nothing stored	same test (#321)	F7
SVG with script uploaded	image/svg+xml only with sanitised: true (the storage module's sanitiser); SVG bytes under another type are a mismatch; /_files serves SVG with Content-Security-Policy: sandbox	RENOX_STORAGE_TYPE svg_unsanitised / mismatch	Nothing stored	same test (#321)	F7
Tenant storage quota reached	Sum of renox__files before the put	RENOX_STORAGE_QUOTA	Nothing stored; reads continue	same test (#321)	F7
Storage: an owner over its quota, a hostile SVG, another owner's folder, a file escaping its folder	The storage module before op_storage_put (config.quota_mb per owner, the sanitiser, storage.manage for another folder, keyOf)	UploadError QUOTA / SVG / INVALID; 403 for another owner's folder	Nothing stored; the owner's other files and the tenant's quota are unchanged	modules/storage/tests/storage.test.ts (folders, the sanitiser), modules/storage/tests/hostile-svg.test.ts (34 hostile SVGs, each cleaned or refused), modules/storage/tests/app.test.ts (the quota, the folders per role, <x-image> on a stored file) (#332)	F7
Expired or altered signed file URL	HMAC of /_files/<key>?exp= with the tenant's storage key, exp checked	404 RENOX_STORAGE_NOT_FOUND (a private file is never distinguishable from a missing one)	Nothing served	crates/renox-cell/tests/storage.rs: a_signed_url_serves_a_private_file_until_it_expires_and_only_unaltered; tests/images.rs: a_private_original_needs_a_signature_on_img_too (#321)	F7
Image transform flood on /_img	Per-tenant transform rate; only cache misses count	429 RENOX_IMAGE_RATE with Retry-After; cached variants keep serving	No transform past the burst	crates/renox-cell/tests/images.rs: the_transform_rate_refuses_past_its_burst_and_cached_variants_still_serve (#321)	F7
Webhooks: bad signature, replayed id, provider rate limit	op_webhook_verify over the raw body (HMAC per provider, Stripe's 300 s tolerance), renox__webhook_events, a per-provider bucket of unverified requests	401 RENOX_WEBHOOK_SIGNATURE; a replay answers 200 {"duplicate":true}; past the bucket 429	The handler never goes on for a bad or repeated delivery; a verified one is never dropped	crates/renox-runtime/tests/webhooks.rs (#322); a module's webhook route on the worker, a bad signature 401 and a replay answered duplicate: modules/billing/tests/app.test.ts (#329)	F7
Payment failure paths: the PSP refuses or is unreachable, a payment fails, a double checkout, a refund over the paid amount	The billing module (modules/billing/lib/core.ts, lib/adapter.ts)	BillingError PSP, CONFLICT, INVALID; nothing is recorded for a refused PSP call	A failed payment moves the subscription to past_due and emits billing.payment_failed; the same order answers its open session (no second charge); a paid order is a conflict; a refund never exceeds paid less refunded	modules/billing/tests/billing.test.ts "one checkout per order…", "refunds…", "webhook paths…" (#329)	F7
Tax: a seller without a TRN, a buyer's bad TRN, a credit note over its invoice, a billing invoice delivered twice	modules/tax_gcc before a number is taken	TaxError NOT_CONFIGURED / INVALID; the event handler fails and the outbox retries it once the TRN is set	No document and no gap in the numbering; one tax invoice per billing invoice (tax_gcc__links)	modules/tax_gcc/tests/tax.test.ts (the refusals, "the numbering series under concurrency", "billing.invoice_issued stamps one…"), tests/app.test.ts (#331)	F7
Booking: double booking, missed reminder	The slot search, then the unique index on a live booking per staff member and start (modules/booking); reminders as queued jobs at 24 h and 2 h before, sent once (booking__reminders) and only while the booking is confirmed	The second customer is answered "taken"; a reminder of a cancelled booking is not sent	One booking per slot under concurrency; a job that fails is retried by the outbox and then a dead letter the jobs page shows	modules/booking/tests/booking.test.ts ("one live booking per staff member and start…", "a deposit…reminders, each sent once", "cancelling…a missed reminder after a cancel is not sent"), tests/app.test.ts ("two customers racing for the same slot", "the reminders are scheduled") (#334)	F7
Slowloris / idle connections	Header read timeout, idle timeout, max connections per IP	—	Connection dropped	renox-gateway tests/limits.rs: one_byte_per_second_is_closed_at_the_header_timeout, idle_connection_is_closed_at_the_idle_timeout, connection_101_from_one_ip_is_rejected	F1
Request flood from one IP	Token bucket	429 + Retry-After	Not metered to tenant	gateway test	F1
Flood from many IPs on one tenant	Per-tenant concurrency and tier CPU quota	503 RENOX_BUSY	Other tenants unaffected (B12)	B12	F1
Abuse: CPU and egress quota breach	The hourly billing pass (control/api/src/billing/sweep.ts): the plan quota per period, and an hour over 10 times the tenant's own 7-day hourly median on requests, CPU-ms or egress	none to the visitor; getBilling signal.over_quota and signal.anomaly	Free: over_quota → grace after 24 h → suspended after 7 days, the owner emailed at each step; an anomaly writes an operator review row (billing_reviews) and counts as over quota for a free tenant; paid: reviewed only, overage is the plan's	control/api/tests/billing.test.ts "the hourly pass drives a free tenant over quota…" and "the anomaly rule…"; control/billing/tests/tier.test.ts; once suspended, the hostname answers 503 RENOX_TENANT_SUSPENDED with the neutral page: crates/renox-gateway/tests/suspended.rs (#325)	F7
Email to an address that bounced or complained	The provider's notification through the module's feedback route (modules/email, webhook.verify)	send refuses with SUPPRESSED; nothing reaches the provider	The address is in email__suppressions and email.bounced / email.complained is emitted; a message queued before it ends suppressed	modules/email/tests/email.test.ts "a bounce suppresses…" and "auth.send_mail fills…" (#328)	F7
Flood on login/register/reset	Stricter route buckets	429	—	security tests	F2
CSRF token missing or wrong	Action dispatcher	403 RENOX_CSRF	Action not run	pages test	F2
Cross-origin request without CORS allow	Gateway CORS	403 / no CORS headers	—	security tests	F2
Unverified custom domain	Domain state	404	Never served before TXT verification	domains test	F3
Browser error (script, island, Alpine) or htmx failure (error response, no target, swap error, timeout)	renox-client.js events with request id	Dev: overlay "Browser" tab; prod: dashboard group with original file and line	Event rate-limited and logged with source=client; nothing touches the isolate	#127 Playwright suite; #128 grouping tests	F2/F3
Script blocked by CSP	CSP report-to → /__renox/events	Dev overlay / dashboard csp.violation with directive and source	Same as above	#126 normalisation test	F2
Client-event flood (bug or abuse)	Per-IP and per-tenant buckets at the gateway	Nothing (204 always)	Dropped and counted; daily quota warning in the dashboard	#126 limit tests; #128 quota test	F2/F3
WebSocket message flood / oversized frame	Per-connection limits, backpressure	Connection closed with code	Fan-out never touches the isolate	realtime tests, B14 (docs/contracts/ops.md §18, "Limits"): crates/renox-realtime/tests/hub.rs (#278); end to end crates/renox-cell/tests/realtime.rs an_oversized_frame_closes_with_1009, a_publish_from_an_action_reaches_two_connections_without_the_isolate (#281); the change feed publishes on commit and never on rollback, crates/renox-data/tests/feed.rs a_rolled_back_transaction_delivers_nothing_and_leaves_nothing_for_the_next_commit and crates/renox-cell/tests/realtime.rs the_change_feed_publishes_committed_rows_and_never_a_rollback (#305); a dropped client link costs one reconnect and nothing inside the replay buffer (SSE Last-Event-ID), chaos scenario pubsub-drop (bench/chaos, #312)	F5

3. Deploy and artifact failures (CLI, control plane, worker)

Failure	Detected by	Tenant sees	Platform does	Proof	Phase
Syntax error, missing renox.toml, bad route file name	renox build / renox check	CLI error with file and line, non-zero exit	Nothing uploaded	cli/tests/negative/messages.test.ts "syntax error in a route", "missing renox.toml", "invalid TOML", "uppercase parameter"; cli/tests/negative/dev.test.ts (#90)	F2
Watcher thrash (many saves per second)	renox dev debounce	One rebuild per burst	Isolate recreated once	same issue	F2
Artifact hash mismatch, symlink, .. path	Worker artifact loader	Deploy rejected	Artifact refused, never loaded	artifact tests	F1
Bundle over 20 MB, route conflict, unknown permission key	CLI check and control-plane validation (same code)	Structured validation error	Deployment state rejected	validation tests	F3
Tarball decompression bomb / too many files	Validation with size and entry caps before extraction	Rejected	Extraction aborted at cap	control/api/tests/extract.test.ts "T1–T6 …", "the suite covers every rule a body can break while it is read (T1-T6)" (#95)	F3
Upload interrupted, retried	Content hash idempotency	Same hash, same deployment	No duplicate	deployments test	F3
Migration fails half-way during activation	Migration runner (transaction per file)	Deployment stays validated with the failing file and SQL error; old deployment keeps serving	No route switch; renox__migrations unchanged for the failed file	crates/renox-cell/src/control/prepare.rs a_failing_second_file_is_reported_and_the_first_stays_applied; crates/renox-cell/tests/activation.rs activation_prepares_on_the_node_and_switches_only_on_success; control/api/tests/activation.test.ts "a failing migration file: …" (#96)	F3
New deployment fails its first request	Activation health probe	Activation reported failed; old stays active	Automatic pointer revert	same issue	F3
Rollback to a hash whose migrations are older than the schema	Expand/contract rule (D17)	Rollback allowed; code must tolerate newer columns	Documented contract; renox check --target <hash> warns	same issue	F3
Route update message lost or out of order	Version number per route entry; periodic full resync	Nothing (converges)	Worker applies only newer versions, resyncs every N s	crates/renox-cell/src/control/routes.rs lossy_reordered_deltas_converge_after_one_resync, an_older_delta_is_ignored (#97)	F3
Control plane unreachable from the worker	Heartbeat failure	Nothing for existing cells	Worker keeps serving from the local route table and artifact cache; meters buffered to disk; new cold starts for uncached artifacts fail with 503 RENOX_UNAVAILABLE	crates/renox-cell/tests/degraded.rs control_plane_down_keeps_serving_and_spools_meters, object_storage_down_refuses_only_uncached_cold_starts (#98)	F3
Object storage unreachable	Artifact fetch error; alert RenoxObjectStorageUnreachable (ops/observability/rules/node.yml) on renox_worker_degraded{component} (#295)	Cold start fails 503 for uncached artifacts only	Retry with backoff; alert	same issue	F3

4. Data and node failures

Failure	Detected by	Tenant sees	Platform does	Proof	Phase
Isolate evicted while a request arrives	Cell state machine; renox_worker_cold_start_duration_seconds, alert RenoxColdStartP95 (#295)	Nothing (slightly slower)	Request waits for the fresh cell	cell test	F1
Deployment switch while requests in flight	Draining state	Nothing	Old cell finishes in-flight, then evicted	cell test	F1
Worker process crashes (a paid tenant's child process, #254)	The supervisor reaps the child (exit status)	503 RENOX_UNAVAILABLE (Retry-After: 1) for requests in flight in that child; the next request waits for the restart. Other tenants see nothing: the shared process and other children are separate processes	Restart on the next request after an exponential backoff (200 ms doubling to 30 s, reset after 60 s alive); logged with the exit status	crates/renox-cell/tests/supervisor.rs own_tenant_runs_in_a_child_that_restarts_refuses_others_and_drains (SIGKILL, served again within the backoff)	F6
SQLite file corrupted, WAL unreadable	PRAGMA quick_check before open when replication is on (#255); SQLite error codes; alert RenoxTenantFileRestoring (ops/observability/rules/tenant.yml)	503 RENOX_DB_UNAVAILABLE until restored	The file is moved aside as app.db.corrupt-<ts> (never deleted), restored from Litestream and checked again before it is opened; alert	crates/renox-cell/tests/replication.rs writes_replicate_and_a_missing_or_corrupt_file_is_restored (corrupt and deleted files); integrity sweeps of idle files → #103; mid-request (#311): crates/renox-cell/tests/integrity.rs a_file_damaged_mid_request_answers_503_then_is_restored (a statement's SQLITE_NOTADB drains the cells, quarantines and restores, 503 RENOX_DB_UNAVAILABLE meanwhile)	F6
Disk full on node	Write errors, disk watermark; alerts RenoxDiskWatermarkWarn and RenoxDiskWatermarkStop (ops/observability/rules/node.yml) on renox_worker_disk_level (#295); the heartbeat's disk (deploy API 0.4) stored by the control plane, getNode, renox_cp_node_disk_level and the page RenoxNodeDiskCritical (#311: control/api/tests/workers.test.ts "stored as sent and answered by getNode…", "the control plane's renox_cp_node_disk_level pages on critical"; crates/renox-cell/src/control/mod.rs the_heartbeat_disk_names_the_level_and_the_watermark)	Writes fail RENOX_DB_QUOTA-like code	Node marked not schedulable; outbox and log pruning; alert	same issue; the data layer refuses tenant writes at 95 % too, for writes that never pass the dispatcher (jobs, event handlers): crates/renox-data/tests/limits_watermark.rs (#311)	F6
Outbox or logs grow unbounded	Size watermarks; renox_worker_outbox_pending and renox_worker_outbox_oldest_age_seconds, alert RenoxOutboxOldestAge (#295)	Nothing	Pruning, dead-letter cap, per-tenant log volume limit	same issue	F4/F6
Node dies	Control-plane heartbeat (unhealthy after 30 s, the sweep of #53 every 5 s; FO1)	Downtime ≤ RTO of the tier	Each tenant is restored on the next healthy node in its rendezvous order before its route entries move there (FO2-FO3, deploy-api.md section 16), pinned there, and a returning node is told to drop it (FO4); one source: platform log line and one audit row per move (FO5)	control/api/tests/failover.test.ts (restore before route change, asserted through the fake node's call log; drop on return); B18 bench/scenarios/B18/	F6
Node dies, restore fails	Every candidate's nodeRestoreTenant answers an error or times out (20 s each); alert RenoxFailoverRestoreFailed (ops/observability/rules/replication.yml)	Downtime until an operator acts	The tenant stays on the dead node (its route entries are not moved to a node without its data), an alert row (failover_restore_failed) and a platform log line are written	control/api/tests/failover.test.ts (first candidate fails → second; every candidate fails → stays with an alert)	F6
Tenant erasure cannot finish (owning node down, object store unreachable)	tenant_erasures row stays erasing with the step that failed; past 24 h failed and an erasure_failed alert row (#308)	getTenant shows erasure.state erasing or failed; the tenant's hostnames are no longer served from the first step	Each step retried hourly (sweepErasures); routes and tokens go first; a node that comes back is dropped before it is healthy (FO4); an operator sets a failed erasure back to erasing	control/api/tests/erasure.test.ts "a node that is down keeps the erasure in erasing; past the 24 h time box it is failed at node_drop with an alert; the node's return lets an operator retry to done"; the replica purge: crates/renox-cell/tests/replication.rs node_listener_restores_before_serving (purge_replica=true), src/replication/purge.rs tests; chaos bench/chaos/scenarios/erasure.ts (2026-10-30 result)	F6
Replica lagging / Litestream behind	replication_status.lag_seconds (the file's last write against the newest replica generation's end) on the node listener and in getBackups; the sidecar restarted with a backoff when it exits; alert RenoxReplicationLag (ops/observability/rules/replication.yml)	Nothing until failover	Alert above 10 s (FO6); failover restores the newest generation	crates/renox-cell/tests/replication.rs (status with generations, last sync and lag; sidecar stop and restart); B19 bench/scenarios/B19/; B18	F6
Meter batch cannot be shipped	Sink retry exhausted	Nothing	Dead-letter file on disk, replayed later; billing never silently loses data	meter tests; ingestion dedup test	F1/F3
Clock skew between gateway and worker	Single deadline set at the gateway, propagated as remaining ms	Nothing	Deadlines are durations, not timestamps	design rule (03-architecture.md)	F1

5. Modules and authorization

Failure	Detected by	Tenant sees	Platform does	Proof	Phase
Module service throws	Broker	RenoxError in caller with module and service name	Transaction rolled back if inside one	broker tests	F4
Event handler fails repeatedly	Outbox retries; renox_worker_dead_letters_total, alert RenoxDeadLetters (#295)	Dead-letter visible in jobs module	N retries with backoff, then dead-letter	outbox tests; jobs: modules/jobs/tests/jobs.test.ts "the page lists the scheduled jobs and the dead run, never its payload", "an admin retries a dead run and cancels it once queued" (#297)	F4
Job overlaps or is missed	renox_worker_job_runs_total, renox_worker_job_failures_total, renox_worker_job_lag_seconds (#297)	The job runs late, once	One durable row per tick; a lease per job, so a tick during a run waits for it; missed ticks coalesce into one run for the latest; the manager wakes an evicted tenant at its next tick; a row left running by a dead process is queued again at start	crates/renox-cell/tests/jobs.rs (missed_ticks_after_downtime_coalesce_into_one_run_as_the_system_principal, a_second_tick_during_a_run_waits_for_it, a_row_left_running_is_requeued_at_start)	F6
Module contract broken on upgrade (missing service, view version)	renox check	Upgrade refused with the broken reference	No migration run	check tests	F4
Template override drifted from module version	Stored original hash	Warning with diff	Upgrade continues, override flagged	override tests	F4
Policy denies	Cedar	403 RENOX_FORBIDDEN with policy id in dev	Audit row for sensitive actions	policy tests per role (docs/contracts/ops.md §17, "authz ops"): crates/renox-runtime/tests/authz_ops.rs require_and_the_audit_row, forbid_beats_permit, explain_is_dev_only; crates/renox-cell/tests/authz.rs guard_denies_before_the_loader_and_on_fragments (#279)	F5
Unscoped query on an owned table	db.scoped enforcement for module code	RENOX_DB_SCOPE	Statement refused	scoped tests (docs/contracts/ops.md §17, "db.scoped()"): crates/renox-data/tests/scoped.rs (#277); crates/renox-runtime/tests/authz_ops.rs db_scoped_for_module_code (#279)	F5
App MCP: tool call without the Cedar action	Cedar check as the token's user before run	JSON-RPC error 403 RENOX_FORBIDDEN	run never starts, no tool record; an allowed call is one TOOL <name> record	crates/renox-cell/tests/mcp.rs a_call_is_authorized_as_the_user (#285)	F5
2FA: refund denied before mfa_verified, TOTP replay within the window refused, recovery code single-use	auth.after_login halts the sign-in until the second factor; mfa_verified on the principal only while the verification is within step_up_minutes; TOTP last_step; recovery codes marked used	The refund policy (when { principal.mfa_verified }) denies with RENOX_FORBIDDEN; the verify page answers the generic error; five failures end the pending sign-in	No session before the second factor; a removed factor needs a recent verification	modules/2fa/tests/twofa.test.ts "enrol TOTP…", "sign-in: halted after the password, a replayed code refused, the next code resumes it", "a recovery code works once", "step-up: the verification expires…", "verify is rate limited…" (#287)	F5
Session fixation, replayed token, OAuth state mismatch, password reset token reuse	auth module	Login refused	Session rotated on login; tokens single-use; state checked	modules/auth/tests/negative/session-fixation.test.ts, token-replay.test.ts ("reset: reuse refused, expiry refused"), oauth-state-pkce.test.ts ("state: missing, foreign or forged is refused") (#100)	F4
Marketplace module unsigned or permission diff on upgrade	Registry verification	Install refused / approval required	Hash can be disabled globally	control/api/tests/registry.test.ts (a tampered tree and an unknown or revoked key refused, the upgrade diff, an approval that is not the version's request refused, disable by hash marks installs and emails owners); control/dashboard/tests/modules.test.ts (the permission review before an install, only the reviewed tree approved, the upgrade review, the disabled banner) (#324); a module running isolate cannot widen what was approved: crates/renox-cell/tests/isolate_mode.rs (#320)	F7

6. Verification practice

• Every phase has a negative path checklist issue listing the rows of this document due in that phase; the gate review is blocked by it: F1 #92, F2 #94, F3 #99, F4 #101, F5 #102, F6 #105, F7 #106.
• From F6 a fault-injection harness (bench/chaos/, #104) runs the node, storage, and control-plane failures above on a schedule against the examples, and B18 is one of its scenarios.
• Error codes shown to tenants are catalogued in docs/contracts/errors.md (#91) with: HTTP status, when it happens, what the tenant should do, what is logged.
• Security threats (attacks rather than failures) are catalogued separately in docs/24-threat-model.md, with their own per-phase security checklist issue blocking the same gate reviews.