Skip to content
Architecture Lab

Architecture Lab

Chaos engineering experiment platform

Running controlled failure experiments safely: hypothesis, guardrails, time-boxed fault injection, steady-state checks and automatic rollback.

  • Chaos engineering
  • Resiliency
  • Observability
  • Cloud

Illustrative reference design built from general engineering practice. It is not the architecture of any employer's or client's production system.

Why it matters

Chaos engineering finds out how an application behaves when its dependencies fail, on purpose and under control, before a real incident does. My work at JPMorgan Chase is in this area. This page is a general model of how such a platform can work, not JPMorgan Chase's architecture.

Diagram

Chaos experiment lifecycle with automatic abortAn experiment definition passes guardrails and approval, then a runner starts a time-boxed fault injector against the target application. The application's telemetry goes to observability, a steady-state check compares it with the hypothesis and either produces a report or, on a breach, triggers halt and rollback on the target.Control planeTarget environmentExperiment definitionHypothesis, blast radiusGuardrails & approvalScope, timing, ownersExperiment runnerSchedules and tracksFault injectorTime-boxed, self-revertingTarget applicationPrivate, public or hybridObservabilityMetrics, logs, tracesSteady-state checkCompare to hypothesisExperiment reportFindings, follow-upsHalt & roll backOn any breachinject faultbreachrevert
  • Request / call
  • Failure path
Illustrative
Diagram as text

Components

  • Experiment definition (input or output): Hypothesis, blast radius
  • Guardrails & approval (control point): Scope, timing, owners
  • Experiment runner (service): Schedules and tracks
  • Fault injector (service): Time-boxed, self-reverting
  • Target application (external system): Private, public or hybrid
  • Observability (data store): Metrics, logs, traces
  • Steady-state check (control point): Compare to hypothesis
  • Experiment report (input or output): Findings, follow-ups
  • Halt & roll back (control point): On any breach

Connections

  • Experiment definition → Guardrails & approval
  • Guardrails & approval → Experiment runner
  • Experiment runner → Fault injector
  • Fault injector → Target application (inject fault)
  • Target application → Observability
  • Observability → Steady-state check
  • Steady-state check → Experiment report
  • Steady-state check → Halt & roll back (failure path, breach)
  • Halt & roll back → Target application (failure path, revert)

How it flows

  1. State a hypothesis

    For example: "if one dependency becomes slow, the application degrades gracefully and recovers." Define steady state with signals the team already monitors.

  2. Limit the blast radius

    Choose the smallest scope that tests the hypothesis: one service, one instance, a short window.

  3. Pass guardrails

    Check the target is in scope, the timing is allowed and owners have approved; refuse to start without a healthy baseline.

  4. Inject

    Start a time-boxed fault, such as latency, an error rate or a stopped process, that reverts by itself when its time is up.

  5. Watch

    Compare live telemetry with the steady-state definition throughout the experiment.

  6. Stop or conclude

    On a breach, halt and roll back immediately; otherwise end on schedule and record what was learned.

Boundaries

Blast radius

Every experiment declares its scope up front, and the injector refuses targets outside it.

Control plane vs targets

The control plane decides and records; agents in each environment only carry out time-boxed instructions.

Environment boundaries

Private, public and hybrid cloud need different injection mechanisms behind one experiment definition.

Failure handling

Impact exceeds the hypothesis
The steady-state check triggers halt and rollback without waiting for a person.
Injector loses contact with the control plane
Faults expire on their own timer, so a lost connection can't leave a fault running.
Telemetry missing or delayed
No data counts as unsafe: the experiment won't start, or it stops.
Real incident during an experiment
A global stop halts all running experiments so they don't add noise to the incident.

Trade-offs

Self-reverting, time-boxed faults

Gain
Safe even when the platform itself fails.
Cost
Long-running scenarios need several chained experiments.

Automatic abort on steady-state breach

Gain
Limits harm and makes teams more willing to experiment.
Cost
Steady-state signals must be chosen carefully, or experiments stop on noise.

Start small, widen gradually

Gain
Builds confidence and keeps risk low.
Cost
Some system-wide weaknesses only show up at larger scopes, later.

Next design

Retrieval-Augmented Generation pipeline