Skip to content
All projects

Chaos & Resiliency Engineering

Enterprise Resiliency Engineering

Engineering solutions for testing application resilience across private, public, and hybrid cloud environments.

Type
Professional
Company
JPMorgan Chase
Period
2022 – Present
Availability
Internal enterprise work. No source code or internal details are shared.

Sanitized, publicly shareable description. It contains no confidential information, and the diagram is an illustrative model of chaos engineering in general, not JPMorgan Chase's production architecture.

Overview

At JPMorgan Chase I work in resiliency engineering, building a chaos engineering framework that tests how applications behave when the infrastructure they depend on fails, across private, public and hybrid cloud environments.

Problem

Applications in a large enterprise depend on many services and run across several kinds of cloud environment. Teams need a safe, repeatable way to find out how those applications respond to failure before a real incident does it for them.

Solution

A chaos engineering framework with backend services and REST APIs for defining and running resilience experiments against applications, built as microservices, with automation and observability so the impact of each experiment can be seen.

What exists today

  • Chaos experiments across private, public and hybrid cloud

    Implemented

  • Backend services and REST APIs (Python, Django, DRF)

    Implemented

  • Workflow automation with Linux and shell scripting

    Implemented

Architecture

  1. Users

    • Application teams
    • CI/CD pipelines

  2. Platform API

    • REST APIs for experiments
    • Scheduling and approvals

  3. Execution

    • Experiment runners
    • Failure injection

  4. Targets

    • Private cloud
    • Public cloud
    • Hybrid environments

  5. Feedback

    • Metrics and dashboards
    • Experiment reports
How a chaos engineering platform is typically structured. Illustrative only: a general model, not a specific production system.

Technology stack

Backend
  • Python
  • Django
  • Django REST Framework
  • REST APIs
  • Microservices
Infrastructure
  • Linux
  • Shell scripting
  • Private, public and hybrid cloud
Operations
  • Observability

My contribution

  • Develop the chaos engineering framework used to test application resilience across private, public and hybrid cloud environments.
  • Build backend services and REST APIs with Python, Django and Django REST Framework in a microservices architecture.
  • Automate engineering workflows with Linux and shell scripting, with a focus on observability.

Engineering challenges

Many kinds of environment

The same experiment has to work against applications in private, public and hybrid cloud.

Seeing the impact

An experiment is only useful if its effect on the application can be observed and understood.

Next case study

PrepForge