c4.17 Practice chaos engineering
As Werner Vogels puts it, "everything fails all the time." Deliberately inject failures — in test first, then carefully in production — to verify your architecture withstands turbulent conditions and to strengthen cyber resilience. This is an advanced practice for teams with strong availability requirements.
Implementation guidance
Verify resilience by injecting failure
High availability in the cloud is possible, but building resilient architectures is your responsibility. Chaos engineering deliberately simulates failures — an AZ outage, a dependency failure — to confirm your architecture behaves as intended (gracefully degrading, failing over) rather than discovering the gaps during a real incident. Ground your resilience work in the Well-Architected Reliability pillar.
Tooling
AWS Fault Injection Service (FIS) is a managed chaos-engineering service to run controlled experiments (for example stopping instances, injecting latency, simulating an AZ failure). Open-source options include Netflix's Chaos Monkey.
Do it safely
This is an advanced recommendation — deliberately introducing failures is not for everyone.
- Practice in test environments before running experiments in production.
- Adopt it if you have strong availability requirements or must prove your environment withstands failures, and a skilled team to run the tests.
- Understand that mistakes in this testing can cause real outages; start with a small blast radius, define stop conditions, and expand gradually.
How to validate
- Do you validate resilience by injecting failures (for example with AWS Fault Injection Service)?
- Do you practice in test environments before production, with defined stop conditions and a limited blast radius?
- Are experiments run by a skilled team, with results feeding architecture improvements?
Threats addressed
- An architecture assumed to be resilient but never tested may fail unexpectedly during a real outage; chaos engineering surfaces those weaknesses in a controlled way first.