ISO 31000 and FMEA Applied to Software Testing

ISO 31000 and FMEA Applied to Software Testing

Most QA teams operate with informal risk instincts — senior engineers know which areas are fragile, and that knowledge guides testing without ever being written down. That works until someone leaves, until the system grows too complex for intuition, or until you're in a regulated industry where "we had a feeling" is not an acceptable audit response.

ISO 31000 and FMEA (Failure Mode and Effects Analysis) are the formal frameworks that take risk management from intuition to documented, repeatable process. They originated in safety-critical engineering — aerospace, automotive, medical devices — but they apply directly to software when you understand how to translate the concepts.

This post explains both frameworks, shows you how to calculate Risk Priority Numbers (RPN) for software features, and helps you decide when this level of rigor is actually worth the overhead.

ISO 31000: The Risk Management Framework

ISO 31000 is an international standard for risk management. It's not specific to software — it's a general framework for any organization managing any kind of risk. What makes it useful for QA is that it provides a structured vocabulary and process that integrates with other quality management systems.

The Core Process

ISO 31000 defines risk management as a continuous cycle with these steps:

1. Establish the context Define the scope of risk management: what system or feature are you assessing, what's in scope, who are the stakeholders, and what are the success criteria? In software testing, this means: which release, which features, which user populations, and which quality dimensions matter (security, performance, correctness, availability).

2. Risk identification Identify what could go wrong. ISO 31000 distinguishes between risk sources (causes), events (what happens), and consequences (effects). In software: a source might be "third-party API with no SLA," an event is "API returns 503 unexpectedly," and a consequence is "users can't complete checkout."

3. Risk analysis Assess likelihood and consequence for each identified risk. This is where you produce your probability × impact scores. ISO 31000 doesn't mandate a specific scoring method — it just requires that you assess both dimensions systematically.

4. Risk evaluation Compare analyzed risks against your acceptance criteria. Which risks fall above the tolerance threshold and require action? Which are acceptable to carry? This produces your prioritized list.

5. Risk treatment Decide how to respond to each risk. In testing, treatment options are:

  • Avoid: Don't ship the risky feature yet
  • Reduce: Add tests that catch the failure before production
  • Transfer: Make the failure visible quickly (alerts, rollback capability) so impact is bounded
  • Accept: Document the risk and carry it consciously

6. Monitor and review Risk management isn't a one-time assessment. The standard requires ongoing monitoring and regular reassessment as conditions change.

How ISO 31000 Applies to QA

The framework gives QA teams a language for discussions that often happen informally. "We should probably test that more carefully" becomes "this risk scores above our acceptance threshold and requires treatment via additional test coverage."

More importantly, ISO 31000's emphasis on documentation and review creates an audit trail. For teams in healthcare, finance, or other regulated spaces, this trail is not optional — it's what you show auditors. For everyone else, it's still useful because it forces explicit sign-off on accepted risks, which prevents the quiet assumption drift that causes production disasters.

FMEA: Failure Mode and Effects Analysis

FMEA is a bottom-up analytical technique that systematically identifies how a system could fail and what the consequences would be. Originally developed by the US military in the 1940s and formalized in aerospace and automotive standards, FMEA is now widely applied in any domain where failure has serious consequences.

The key difference between FMEA and a simple risk matrix: FMEA is structured around failure modes — specific ways a component or function can fail — rather than broad risk areas. It's more granular and more systematic.

The Three FMEA Dimensions

FMEA rates each failure mode on three dimensions, each scored 1–10:

Severity (S): How serious is the effect of the failure on the user or system?

Score Effect
1–2 No effect or negligible effect on user
3–4 Minor effect, workaround available
5–6 Moderate effect, partial loss of function
7–8 Major effect, significant feature loss, user frustration
9 Critical effect, safety or data integrity at risk
10 Catastrophic, system unusable, data loss, regulatory violation

Occurrence (O): How frequently does this failure mode occur (or how likely is it to occur given current controls)?

Score Frequency
1–2 Extremely unlikely, no history of failure
3–4 Low occurrence, isolated incidents
5–6 Moderate occurrence, occasional failure
7–8 High occurrence, recurrent failure
9–10 Very high occurrence, near-certain failure without intervention

Detection (D): How likely is it that current controls (tests, monitoring, review processes) will detect the failure before it reaches the customer?

Score Detectability
1–2 Certain detection — automated tests catch this reliably
3–4 High detection — test coverage is good, likely to catch it
5–6 Moderate detection — some coverage, failure might slip through
7–8 Low detection — sparse coverage, failure probably won't be caught
9–10 Almost no detection — no tests, no monitoring, failure will reach production

Risk Priority Number (RPN)

RPN = Severity × Occurrence × Detection

Range: 1 to 1000. Higher scores indicate higher priority for remediation.

The detection dimension is what makes FMEA different from a simple risk matrix. A high-severity, high-occurrence failure that your automated tests catch reliably has a low RPN — you're already handling it. The scary failures are the ones that are both likely and undetectable with your current controls.

This makes FMEA a direct tool for identifying coverage gaps, not just risk areas.

Practical FMEA Worksheet for a Software Feature

Let's apply FMEA to a user authentication system — one of the most critical components in any web application.

Feature: User Authentication System

Function under analysis: Login flow including password, SSO (Google OAuth), and MFA (TOTP)


Failure Mode 1: Password login accepts incorrect credentials

  • Potential effect: Unauthorized account access, security breach
  • Severity: 10 (security breach)
  • Potential cause: Comparison logic bug, hash function mismatch after library update
  • Occurrence: 2 (rare, but auth libraries do have bugs)
  • Current controls: Unit tests on auth service, integration tests with known credential pairs
  • Detection: 2 (well-covered by tests)
  • RPN: 10 × 2 × 2 = 40 — Low priority (well-detected)

Failure Mode 2: Password login rejects valid credentials

  • Potential effect: Users locked out of accounts, immediate support volume spike
  • Severity: 8 (major feature loss)
  • Potential cause: Hash comparison issue, session generation failure, database connection timeout
  • Occurrence: 3 (more common than false acceptance due to more code paths)
  • Current controls: Happy path integration test
  • Detection: 5 (happy path covered, but edge cases like DB timeout not well tested)
  • RPN: 8 × 3 × 5 = 120 — Medium priority

Failure Mode 3: SSO callback fails silently

  • Potential effect: Users who use Google login cannot access accounts
  • Severity: 7 (Google SSO users completely locked out)
  • Potential cause: OAuth callback URL mismatch, token validation error, provider-side changes
  • Occurrence: 5 (third-party OAuth changes are common; this has happened before)
  • Current controls: Manual test, no automated test for OAuth flow
  • Detection: 8 (no automated detection; manual only catches it if someone tests it)
  • RPN: 7 × 5 × 8 = 280 — High priority

Failure Mode 4: MFA code accepted after expiry window

  • Potential effect: Reduced security for MFA-enabled accounts
  • Severity: 7 (security degradation)
  • Potential cause: Server clock skew, incorrect tolerance window in TOTP logic
  • Occurrence: 3 (clock skew is rare but possible in distributed systems)
  • Current controls: No specific tests for time-boundary behavior
  • Detection: 9 (no tests for this scenario at all)
  • RPN: 7 × 3 × 9 = 189 — High priority

Failure Mode 5: Password reset link doesn't expire

  • Potential effect: Attacker can use old reset links, account takeover risk
  • Severity: 9 (security breach potential)
  • Potential cause: Expiry check logic bug, timezone handling error
  • Occurrence: 2 (rarely introduced but catastrophic if present)
  • Current controls: No automated test for link expiry behavior
  • Detection: 9 (expiry is not tested)
  • RPN: 9 × 2 × 9 = 162 — High priority

Failure Mode 6: Session token not invalidated on logout

  • Potential effect: Sessions remain valid after logout; shared-device security risk
  • Severity: 7 (security issue for shared environments)
  • Potential cause: Token blacklist not updated, cookie not cleared correctly
  • Occurrence: 3
  • Current controls: Manual check during QA rounds
  • Detection: 7 (manual only, not run every release)
  • RPN: 7 × 3 × 7 = 147 — High priority

Ranked Results

Failure Mode RPN Priority
SSO callback fails silently 280 High
MFA code accepted after expiry 189 High
Password reset link doesn't expire 162 High
Session not invalidated on logout 147 High
Valid credentials rejected 120 Medium
Invalid credentials accepted 40 Low

Interpreting the Results

The lowest RPN in this analysis (40) is for "invalid credentials accepted" — the most obviously dangerous failure mode. Why? Because it's well-tested. The tests that already exist provide strong detection, which pulls the RPN down.

The highest RPN (280) is for SSO callback failure — a failure mode that's not as obviously catastrophic but has both meaningful occurrence probability and almost no automated detection. This is exactly the kind of thing FMEA surfaces that intuition misses.

The corrective action list writes itself:

  1. Automate SSO callback testing (reduces Detection score from 8 to ~3)
  2. Add TOTP time-boundary tests (reduces Detection from 9 to ~2)
  3. Add password reset expiry tests (reduces Detection from 9 to ~2)
  4. Automate logout/session invalidation verification (reduces Detection from 7 to ~3)

After implementing these tests, the RPN for items 1–4 drops by 50–70%. You haven't changed the severity or occurrence of the failures — you've improved your ability to catch them before they reach production.

When Does This Level of Rigor Make Sense?

FMEA and ISO 31000 add real overhead. You need time to build the worksheets, maintain them, and keep them aligned with the evolving system. That overhead is worth it in some contexts and overkill in others.

When to use formal FMEA:

  • Regulated industries: Medical devices, healthcare software, financial systems, aviation, automotive. You may be required to demonstrate this rigor.
  • Safety-critical features: Authentication, payment processing, data deletion, access control, data export. These are areas where a failure mode that reaches production can cause serious harm.
  • High-complexity, high-change areas: When a system component is both complex and changing rapidly, informal risk intuition breaks down. FMEA provides a systematic alternative.
  • New team members on critical systems: Documenting failure modes and detection status is an excellent knowledge transfer mechanism.
  • Post-incident analysis: After a production failure, running FMEA on the affected component identifies what controls failed and what new tests would have caught it.

When lighter-weight approaches are better:

  • Early-stage products with rapidly evolving specs (the FMEA will be outdated before the ink dries)
  • Low-stakes features with obvious failure modes and existing coverage
  • Time-constrained releases where a risk matrix suffices for the decisions needed
  • Teams without the process maturity to maintain FMEA documents (a stale FMEA is worse than none)

Combining ISO 31000 and FMEA

In practice, these frameworks complement each other:

  • Use ISO 31000 as the governance layer: context setting, risk acceptance criteria, treatment decisions, audit documentation, and review cadence.
  • Use FMEA as the analytical tool: the detailed failure mode enumeration, RPN scoring, and corrective action tracking.

ISO 31000 answers "how do we manage risk as an organization?" FMEA answers "specifically what can fail here and how do we know we'll catch it?"

Together they give you a complete system: a framework for decisions (ISO 31000) and a systematic method for generating the inputs those decisions need (FMEA).

Practical First Steps

If you want to introduce FMEA to your team without a full process overhaul:

  1. Pick one critical feature — authentication, payment, data export
  2. Run a 90-minute workshop with QA, development, and product
  3. Enumerate failure modes collaboratively (aim for 10–20 per feature)
  4. Score Severity, Occurrence, and Detection as a group
  5. Sort by RPN and identify the top five
  6. Create tickets to add tests for the top five (focus on reducing Detection scores)
  7. Re-run RPN calculation after tests are written to confirm improvement

That's a defensible first FMEA. Iterate from there.


Once you've identified your highest-RPN failure modes, the next step is getting tests in place. HelpMeTest makes it straightforward to build automated coverage for complex flows like auth and payment — without writing code from scratch. Pricing is usage-based, starting at $0.003 per test run.

Read more

Start now free