Guest Column | January 15, 2021

Using Production And Postmarket Data To Validate FMEA Assumptions

By Mark Durivage, Quality Systems Compliance LLC

Data

Failure mode and effects analysis (FMEA) risk priority number (RPN) scores have traditionally been used to quantify risks for users, designs, and processes. Recently, action priority (AP) tables were introduced as an alternative method to compute relative risk. 

One common issue with regulatory agencies and certification bodies regarding FMEA is that organizations are not utilizing production and postmarket surveillance data to validate the probability of occurrence and probability of detection assumptions, which leads to inspectional observations and audit findings.

Risk Ranking

FMEA helps quantify and prioritize risk using criticality/severity, occurrence, and detection ratings that are used to determine the level of risk. Criticality/severity (S) is a measure of the seriousness of the possible consequences of a hazard. Probability of occurrence (O) is the likelihood that a cause will occur and that it will induce the detrimental consequences of the associated hazard. Detection (D) is the probability that controls (design, in-process, inspection, alert/warning) will eliminate, mitigate, or catch the defect and prevent escapes.

The output of an FMEA is the relative risk for each failure mode, which is used to rank the failure modes on a relative risk basis. There are two methods of rating the risk: RPN and AP tables.

The RPN output of an FMEA is a relative risk score for each failure mode, which is used to rank the failure modes on a relative risk basis. The scoring system typically used for S, O, and D is from 1 to 5, with 1 considered low risk and 5 high risk. The RPN score is calculated by multiplying the scores for S, O, and D. 

Criticality/Severity

Definition

5

Catastrophic

3

Serious

1

Negligible

Occurrence

5

Certain; 300 failures / year

3

Occasional; 30 failures / year

1

Remote; 3 failures / year

Detection

5

Undetectable; 300 escapes / year

3

Moderate; 30 escapes / year

1

Excellent; 3 escapes / year

Table 1 – Example Criticality/Severity, Occurrence, and Detection Ratings (RPN Method) (See Appendix I – Example RPN Action Criteria and Appendix II – Example Criticality/Severity and Occurrence Action Requirements.)

The AP table assigns one of three suggested rankings for each action based upon the S, O, and D values. The AP tables assign risk weighted first on S, then O, and finally D. Table 2 provides example S, O, and D ratings for the AP method.

The AP rankings are as follows:

  • High Priority: The FMEA team must identify appropriate actions or improve the prevention or detection controls or reduce the probability of occurrence.
  • Medium Priority: The FMEA team should identify an appropriate action or improve the prevention or detection controls or reduce the probability of occurrence.
  • Low Priority: The FMEA team could improve upon the prevention and detection rankings or reduce the probability of occurrence.

Criticality/Severity

Definition

Catastrophic

Moderate

Minor

Occurrence

Certain; 300 failures / year

Moderate; 30 failures / year

Remote; 3 failures / year

Detection

Slight; 300 escapes / year

Adequate; 30 escapes / year

Excellent; 3 escapes / year

Table 2 – Example Criticality/Severity, Occurrence, and Detection Ratings AP Method (See Appendix III – Example AP Table)

Probability Of Occurrence

Production data, specifically nonconformance data, can be used for validating the probability of occurrence assumptions. Nonconformance data can be thought of as the probability of inspection and verification systems catching failure modes internally. But the true probability of occurrence is the sum of the internal failures (nonconformances) and external failure data (complaints) or escapes of nonconformances.

Probability Of Detection

Complaint data can be used for validating the probability of detection assumptions. The probability of detection can also be thought of as the probability of inspection and verification systems allowing an escape of a failure mode.

Organizing Data

The key is to have the nonconformance and complaint systems organized with logical failure modes that are readily identifiable. The data should be easy to collect, quantify, and analyze. The more sophisticated and complex the products and the organization, the more sophisticated and complex the monitoring and measuring systems should be implemented. The examples shown here are simple counts per year; however, normalized rates per units produced or sold per month are probably better as normalized to account for fluctuations in production and sales.

Example failure modes could include defects involving:

  • Labeling
  • Packaging
  • Cosmetics
  • Dimensional
  • Functional
  • Other    

It should be obvious that each of the failure modes identified here can be subdivided further. I strongly suggest that there should be no more than five to 10 top-level failure modes to help facilitate organizing nonconformances and complaints. Each of the top-level failure modes can then be further subdivided, as necessary. Table 3 provides an example of a top-level failure mode for labeling and possible sub-level categories.

  L1 Labeling

L10 Wrong Label

L11 No Label

L12 Illegible Label

Table 3 – Example top-level failure mode for labeling and possible sub-level categories for labeling

Traditional FMEA Example

A team has developed an FMEA for labeling with three failure modes, as shown in Figure 1. Note the numbers in parentheses are the total number of failures/escapes assumed per year. After one year, the FMEA team collected and summarized the internal nonconformance and external complaint data to validate the initial assumptions of probability of occurrence and probability of detection. The summarized data is shown in Figure 1. See Table 1 for FMEA ratings.

Failure

S

O

D

RPN

Nonconformances

Complaints

Total

L10 Wrong Label

3

1

(3)

5

(300)

15

2

9

11

L11

No Label

3

1

(3)

3

(30)

9

0

16

16

L12 Illegible

Labels

3

3

(30)

3

(30)

27

15

18

33

Original Assumptions

Actual

Figure 1 – Traditional FMEA assumptions

For failure mode L10 Wrong Label, the probability of occurrence was estimated to be 1 (three failures per year), but the actual number of failures for the year was 11 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 1 (three failures per year) to 3 (30 failures per year) to reflect the actual data. The probability of detection was estimated to be 5 (300 failures per year). Therefore, the probability of detection can reduce from 5 (300 failures per year) to 3 (30 failures per year) to reflect the actual data of nine reported complaints.

For failure mode L11 No Label, the probability of occurrence was estimated to be 1 (three failures per year), and the actual number of failures for the year was 16 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 1 (three failures per year) to 3 (30 failures per year) to reflect the actual data. The probability of detection was estimated to be 3 (30 failures per year). Therefore, the probability of detection assumption is valid, as there were 16 reported complaints.

For failure mode L12 Illegible Labels, the probability of occurrence was estimated to be 3 (30 failures per year), and the actual number of failures for the year was 33 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from 3 (30 failures per year) to 5 (300 failures per year) to reflect the actual data. The probability of detection was estimated to be 3 (30 failures per year). Therefore, the probability of detection assumption is valid, as there were 18 reported complaints.

Failure

S

O

D

RPN

Risk

Comments

L10 Wrong Label

3

3

(30)

3

(30)

27

Additional mitigations should be implemented. Consider an automated vision inspection system.

L11

No Label

3

3

(30)

3

(30)

27

Additional mitigations should be implemented. Consider an automated vision inspection system.

L12 Illegible

Labels

3

5

(300)

3

(30)

45

Additional mitigations should be implemented. Consider an automated vision inspection system.

Updated Assumptions

 

 

Figure 2 – Updated traditional FMEA

FMEA Action Priority (AP) Method Example

A team has developed an FMEA for labeling as shown in Figure 3 for labeling with three failure modes. Note the numbers in parentheses are the total number of failures/escapes assumed per year. After one year, the FMEA team collected and summarized the internal nonconformance and external complaint data to validate the initial assumptions of probability of occurrence and probability of detection. The summarized data is shown in Figure 3. See Table 2 for FMEA ratings.

Failure

S

O

D

AP

Nonconformances

Complaints

Total

L10 Wrong Label

Moderate

Remote

(3)

Slight

(300)

Medium

2

9

11

L11

No Label

Moderate

Remote

(3)

Excellent

(3)

Low

16

0

16

L12 Illegible

Labels

Moderate

Moderate

(30)

Adequate (30)

High

15

2

17

Original Assumptions

Actual

Figure 3 – Action priority (AP) FMEA assumptions

For failure mode L10 Wrong Label, the probability of occurrence was estimated to be remote (three failures per year), but the actual number of failures for the year was 11 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from remote (three failures per year) to moderate (30 failures per year) to reflect the actual data. The probability of detection was estimated to be slight (300 failures per year). Therefore, the probability of detection can be reduced from slight (300 failures per year) to adequate (30 failures per year) to reflect the actual data of nine reported complaints.

For failure mode L11 No Label, the probability of occurrence was estimated to be remote (three failures per year), and the actual number of failures for the year was 16 (nonconformances + complaints). Therefore, the probability of occurrence must be increased from remote (three failures per year) to moderate (30 failures per year) to reflect the actual data. The probability of detection was estimated to be excellent (three failures per year). Therefore, the probability of detection assumption is valid, as there were no reported complaints.

For failure mode L12 Illegible Labels, the probability of occurrence was estimated to be moderate (30 failures per year), and the actual number of failures for the year was 17 (nonconformances + complaints). Therefore, the probability of occurrence assumption is valid. The probability of detection was estimated to be adequate (30 failures per year). Therefore, the probability of detection can be reduced from adequate (30 failures per year) to excellent (three failures per year) to reflect the actual data of two reported complaints.

Failure

S

O

D

AP

Risk

Comments

L10 Wrong Label

Moderate

Moderate

(30)

Adequate (30)

High

Additional mitigations should be implemented. Consider an automated vision inspection system.

L11

No Label

Moderate

Moderate

(30)

Excellent

(3)

Medium

Additional mitigations should be implemented. Consider an automated vision inspection system.

L12 Illegible

Labels

Moderate

Moderate

(30)

Excellent

(3)

Medium

Evaluate ways to reduce risk.

Updated Assumptions

 

 

Figure 4 – Updated action priority (AP) FMEA assumptions

Conclusion

FMEAs should never be one and done. It is time to ensure that your FMEA process benefits from the use of production and postmarket surveillance data. Using production and postmarket surveillance data can help your organization design and manufacture safer products and demonstrate compliance with international standards, guidances, and regulatory requirements.

Ensure the production and postmarket surveillance data is easy to collect, quantify, and analyze using logical failure modes that are readily identifiable.

I cannot emphasize enough the importance of establishing procedures (documenting) to manage the tools and methods used. Best practice includes providing the rationale for your organization’s use of risk management tools and activities.  The requirements and risk management tools presented in this article can and should be utilized based upon industry practice, guidance documents, and regulatory requirements.

References:

  1. FMEA Handbook 1st Ed, 2019, Automotive Industry Group (AIAG) and the Verband der Automobilindustrie (VDA), Southfield, MI.
  2. Durivage, Mark A., “Is It Time To Say Goodbye To FMEA Risk Priority Number (RPN) Scores?” https://www.pharmaceuticalonline.com/doc/is-it-time-to-say-goodbye-to-fmea-risk-priority-number-rpn-scores-0001

Appendices:

Appendix I – Example RPN Action Criteria

RPN

Risk Acceptability

Action

1 to 14

Low

Although low risk, continue mitigation process as far as possible

15 to 29

Tolerable

This should only be revisited in future design review if corrective action enhances the reliability or product appeal.

30 to 49

Undesirable

Risk is acceptable only if it cannot be further mitigated by organizational or technological solutions which do not reduce the clinical/functional utility of the product. 

Above 49

Intolerable

Risk should be eliminated or reduced by protective measures.  Justification required for risk that is accepted.

 

Appendix II – Example Criticality/Severity and Occurrence Action Requirements

 

 

 

Criticality/Severity

 

Catastrophic

Serious

Negligible

 

Occurrence

 

5

3

1

 

Certain

5

Unacceptable

Unacceptable

ALARP

 

Occasional

3

Unacceptable

ALARP

ALARP

 

Unlikely

1

ALARP

ALARP

Acceptable

 

 

*ALARP - As Low as Reasonably Possible

Appendix III – Example Action Priority (AP) Table

Severity

Criticality

Occurrence

Detection

Action

Priority

Catastrophic

Certain

Slight

High

Adequate

High

Excellent

High

Moderate

Slight

High

Adequate

High

Excellent

High

Remote

Slight

High

Adequate

High

Excellent

Medium

Moderate

Certain

Slight

High

Adequate

High

Excellent

Medium

Moderate

Slight

High

Adequate

High

Excellent

Medium

Remote

Slight

Medium

Adequate

Medium

Excellent

Low

Minor

Certain

Slight

Medium

Adequate

Medium

Excellent

Low

Moderate

Slight

Low

Adequate

Low

Excellent

Low

Remote

Slight

Low

Adequate

Low

Excellent

Low

 

About The Author:

MarkMark Allen Durivage has worked as a practitioner, educator, consultant, and author. He is managing principal consultant at Quality Systems Compliance LLC, an ASQ Fellow and SRE Fellow. Durivage primarily works with companies in the FDA regulated industries (medical devices, human tissue, animal tissue, and pharmaceuticals), focusing on quality management system implementation, integration, updates, and training. Additionally, he assists companies by providing internal and external audit support as well as FDA 483 and warning letter response and remediation services.He earned a BAS in computer aided machining from Siena Heights University and an MS in quality management from Eastern Michigan University. He holds several certifications including CRE, CQE, CQA, CSSBB, RAC (Global), and CTBS. He has written several books available through ASQ Quality Press, published articles in Quality Progress, and is a frequent contributor to Life Science Connect. You can reach him at mark.durivage@qscompliance.com and connect with him on LinkedIn.