CAPA Effectiveness Evaluation Methods in Pharmaceuticals

The Corrective and Preventive Actions (CAPA) System is a critical aspect of managing the quality of pharmaceuticals. At the same time, CAPA is an important element to assist companies to evaluate deviations, find the root cause(s), corrective actions implemented, and prevent the occurrence of quality issues. However, just because a company implemented a CAPA, that does not mean the CAPA was effective. Increasingly, regulators are focusing on whether the implemented CAPA was effective for the original deviation or problem.

From my experience, many pharmaceutical companies work hard at finding and implementing corrective actions; however, they often do not sufficiently evaluate whether the corrective action is effective. Because of this, the same deviations, complaints, excursion incidents due to environmental monitoring, and process failures continue to be reported.

This is one of the reasons CAPA effectiveness evaluation becomes a primary focus area during regulatory inspections. Inspectors want to see documentation that the company has successfully completed corrective actions and that the original cause of the problem was effectively eliminated and the potential of it occurring again has also been eliminated.

The measure of an effective CAPA system is not how many CAPAs have been closed out, but how many problems have been solved permanently.

What is CAPA Effectiveness Evaluation

CAPA effectiveness evaluation is verifying that corrective and preventive actions (CAPA) taken have met their objectives. This evaluation will confirm:

• Has the root cause been eliminated?
• Has the problem ceased to recur?
• Has there been a change in process performance?
• Has there been a reduction in compliance risk?
• Have any new risks been created?

Effectiveness checks should be based on objective evidence, not assumptions or opinions. CAPAs should never be considered successful based solely on the fact that all action items have been completed.

Why CAPA Effectiveness is Important

One of the most frequently seen compliance observations in pharma companies is linked to an ineffective CAPA system. Some common examples include:

• Deviations that re-occur
• OOS results that re-occur
• Customer complaints that re-occur
• Environmental monitoring failures that re-occur
• Audits with observations that re-occur

Often, after assessing the information supplied during an inspection, the inspection team will provide a conclusion that the real issue is not the initial deviation but the company’s inability to demonstrate whether the corrective actions were effective.

An ineffective CAPA system creates unnecessary risk including but not limited to:

• Product quality issues
• Compliance observations
• Warning letters
• Added investigation workload
• Reduced efficiency for operations

Regulatory Expectations for CAPA Effectiveness

Regulatory agencies expect pharmaceutical companies to use objective evidence to confirm the effectiveness of their Corrective and Preventive Actions (CAPAs). Inspectors will typically review:

• CAPA procedures
• Investigations of the Root Causes
• CAPA verification plans
• Performance metrics
• Historical trend analysis
• Justifications for CAPA closure

CAPAs should be closed with an adequate, documented rationale rather than based on intuition or assumption.

In my experience, inspectors tend to place more emphasis on verifying the effectiveness of CAPAs than on how they were actually implemented.

When Should CAPA Effectiveness be Evaluated?

Evaluating effectiveness at the appropriate time is critical to ensure meaningful evaluations. If you evaluate too soon, you may not have enough data to assess whether recurrence has been prevented. If you evaluate too late, there may be a continuing recurrence.

Some different factors can help determine how long to evaluate, such as:

• How often the process is performed
• The life cycle of the product
• The risk of operational failure associated with the process
• How often the same type of failure occurred in the past

For example:

• If a packaging line deviation occurred, it might take several production runs before the evaluation can be concluded.
• If a CAPA relates to a stability issue, it may take several months of monitoring before it can be assessed.
• If a CAPA relates to a water system failure, it may take more than one time to evaluate a microbial trend.

Ultimately, the evaluation period should be supported by scientific justification.

Method 1: Recurrence Monitoring

Recurrence monitoring is an effective way to monitor the success of CAPA implementation. It looks for a repeat of the original problem after the CAPA has been put into place. Common examples of recurrence are:

• Repeated deviations
• Repeated complaints
• Repeated contamination events
• Repeated equipment failures

If no recurrence occurs in the defined monitoring period, this indicates that the CAPA has been successful. However, lack of a repeat alone does not guarantee success.

Method 2: Trend Analysis

The second method of evaluating the effectiveness of CAPAs is through trend analysis. Trend analysis is one of the most effective ways to evaluate CAPAs. A trend analysis provides information such as:

• Deviation rates
• Number of complaints
• Results of environmental monitoring
• Indicators that measure process performance
• Trends in laboratory investigations

For example, a trend analysis would demonstrate a measurable improvement over time if the CAPA was intended to reduce excursions in monitoring the environment.

Based on my experiences, I have found that companies with good trend programs are able to identify CAPA failures sooner than those that rely solely on recurrence monitoring.

Method 3: Performance Metric Evaluation

CAPAs are often evaluated by pharmaceutical companies using quantitative indicators. Some examples include:

• Reduced deviation frequency
• Reduced complaint rate
• Reduced contamination
• Increased equipment uptime
• Increased number of batches accepted

When developing performance indicators consider the following:

• Must be measurable
• Must be relevant to the CAPA
• Should be objective
• Must have a scientific basis for supporting each indicator

Vague effectiveness measures may lead to ineffective assessment of CAPAs.

Method 4: Process Monitoring

Certain CAPAs are intended to improve the performance of the process. For these CAPAs, effectiveness can be assessed using process monitoring. Some examples include:

• Improved yield
• Process capability analysis
• Reduced variability
• Improved cycle time

Statistical process monitoring provides a high degree of evidence that corrective actions achieved the intended results.

Method 5: Internal Audits

The value of internal audits in verifying the effectiveness of a CAPA system is significant. The goal of an audit includes determining compliance with:

• The rules established by new processes
• If the controls are working as intended
• If the employees understand the revised requirements
• If improvements to compliance have been sustained over time

If an audit shows that an organization was compliant at the time of the audit it can serve as evidence of the effectiveness of a CAPA. Audits should not be the only method of evaluating CAPA effectiveness.

Method 6: Observation of Operational Practices

Direct observation is one way to verify the effectiveness of CAPAs that include behavioral or procedural changes. Examples of this include:

• Gowning
• Documentation
• Cleaning
• Material Handling

In my experience, many managers assume that changes to a process have been implemented successfully based on the results of an audit; however, direct observation of the activity many times has shown that is not the case. Operational verifications must be performed anytime the performance of an individual is a major contributor to risk.

Method 7: Assessment of Training Effectiveness

Many CAPA's involve retraining employees; training completion isn't valid evidence that training past has been effective since employees can complete their training but not comprehend or apply it. The most effective means of assessing training effectiveness:

• Practical Demonstration
• Interview
• Knowledge Assessment
• Workplace Observation

Regulators increasingly expect sound evidence to demonstrate that training has positively impacted staff performance.

Method 8: Quality Risk Assessment Review

A risk-based assessment is a useful assessment methodology for CAPA's with high impact. Organizations may want to reassess:

• Severity
• Likelihood
• Detectability

A lower overall risk score indicates that the CAPA has been effective. Risk assessment and review can be beneficial for validating processes, controlling contamination and improving the manufacturing process.

Method 9: Monitoring of Customer Complaints

Evidence of CAPA’s effectiveness may be obtained from the collection of complaint information. An organization should review:

• The frequency of complaints
• Categories of complaints
• Product Trends
• Market Feedback

When you have successfully improved your CAPA, there will be fewer related complaints about that particular issue. Complaints trending is usually done for CAPAs related to packaging, labeling, and/or product quality.

Method 10: Review of Investigations

One additional practical approach is to analyze repeated investigations of the same or similar problem or complaint. Questions to consider include:

• Are there repeating root causes of the same problem?
• Are there an increasing number of recurring deviations?
• Do you have CAPA related failures being experienced in any other area?

Repeated investigations of the same or similar type of problem or complaint may indicate the failure of the CAPA process to produce the required results.

Common Mistakes in CAPA Effectiveness Evaluation

The repeated mistakes detailed below detract from the effectiveness assessment. Problems recurrently encountered are:

• Closing out CAPAs prematurely
• Using subjective methods
• Not using metrics
• Not measuring trends in data
• Ignoring recurrent trends
• Measuring implementation rather than measuring effective outcomes

A significant error I continuously see is linking the completion of CAPA with its effectiveness, completed actions do not equate to resolving the issue.

CAPA Effectiveness Metrics

Performance indicators that are measurable and relevant should be developed by organizations. Examples of these metrics could be:

• Recurrence rates of deviations
• Recurrence rate of complaints
• Decrease in audit observations
• Percentage decrease in OOS results
• Improvement of your environmental monitoring
• Improvement in process capabilities

Metrics should support the objectives of your CAPA.

Management Review of CAPA Effectiveness

CAPA effectiveness trends should periodically be reviewed by senior management. The management review process can include:

• Occurrence of the same quality problems
• CAPA(s) that are still open
• Age of CAPA(s) and associated trends
• CAPA(s) that were failed to be effective
• Needed resources

Senior management’s participation adds to continuing improvement and strengthens accountability.

CAPA Effectiveness in Regulatory Inspections

Evaluators of CAPA who perform regulatory inspections often evaluate the effectiveness of CAPA because they correlate with the overall maturity of the Quality System. Commonly evaluated components include:

• Criteria of Effectiveness
• Monitoring Periods
• Justification of Closure
• Trend Analysis
• Repeating Deviation

Facilities that demonstrate repeated results of a particular issue are subjected to increased scrutiny of their CAPA system.

Best Practices to Evaluate CAPAs Effectively

These organizations typically have well-established systems and maintain several best practices. Recommended Best Practices:

• Definition of effectiveness criteria prior to implementation
• Use of objective evidence
• Metrics are established which can be quantified
• Consistent trend analysis
• Adequate monitoring for recurrence
• Evaluation based on risk
• Thorough Documentation of the results of effectiveness with various stakeholders

Utilization of these methods provides organizations with increased rates of compliance performance and improved operational performance.

Evaluating CAPA effectiveness is critical and often neglected in the way of managing pharmaceutical quality. Root cause identification and implementing corrective actions are key; however, the most important aspect of evaluating CAPA effectiveness is proving the action will permanently solve the problem and prevent further occurrences.

In my experience, pharmaceutical companies who treat CAPA effectiveness as a structured, data-driven process, with measurable criteria, trending data, operational verification/confirmation and risk-based decision-making ultimately yield successful CAPAs. Companies that consistently evaluate CAPA effectiveness objectively not only enhance their ability to remain in compliance with regulatory requirements, but also maximize the quality of their products, reliability of their operations, and the maturity of their overall quality system.

Regulatory References

1. FDA Guidance for Industry: Quality Systems Approach to Pharmaceutical cGMP Regulations

https://www.fda.gov/media/71023/download
2. FDA Investigations Operations Manual (IOM)

https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/inspection-references/investigations-operations-manual
3. ICH Q10 Pharmaceutical Quality System

https://database.ich.org/sites/default/files/Q10_Guideline.pdf
4. EU GMP Guidelines Volume 4, Chapter 1 Pharmaceutical Quality System

https://health.ec.europa.eu/medicinal-products/eudralex/eudralex-volume-4_en
5. WHO Good Manufacturing Practices Guidelines

https://www.who.int/teams/health-product-policy-and-standards/standards-and-specifications/gmp

Share

Tweet

Share

Pin it

Share