Validation can be considered successful only when it presents dependable proofs that a process, equipment, utility, analytical technique, or IT system is under control during its entire lifetime and can fulfill its intended function. Therefore, assessment of the productivity of validation presupposes more than counting number of completed protocols, fully authorized reports and qualified activities.
The pharmaceutical company may be completing the annual validation schedule successfully, but may still face continuous incidents of deviations in the process, breakdowns of equipment, failure in cleaning or unexplainable analytical data variations. In this case, it is necessary to conclude that completion of validations and actual validation results may not always coincide.
A relevant assessment framework must involve three levels of measurement.
1. Execution performance: Whether the validation is done according to plan in terms of time and protocol.
2. Technical performance: Whether the study was able to show that criteria are met.
3. Continuous performance: Whether the operation of the system supports the relevant or qualified state.
The third level is very important because a successful validation is a proof of correctness only at some particular point and further monitoring is required to confirm that the control state is preserved.
Formula:
Validation Plan Completion (%) = (Number of completed validations in the period ÷ Number of planned validations for the period) x 100
The measurements can cover such processes as qualification of equipment, process validation, cleaning validation, analytical methods validation, qualification of utilities and validation of computer systems, if applicable.
However, the high completion rate does not mean that it is effective. The completion shall not be achieved via approvals of incomplete research or reducing the number of tests required. It is also necessary to categorize the delayed activities.
The formula can be represented as:
Success Rate of the First Attempt of Validation (%) = (Number of studies with successful execution of the criteria ÷ total number of studies executed) x 100
A low rate can point to weaknesses in the process understanding, protocol design, equipment readiness, sampling approach, or implementation controls.
The metric requires careful interpretation. A scientifically sound amendment of protocol is not to be regarded as a validation failure. A second attempt after an invalid analytical run should not necessarily be treated in the same way as a second attempt after a genuine failure of the process.
It is necessary to be aware of the number of occurrences based on the following components: type of occurrence, severity of the problem, department involved in the problem-solving process, validation type of occurrence, whether the occurrence happened repeatedly. Some instances of occurrences include forgetting to sample or temporary breakdown of the equipment and running under wrong conditions, etc.
In general, a decline in the number of deviations indicates some kind of improvement but it doesn’t mean that you should be discouraged from announcing deviations. If the number of declared deviations rises, it should mean that the number of those detected increased rather than a decrease in the quality of validation.
When evaluating deviations, I would be more interested in the occurrences with regard to their frequency rather than the total number of them.
- Trends in critical process parameters
- Results of critical quality attributes
- Incidences of batch rejection and failure
- Variability in terms of yields and deviations that keep occurring
- OOS and OOT results if applicable
- Customer feedback
For cleaning validation the data set should include cleaning incidents and failures, residue monitoring results, repeat cleaning incidents and possible deviations.
If the utilities are concerned the data set should include relevant monitoring results, excursions, maintenance history and trends in system performance. Depending on the level of process risk, available data and site monitoring strategy, it is necessary to justify the period and frequency of the review. Any valid process that produces anomalies has to be analyzed.
For instance, if cleaning validation fails because residue remains at a location on equipment that is difficult to clean, repeating the cleaning process may resolve the immediate issue. However, it is important to look beyond that into the possibility of the cleaning process coming under scrutiny with a view to be improved as necessary if need be.
Evaluate CAPA effectiveness based on evidence such as:
These are only examples of effective indicators and not a set of criteria for acceptance. Each company needs to develop their own indicators, according to their specifics, risk, system complexity and necessary procedures.
The validation team has to work with QA, QC, production, engineering departments and relevant system owners to analyze the performance data. During such review, it has to be decided whether the initial assumptions are still valid and whether any changes make any difference with regard to the state of the process validation.
For instance, a manufacturing process may pass the validation process, but after that, because of a change in the raw materials suppliers, equipment configuration, batch size or operating procedures, the process can be affected.
A functional periodic review should cover the following points:
- Performance versus existing validation KPIs.
- Occurrences of significant deviations and repeated errors.
- Noteworthy changes and influence on the validation.
- Effectiveness of CAPA and pending activities.
- Information proving continuing validated or qualified status.
- Suggestions for extra observations, re-validation, or enhancement.
QA’s task is to make sure that the review had a risk-based character and that marked negative tendencies are raised. Statistical methods are useful in detecting variations, however, the method used should match the available data. An isolated result should not be considered a reason for unnecessary actions, while a slow negative tendency should not be neglected for the sake of compliance with the specification in the meantime.
Additional disadvantages are:
Start with Outcomes, Not the Number of Completed Studies
The first step involves setting goals that vivid or desired targets for validation. The goal would vary depending on the activity being performed. If we are talking about the process validation, effectiveness refers to the constant production of processed items of good quality according to the established criteria. If cleaning validation process is in question, effectiveness means the procedure has been successfully applied to manage residues and possible contamination. In the case of equipment validation, it would mean that the device functions properly.A relevant assessment framework must involve three levels of measurement.
1. Execution performance: Whether the validation is done according to plan in terms of time and protocol.
2. Technical performance: Whether the study was able to show that criteria are met.
3. Continuous performance: Whether the operation of the system supports the relevant or qualified state.
The third level is very important because a successful validation is a proof of correctness only at some particular point and further monitoring is required to confirm that the control state is preserved.
Key Indicators for Measuring Validation Performance
It is important to note that one KPI cannot be enough to assess all types of validation activities. The most appropriate approach is the combination of leading and lagging KPI. The following metrics can be adjusted to suit the validation plan used by the site:1. Validation Plan Completion Rate
This KPI assesses if the validation activities started in line with the validation program were completed within the schedule approved in advance.Formula:
Validation Plan Completion (%) = (Number of completed validations in the period ÷ Number of planned validations for the period) x 100
The measurements can cover such processes as qualification of equipment, process validation, cleaning validation, analytical methods validation, qualification of utilities and validation of computer systems, if applicable.
However, the high completion rate does not mean that it is effective. The completion shall not be achieved via approvals of incomplete research or reducing the number of tests required. It is also necessary to categorize the delayed activities.
2. First-Time Validation Success Rate
This parameter indicates the ratio of successful validation studies carried out per the set criteria without needing to repeat the study.The formula can be represented as:
Success Rate of the First Attempt of Validation (%) = (Number of studies with successful execution of the criteria ÷ total number of studies executed) x 100
A low rate can point to weaknesses in the process understanding, protocol design, equipment readiness, sampling approach, or implementation controls.
The metric requires careful interpretation. A scientifically sound amendment of protocol is not to be regarded as a validation failure. A second attempt after an invalid analytical run should not necessarily be treated in the same way as a second attempt after a genuine failure of the process.
3. Validation-Related Deviation Rate
The frequency as well as the actual type of deviation in the conducted validation activities offers valuable insights relating to the quality of their implementation and whether the study is ready to go into the next phase of the research.It is necessary to be aware of the number of occurrences based on the following components: type of occurrence, severity of the problem, department involved in the problem-solving process, validation type of occurrence, whether the occurrence happened repeatedly. Some instances of occurrences include forgetting to sample or temporary breakdown of the equipment and running under wrong conditions, etc.
In general, a decline in the number of deviations indicates some kind of improvement but it doesn’t mean that you should be discouraged from announcing deviations. If the number of declared deviations rises, it should mean that the number of those detected increased rather than a decrease in the quality of validation.
When evaluating deviations, I would be more interested in the occurrences with regard to their frequency rather than the total number of them.
4. Post-Validation Performance
In the case of validation (process validation), post-validation performance represents a strong indicator of validation’s success. Monitoring can be based on the following data:- Trends in critical process parameters
- Results of critical quality attributes
- Incidences of batch rejection and failure
- Variability in terms of yields and deviations that keep occurring
- OOS and OOT results if applicable
- Customer feedback
For cleaning validation the data set should include cleaning incidents and failures, residue monitoring results, repeat cleaning incidents and possible deviations.
If the utilities are concerned the data set should include relevant monitoring results, excursions, maintenance history and trends in system performance. Depending on the level of process risk, available data and site monitoring strategy, it is necessary to justify the period and frequency of the review. Any valid process that produces anomalies has to be analyzed.
5. CAPA Effectiveness After Validation Failures
CAPA concerning validation should aim to eliminate the source of the failure rather than just bring the activity back to a result that is acceptable.For instance, if cleaning validation fails because residue remains at a location on equipment that is difficult to clean, repeating the cleaning process may resolve the immediate issue. However, it is important to look beyond that into the possibility of the cleaning process coming under scrutiny with a view to be improved as necessary if need be.
Evaluate CAPA effectiveness based on evidence such as:
- Recurrence of the same or similar deviation.
- Results of defined follow-up monitoring.
- Compliance with revised operational procedures.
- Consistency in the fulfillment of acceptance standards.
- Decrease in related incidents and process variability throughout the process.
Use a Different Measurement Approach for Each Validation Type
It’s a common error to apply the same KPI to all validation actions. The measures must consider the technical objectives of the study.| Validation type | Useful performance measures |
| Process validation | Batch consistency, CPP and CQA variations, deviations from the procedure, yield variation |
| Cleaning validation | Residues found, cleaning failures, repeat cleaning, difficult places to be cleaned regularly |
| Equipment qualification | Alarm minuteness, breakdowns, calibration problems, repeat test |
| Analytical method validation | Precision, accuracy, robustness, system usability |
| Utility qualification | Monitoring deviations, maintenance issues, microbiological and chemical results where applicable |
| Computerized systems validation | Critical defect monitoring, access control issues, audit trail findings. |
These are only examples of effective indicators and not a set of criteria for acceptance. Each company needs to develop their own indicators, according to their specifics, risk, system complexity and necessary procedures.
Build Validation Effectiveness into Periodic Review
It is important to carry out the validation effectiveness assessment regularly and not only after completion of a new validation study.The validation team has to work with QA, QC, production, engineering departments and relevant system owners to analyze the performance data. During such review, it has to be decided whether the initial assumptions are still valid and whether any changes make any difference with regard to the state of the process validation.
For instance, a manufacturing process may pass the validation process, but after that, because of a change in the raw materials suppliers, equipment configuration, batch size or operating procedures, the process can be affected.
A functional periodic review should cover the following points:
- Performance versus existing validation KPIs.
- Occurrences of significant deviations and repeated errors.
- Noteworthy changes and influence on the validation.
- Effectiveness of CAPA and pending activities.
- Information proving continuing validated or qualified status.
- Suggestions for extra observations, re-validation, or enhancement.
QA’s task is to make sure that the review had a risk-based character and that marked negative tendencies are raised. Statistical methods are useful in detecting variations, however, the method used should match the available data. An isolated result should not be considered a reason for unnecessary actions, while a slow negative tendency should not be neglected for the sake of compliance with the specification in the meantime.
Avoid KPIs That Reward the Wrong Behaviour
Using a validation dashboard for counting easily measurable actions can be counterproductive. For instance, counting the number of approved protocols may lead to quicker document processing without adding value to the study.Additional disadvantages are:
- Formulating targets randomly, without using previous experience or risk assessment;
- Merging minor errors in documentation with serious technical mishaps;
- If CAPA process is completed, but its efficiency has not been confirmed;
- Taking into account data obtained after the validation;
- Considering every repeated test the same;
- Conducting review of measures without allocation of responsibilities or carrying out actions.
Get ready to use editable Validation Protocols in MS-Word FormatView List
No comments:
Post a Comment
Please don't spam. Comments having links would not be published.