Using a Delphi Survey to Assess the Value of Pharmaceutical Process Validation Part II: Expert Opinions

Many doubts regarding the value of process validation (PV) have been raised during the past 30 years. For many, validation is still associated with bureaucracy and excessive paperwork, which only serves to hinder progress. However, a systematic evaluation of its real value has long been overdue.

The value of PV is difficult to estimate because it cannot be evaluated using solid empirical measurements. Therefore, a method capable of both evaluating the subject by informed judgement and structuring the information for which there is some evidence, but not quantitatively measurable knowledge, was needed. Therefore, the synthesis method of technology assessment (TA) was applied.1

For the first part of the assessment, a literature search was performed,2 but a lack of European opinion was evident. To address this, an Internet Delphi survey was performed in autumn 2001 involving 36 experts from the pharmaceutical fields of industry, regulation and academia.

Objectives

The objectives of the survey were:

• to explore the value of pharmaceutical PV and to ascertain the best tools to perform it

• to explore the advantages and disadvantages of PV using the experiences of the pharmaceutical industry and regulators

• to investigate academic attitudes to PV

• to search for differences in approach between the three parties or between countries.

Once collected, the information and knowledge was used to provide support for the further development of pharmaceutical PV.

Methods

The Delphi method. The survey was performed using the Delphi technique - a series of questionnaires based on a structured process with controlled feedback. The results of one round of questions were summarized and used to construct the next questionnaire.

The method attempts to mimic discussion through controlled feedback allowing respondents to change their minds on the basis of the opinions from the previous round. This reiterative process is continued until the participants reach consensus or clear disagreement. To accelerate the process, the Internet and e-mail were used. Furthermore, the Internet offered additional tools during the survey.3

Questionnaires and analysis. Only two rounds of questionnaires were required for adequate consensus. Both questionnaires mostly consisted of multiple-choice questions under the following subheadings:

• How do you feel about PV?

• What are the benefits of PV?

• What are the negative issues of PV?

• How can we get the most out of PV and how can we make it easier and more effective?

• What are the main barriers to positive thinking of PV?

• What is the cost of validation?

For quantitative measurement regarding PV, the multiple-choice answers were categorized using a 1–5 Likert scale, which was used to summarize the results.4 A "no opinion" option was added and these were omitted from the mean calculation.

Attitudes were also tested by asking the participants how far they agreed/disagreed with a number of statements concerning PV in general and its usefulness in specific areas. The overall attitude was also estimated through one control question, which was the first question in the first round (Q1) and repeated as the last question in the second round (Q2). The estimation of the application possibilities of PV was based on the responses to statements presenting applications of PV and statements under the heading of "Benefits of Process Validation."

The sections of the questionnaire trying to ascertain the reasons for the overall attitude were evaluated question by question, calculating percentages for different choices. To determine whether there had been any evolution of the opinions between Q1 and Q2, the quantitative values of Q1 for certain questions were compared with the Q2 values for the same questions. The decision to continue the survey with further rounds rested on this comparison. Because the size of the expert group was relatively small, no further statistical analysis could be performed on the results.

Figure 1: Q1 participants' attitudes towards process validation (using a scale of 1-5, where 55highly positive and 15highly negative) and Figure 2: Overall attitude to process validation (using a scale of 1-5, where 55highly positive and 15highly negative).

Results

In total, 36 experts from the three pharmaceutical parties participated in Q1, 28 of whom continued to Q2. There were participants from 10 European countries - Belgium, Denmark, Finland, France, Germany, Iceland, Norway, Sweden, Switzerland and the UK.3

Overall attitude and application possibilities. The overall attitude to PV appeared to be very positive, but some stated that its applications were not practicable and others believed that PV should be reserved for its original purpose: to ensure patient security. Therefore, the mean value for application possibilities was less than that for overall attitude (Figure 1).

No one had the opinion that PV was just a regulatory requirement without real benefits for the manufacturer. There were no differences in this opinion between the three parties and the opinion was not dependent on the participants' age, gender, education, primary business or function. Q1 answers did not differ statistically from the Q2 results and results for the overall attitude are presented in Figure 2.

The proposed application areas of PV were:

• as an opportunity to learn about the process

• as a quality management tool

• as a marketing tool

• as a cost saving tool

• as a general good business practice.

The lowest reported value related to the usefulness of PV as a marketing tool. The participants agreed that PV increases patient security, and provides cost savings through better performing manufacturing processes and less rejectable material, reprocessing and retesting. Further benefits of PV included cost savings through the early detection of failures and less troubleshooting. However, half of the participants had doubts concerning whether PV could provide cost savings by reducing control requirements for the final product. It was also unclear whether PV would ensure better conformance with regulatory requirements and thus shorten delays in obtaining marketing authorization. No clear differences regarding these attitudes were apparent between Q1 and Q2 - all application areas received positive mean values (Figure 3).

Figure 3: Q1 and Q2 participants' attitude towards PV application (using a scale of 1-5, where 55highly positive and 15highly negative) and Figure 4: Opinions of different pharmaceutical groups regarding the statement: "Process validation can be used as a marketing tool" (using a scale of 1-5, where 55 totally agree and 15 strongly disagree).

Two questions in Q1 highlighted major differences in opinion. Little agreement could be found on whether PV simply constitutes a bureaucratic exercise or whether validation provides an enormous opportunity for consulting companies to earn money at the cost of the pharmaceutical manufacturers.

Some differences between the attitudes of the pharmaceutical parties could be seen regarding the application possibilities of PV. The regulators were most critical of PV as a useful marketing tool (Figure 4); however, they found the application possibilities quite extensive. Of the three pharmaceutical parties surveyed, academia most strongly agreed with the survey statement that PV is a hindrance to flexible product development and a destroyer of innovation (Figure 5); however, they had the least positive attitude to the application possibilities of PV, unlike the industry participants who found merit in all suggested applications.

Figure 5: Opinions of different pharmaceutical groups regarding the statement: "Process validation as it is currently practised is a hindrance to flexible product development and a destroyer of good ideas" (using a scale of 1-5, where 55 totally agree and 15 strongly disagree).

Cost of validation. In Q1 there were seven questions regarding validation costs. The results showed that the costs are difficult to estimate.

Half of the participants, particularly the regulators, considered that validation costs are not one of the main reasons for the high prices of medicinal products. Most were of the opinion that negative attitudes regarding PV could be blamed

on the general assumption that validation is expensive. In Q2, a new question was added, asking participants whether they had ever seen reports clearly showing that validation had increased total quality costs - 92% had not.

The participants were also asked to estimate the cost-effectiveness level of PV if the minimum effort to meet the requirements of the European regulators was made. Sixty eight per cent of Q1 participants found that that level would represent the optimum economic level of PV; this attitude was further strengthened in Q2 (81%).

Figure 6: The traditional quality cost model and Figure 7: The continuous improvement model for quality costs.

In Q2, participants had to choose between two suggested models for pharmaceutical quality costs - the traditional quality cost model (Figure 6) and the continuous improvement model (Figure 7). Eighty five per cent favoured the continuous improvement model.

Optimizing process validation work. The best organizational model. In Q1, the participants were asked to choose the best organizational model for PV. The most popular (86%) was a validation group composed of production, quality assurance (QA) and product development personnel because they felt that as a cross-functional discipline, PV would benefit from experience from all of these departments. Some people even added engineering experts to the group.

Those who opted for a separate validation department did so because they believed that this provided a company with separate resources with validation knowledge. Having a validation group as part of the product development department was supported by some people because they believed it would offer the perfect combination for process improvements. Other participants felt that having a validation group as part of QA department would offer the most benefits.

In Q2, the question was modified, asking participants to name the model that is currently employed by their company or was used by previous employers. Again, the most popular model (55%) was a validation group composed of production, QA and product development staff. It was discovered that in some companies, validation activities had been divided between several departments, each performing different parts of the PV, but this model never worked perfectly, because communication between the departments was ineffective. Other named models are listed in the sidebar "Other named models for optimizing PV."

Prioritizing process validation. Results showed that the most popular (63%) method for choosing which processes to validate was the use of common sense. Failure Mode Effect Analysis (FMEA),5–7 was unknown by 43% of the experts and the process capability method,8 was only slightly better known.

Other named models for optimizing PV

Making process validation easier. Seventy seven per cent of participants were against outsourcing PV and only 3% supported it. In Q1, 69% believed that PV should be started at the earliest stage of product development, but 23% did not find this profitable. In Q2, however, there was less resistance to this idea. Participants who were against this idea interpreted the question differently to those who found it to be a good idea. Apparently, process optimization and PV were mixed here.

Half of the participants based PV on the ongoing follow-up of process parameters rather than on the evaluation of three consecutive production batches, because as one expert stated: "Statistics on many batches gives higher assurance of performance than statistics on three batches."

Barriers to positive thinking. Although this article has already mentioned some of the barriers to positive thinking on PV, further obstructions were identified.

Eighty three per cent of participants thought the lack of PV education was to blame. Furthermore, 45% of industry experts reported that senior management rarely understands quality attributes, which in turn breeds negativity throughout the company. Finally, 89% of the participants (excluding the authorities) found that the unconditional attitude of the regulators regarding PV and their inability to evaluate the need for it on an individual basis, hindered positive thinking.

Individual comments regarding process validation

Discussion

Applications of process validation. The overall positive attitude regarding PV corresponded well to the results of the former literature review.2 Therefore, despite the criticism concerning PV, experts have not only found clear advantages to it, they have also learnt to use it for many other purposes. One possible reason for the former criticism regarding PV is that it was originally viewed as a regulatory burden.9 At that time, people working in the pharmaceutical industry were not trained in PV and, consequently, resented its inherent bureaucracy and expensive scientific work.10

If PV was more business oriented and was introduced, for example, as a part of total quality management, a lot of unnecessary work would have been avoided.9,10 One participant stated: "It is of great importance to involve the entire company in a general quality progress strategy" and it is also important to give sufficient education regarding the meaning of different quality tools to the whole company. When the benefits of PV have been recognized and taught to the younger generation, attitudes have become more positive. From the survey, it was evident that 'traditional' opinions still exist because some academic participants view PV as a "hindrance for creative product development" and those from regulation see it as "nothing to do with marketing." It was also the participants' experience that PV still does not mean less quality control regarding the final product, and that the regulations have partly been applied too schematically by the authors. The new annex to the European Guide to Good Manufacturing Practice (GMP) concerning parametric release11 and the decision of the US Food and Drug Administration (FDA) to re-evaluate its pharmaceutical manufacturing regulations to focus on the highest risks to public health, and to ensure that enforcement of standards does not impede innovation,12 are thus welcome.

Application possibilities should be looked upon with an open mind and new ideas be created, as expressed by one participant: "Validation, although a key GMP requirement, can contribute to the marketing of a product because well developed and validated processes should produce a product that not only meets specifications but is consistent from one batch to another. Dependent upon the medication in question, this could be very important."

Process validation tools

It cannot be denied that PV is a serious issue requiring patience. To survive its challenges and avoid meaningless work, PV has to be supported by a good organizational model, document management systems and other tools for simplifying and prioritizing the procedures.13,14 The organizational model widely favoured is a group consisting of QA, product development and production personnel.15–17 The majority of experts appear to support a team approach that uses experience from different departments. Which departments to include can vary depending on the process to be validated.10,17 To avoid communication problems between the different departments involved, a project leader is required. Furthermore, using this type of teamwork enables the most effective use of the knowledge and documentation available.10,18

Because it is expensive to examine every combination of variables in every process, only the most critical ones should be chosen.19 The favourite prioritizing method for choosing the most critical steps - common sense - can lead to a wrong order of importance, and thus, the most critical processes may not be validated. This approach is a clear indication that PV has become more of a rote activity rather than one based on scientific principles.9 Risk evaluation methods such as FMEA and process capability could be very useful if the experts could be trained to use these methods.

One item discussed at length among the experts was whether to base PV on the ongoing follow-up of process variables or on the validation of three consecutive production batches. Clearly, these two approaches are not alternatives to PV, but they are essential parts of the process life cycle. Manufacturing processes for products that have already been on the market for some time can be validated retrospectively, whereas new products should be validated at the beginning.13,20

Outsourcing different activities of pharmaceutical manufacturing, including PV, has become a trend in recent years. However, the participants did not warm to that idea because they believe that PV includes such essential knowledge regarding the process that it cannot be left to outsiders. The results of PV are also useful in everyday processing and so it is important that these results are created by those closest to it.21 The best way to obtain good control of the processes is to start defining the critical process variables from the beginning of the product development and to progress through stepwise process optimization to the final PV. To be able to get the maximum knowledge of the process permanently, in-house staff should be involved.22–25 Outsourcing could, however, be useful in conjunction with equipment qualifications.26

Process validation cost. It is a common assumption that validation and other quality procedures are expensive, but only limited empirical evidence regarding the behaviour of quality costs exists.27 Few, if any, companies use experimentation to identify their quality cost curves. Even then, most underestimate the non-conformance costs, which can be surprisingly high, particularly in pharmaceutical manufacturing; for example, one multinational pharmaceutical company was fined $500 million because of GMP violations.28

To manage quality costs and to avoid misleading assumptions, every company should have a continuous reporting system for quality costs.29 But for many quality procedures the problem is that the costs are spread among different departments and that the biggest non-conformance costs, such as loss of good reputation, can never be precisely estimated. Many participants found PV expensive although, at the same time, they agreed with the statement that PV could be used to save costs.

Quality control (QC) procedures included in the GMP of pharmaceutical production aim to conform to specifications, but QA procedures such as PV aim to meet the specific requirements of the product. Previously, Taguchi argued that losses occur whenever output deviates from the product specifications and, therefore, investments regarding improving and optimizing processes decrease failure costs more effectively than just the conformance testing.30 Although Taguchi's calculations have not been verified empirically, his suggestion correlates well with the continuous improvement model of quality cost behaviour (Figure 7) that was chosen in the survey as the best model for pharmaceutical quality costs.31 This model indicates that once an effective quality programme is established, ongoing non-conformance cost reductions can be achieved with little or no subsequent increase in conformance expenditures, even with a decrease in conformance investments. Aiming at the target value means investing in the total quality management. These types of investments typically pay back with some delay, but can then further reduce the total quality costs, even if investment in conformance quality decreases.

PV can be used as a tool to reduce total quality costs in the long-term, but to achieve this, a quality culture has to first be created within the company. If PV is performed only to satisfy the regulators, the most probable result is that total quality costs continue to increase. But, if a company fully realizes the meaning of validation and carefully investigates the critical points, then the validation can be performed cost-effectively, resulting in reductions in total quality costs.

Summary and conclusion

The main finding of this survey is the overall positive attitude to PV. However, closer investigation of the results showed that uncertainty and negativity concerning the value of PV, either through a lack of education or because of experiences of the regulators' unconditional attitude, still exists. Furthermore, economic evidence of the advantages of PV in quality costs is difficult to verify, which feeds many old negative assumptions.

To obtain support from the company management and to strengthen the trust of the operating personnel in the value of PV, investment should be made in better quality cost reporting. All this evidence and better communication with people practising PV in their work would also help the representatives of the pharmaceutical academia to understand the positive value of process validations, and thus they would be able to transmit the attitude to future pharmaceutical students.

Results also indicated that many experts were not familiar with the tools for prioritizing the work - these principles should be included in basic education. Regulators should also invest in their own thinking and ability to evaluate the need for validations from case to case.

The statistics regarding the results of the survey, as well as the possibilities of comparison between different groups, were limited because of the low quantity of participants. But, as the group consisted of many European nationalities, different ages and functions, the group can be regarded as representative of pharmaceutical European experts. However, it has to be noted that approximately half of the participants were from Finland.

The survey was successful in providing a general impression of the attitudes to pharmaceutical PV. This type of survey offers a suitable environment for the experts to anonymously discuss and express themselves. From the significant number of extra comments in Q2, it could be concluded that most of the participants found the subject important. Some of these comments have been cited in the text and some others are presented in the sidebar "Individual comments regarding process validation."

References

1. J.M. Morgall, Developing Technology Assessment . . . A Critical Feminist Approach (Lund, Sweden, 1991) pp 21-66.

2. M-R. Helle, J. Yliruusi and J-P. Mannermaa, "A Literature Review of Pharmaceutical Process Validation," Pharm. Technol. Eur. 15(3), 52-57 (2002).

3. M-R. Helle, M. Reijonen and J-P. Mannermaa, "Using a Delphi Survey to Assess the Value of Pharmaceutical Process Validation Part 1: Survey Methodology," Pharm. Technol. Eur. 15(4), 43-48 (2002).

4. F.J. Fowler Jr, Survey Research Methods (Sage Publications Ltd, London, UK, 1988).

5. R.G. Kieffer, S. Bureau and A. Borgmann, "Applications of Failure Mode Effect Analysis in the Pharmaceutical Industry," Pharm. Technol. Eur. 9(8), 36-49 (1997).

6. A. Sahni, "Using Failure Mode and Effect Analysis to Improve Manufacturing Processes," Med. Dev. Diag. Ind. 15(7), 47-51 (1993).

7. R.G. Kieffer, "Validation and the Human Element," PDA J. Pharm. Sci. Technol. 52(2), 52-54 (1998).

8. R.G. Kieffer, "Validation, Risk-Benefit Analysis," PDA J. Pharm. Sci. Technol. 49(5), 249-252 (1995).

9. J. Agalloco, "Validation: An Unconventional Review and Reinvention," PDA J. Pharm. Sci. Technol. 49(4), 175-179 (1995).

10. P. James, "Integrated Validation: A Way of Streamlining Projects to Reduce Project Validation Time and Cost," Pharm. Eng. 8(1), 72-82 (Jan-Feb 1998).

11. "Annex 17 to the EU Guide to Good Manufacturing Practice - Parametric Release" (European Commission, Rue de la Loi, Wetstraat 200, B-1049 Brussels, Belgium, 2001).

12. "FDA Reassessing GMP Inspection," Scrip No. 2776 (28 August 2002) p 16.

13. J. Lignau, "Optimization and Validation of Manufacturing Processes," Drug Dev. Ind. Pharm. 15(6&7), 1029-1046 (1989).

14. A. Chao, "Validation of Oral Dosage Forms: An Industry Viewpoint," paper presented at the Academy of Pharmaceutical Sciences' Midwest Regional Meeting (21 May 1979, Chicago, Illinois, USA).

15. A. Laicher, "Validierung in der Galenischen Entwicklung," Pharm. Ind. 51(10), 1145-1149 (1989).

16. R.A. Nash, "Process Validation: Solid Dosage Forms," Drug Cosm. Ind. (September 1981) pp 38-39, 104-106.

17. T. Byers, "Validation . . . Burden or Benefit?" Drug Cosm. Ind. (June 1982) pp 43-44, 82, 86.

18. J.E. Gillett, "The Six-Stage Hazard Study Procedure and Process Validation," Pharm. Technol. Eur. 6(9), 58-66 (1994).

19. I.R. Berry, "Process Validation of Raw Materials," Pharm. Technol. 5(2), 38-39 (1981).

20. K.G. Chapman, "The PAR Approach to Process Validation," Pharm. Technol. 8(12), 22-36 (1984).

21. P.H. Derrien and J. Deutch, "Deroulement d'une Validation: Approche Concrete," S.T.P. Pharma Pratiq.7(5), 340-344 (1997).

22. R.F. Tetzlaff, "Project Plans for New Drug Development Activities," Pharm. Technol. 22(1), 46-60 (1998).

23. C. DeSain et al., "Process Development That Supports Process Validation," Pharm. Technol. 19(10), 130-136 (1995).

24. V.I. Zimmermann, "Up Scaling Pharmazeutischer Herstellverfahren," Pharm. Ind. 53(4), 377-383 (1991).

25. R.A. Nash, "Process Validation: A 17-Year Retrospective of Solid-Dosage Forms," Drug Dev. Ind. Pharm. 22(1), 25-34 (1996).

26. C. Chemtob, "Principes Generaux de la Qualification des Installations," S.T.P. Pharma Pratiq. 7(6), 431-434 (1997).

27. C.I. Ittner, "Exploratory Evidence on the Behavior of Quality Costs," Operations Research 44(1), 114-130 (1996).

28. "Cautious Progress for S-Plough," Scrip, No. 2740 (24 April 2002) p 10.

29. O. Lecklin, "Laatu Yrityksen Menestystekijana" (Talentum Media Oy, Kauppakaari, Jyvaskyla, Finland, 2002) pp 171-182.

30. G. Taguchi and D. Clausing, "Robust Quality," Harvard Bus. Rev. (Jan/Feb 1990) pp 65-75.

31. H.J. Harrington, Poor-Quality Cost (Marcel Dekker, New York, New York, USA, 1987).