Using a Delphi Survey to Assess the Value of Pharmaceutical Process Validation Part 1: Survey Methodology

Despite the long history of pharmaceutical validation, process validation in pharmaceutical manufacturing continues to be topical. European regulations regarding process validation were renewed in autumn 2001, which again brought the subject under the spotlight. Many people working in pharmaceutical production are now reviewing the state of their compliance practices and posing the question: "How will we benefit from process validation?"

To estimate the value of process validation, a systematic evaluation of the collected opinions and experiences of it was performed using technology assessment (TA). TA is an evaluation process that aims to protect people/society from the consequences of rapid technological developments and attempts to identify all the possible impacts of a technology, not just the intended ones.1 Today, technology in this context refers not only to the logical products of science, but also to the attitudes, processes, apparatus and consequences associated with it, and in that wider meaning, the principles of TA are well suited to the evaluation of process validation as a tool for pharmaceutical manufacturing.

There are a number of different methods of TA; this study used the synthesis (the compilation and evaluation of all available knowledge1) method. Initially, a literature search was performed,2 which revealed a lack of European experts' comments. Therefore, during autumn 2001, an experimental survey was conducted amongst European experts in pharmaceutical fields of manufacturing, regulation and academia to unearth their opinions.

Changes in the Delphi survey

Objectives

The main objectives of the study were to explore the value of process validation and to ascertain the best tools to perform it. Additionally, the study provided an opportunity to test how the principles and methods of TA could be used in the field of pharmaceutical quality assurance.

Methodology

Given that there is no single solution to process validation and that the value of process validation cannot be evaluated by using solid empirical measurement, but rather by informed judgement, 3 a discussion group was found to be an effective method for collecting information. For this reason, the Delphi method was chosen.

Delphi method. The procedure used in the Delphi method aims at structuring and distilling the mass of information from a selected group of experts by means of a series of questionnaires based on a structured process with controlled feedback.4 Moreover, this method was chosen for the following benefits:

• it enables participants from various countries and different fields (industry, authorities, schools) to take part

• it allows anonymous participation; a benefit that was of special value because the survey intended to facilitate discussion between the industry and its authorities, and to obtain comments from different organizational levels

• participants can take part asynchronously; that is, one may choose when to participate

• participants can choose to contribute to areas in which they are best qualified.

For development in the Delphi survey, see sidebar "Changes in the Delphi survey."

The Internet and e-mail. To accelerate communication, the Internet and e-mail were used. Using the Internet was also beneficial because it offered supporting tools for group communication, such as the potential for online discussion.6 Furthermore, the Internet provides a better and more illustrative means of informing the participants of the survey's key elements.

Figure 1 and Figure 2.

Principles of the Delphi technique

Although TA synthesis methods are frequently used to predict future scenarios, they can also be employed to critically examine the state-of-the-art of a given field.1,7 One of the most popular tools of synthesis is the Delphi technique. The aim of most Delphi techniques is the reliable and creative exploration of ideas or the production of suitable information for decision-making.

The replies to one round of questions are summarized and used to construct the next questionnaire. This reiterative process is continued until consensus or clear disagreement is reached among the participants.

Locating experts

One disadvantage of the Delphi method is the definition and selection of experts; that is, whom to regard as an expert and how to create a representative group.

For this survey, experts were defined as those people working on process validation in:

• the pharmaceutical industry

• pharmaceutical authorities

• pharmaceutical schools

• consultant companies.

Representatives of the pharmaceutical industry were chosen from quality assurance, production and product development positions; representatives of the authorities had to either evaluate pharmaceutical and chemical aspects of the marketing authorization applications or work in inspection; and the representatives of pharmaceutical education had to teach process validation. Further details were unspecified; educational background was disregarded and the objective was to obtain the widest possible representation of different organizational levels. The level of experience of pharmaceutical process validation would have been of interest, but because of the limited number of experts in the field, this issue was not used as an exclusion criterion.

Given the pharmaceutical industry's sensitivity concerning knowledge sharing and to get both the regulating authorities and the regulated industry involved in the survey, anonymity was regarded as essential.

The search for participants took 5 months and was the most challenging part of the survey, particularly contacting industry experts because their e-mail addresses could not be located. To find suitable people from the pharmaceutical industry, a snowball method was used.23 Additionally, addresses for quality assurance, product development and production experts were requested from the representatives of appropriate foreign companies in Finland and from the qualified persons of appropriate domestic companies. However, only a few participants from the foreign companies participated because the representatives did not know who the right people were to ask, or the company refused to participate because of time constraints. The only effective way of obtaining willing participants was finally found to be direct telephone contact; this method also worked well for all three pharmaceutical fields.

Figure 3 and Figure 4.

The size of the expert group is also important to the outcome of a Delphi study, but it depends on the homogeneity of the expert population and whether the study searches for qualitative or quantitative results.23 Other Delphi surveys have varied in size from 10–15 up to 2000–3000.23,24 For this survey, the number of experts was limited and the study mainly searched for qualitative results. Thus, a group of approximately 30–50 participants was considered apposite.

The questionnaires

An extremely important part of the Delphi method is the questionnaires, particularly the first round questionnaire (Q1). Q1 needs to be easy and clear, to motivate and encourage the respondents; otherwise they may lose interest.

The design of the questionnaires was carefully discussed by the advisory group before the start. The objective was to form easy-to-use and fast-to-complete questionnaires. It was agreed to keep the number of questions to a minimum and to limit the use of open-ended questions. Not only would these measures avoid making the survey too time consuming, but they would deter those who would use these factors as reasons for abandoning the survey. The use of open-ended questions is, though, often found necessary to eliminate the possible external bias linked to the guiding role of the investigator.7 Instead of open-ended questions, the external bias was eliminated using a mixture of negatively and positively influenced arguments. The questionnaires were also pre-tested with a pilot study among the advisory group and external experts.

Although the aim was to identify the attitude on the usefulness of pharmaceutical process validation, a further aim was to establish whether there are differences in attitudes between European countries or the different parties.

Operationalization of the subject was done under five headings:

• How do you feel about process validation?

• What are the benefits of process validation?

• What are the negative aspects of process validation?

• How can we make process validation easier and more effective to get the most from it?

• What hinders positive thinking on process validation?

The first three questions were designed to measure the overall attitude towards process validation and the final two were mostly determined the reasons underlying the attitudes, and also attempted to find out if some specific tools of process validation were known among the participants.

The initial round started with a background information questionnaire, which also included one final question estimating the overall opinion on process validation. This questionnaire was to be completed before commencing Q1, and thus, the last question served as a control, measuring the attitude at the beginning of the study. The same question was repeated at the end of the second round questionnaire (Q2).

Q1 consisted of 31 questions, most of which were multiple-choice. Only three of the questions were open-ended to increase the inclination and quality of personal expression. Q2 included some new or modified questions, but most were repeated in their original form together with a summary of the answers from Q1.

Figure 5: Pharmaceutical participant demography: gender, education and age.

In Q2, the opportunity to add comments was provided after every question and was encouraged by offering participants the option to answer in their native language. Also in this round, the questions were posed in a slightly different order, with the addition of one new heading - "Cost of validation." In Q1, questions concerning the cost of validation had been spread under other headings.

The Internet as an environment for the survey

The WebCT (WebCT, Inc., Lynnefield, Massachusetts, USA) learning environment26 was chosen as the platform for the survey because it was already being used in Helsinki University and offered the required tools. Extranet homepages were constructed and the questionnaires, together with supporting communication tools and information pages, were available from the Internet public page. The participants were e-mailed the passwords required to access these pages.

Guidance and other information, including definitions of the critical terms, were offered on the homepages and an online forum was available for anonymous discussion. The discussion area was continuously available and an online forum was organized twice during the survey - before and after Q2. The surveys' Internet public homepage can still be viewed.

Methodological results

Response rate and expert demography. Of the total 73 used e-mail addresses, 36 experts' responses to Q1 were received, 28 of whom continued to Q2. Thus, the response rates were 49% and 38% respectively, for Q1 and Q2.

Some of the experts' demography can be seen in Figures 1–5, which show a comprehensive variation in their background. There were participants from Finland, Denmark, France, Germany, the UK, Norway, Sweden, Belgium, Iceland and Switzerland; however, the number of participants from each country was not equal.

Table I: Visits (hits) on the information pages during the survey.

Activity reports

Throughout the survey, the level of activity from the participants varied. During Q1, 19% wrote extra comments in their answers, but in Q2, 46% wanted to define their opinions and, therefore, expanded on their answers.

Some participants experienced technical problems and requested an extension of the deadline. At the beginning of the study, it became apparent that entering the Extranet pages was not possible for all participants because of problems with firewalls or browsers. For these individuals, an HTML alternative was provided. Eight respondents used this alternative in Q1 and three in Q2.

At the start of the survey, one participant initiated an Extranet discussion, however, no one replied. Thirty nine per cent had visited the discussion page, but only 8% had read all the comments on that page. No one took part in the two organized online forums.

From the system report, it could be seen that 69% of the participants had visited the Extranet pages not only to fill in the questionnaires, but also to check the other information available on the pages (Figure 6). The average time to fill in the questionnaire was approximately 5 min for background information, 21 min for Q1 and 41 min for Q2.

Respondent feedback

At the end of the survey, respondents were offered the chance to provide feedback on the survey through an anonymous evaluation form. Only four participants returned the form, all of whom found the subject of the survey interesting, and three out of the four found the methodology suitable for the survey. No one found the Internet technology difficult to use. All found the instructions clear, and that the survey matched their expectations. Two had participated because of interest in the subject, and three out of four gave lack of time as a reason for not participating in discussion and the online forum.

Discussion

Altogether, the methodology worked fairly well for this type of opinion survey. The number of respondents and their written comments indicated that most welcomed the opportunity to express their views. The Delphi method fulfilled the expectations and was the appropriate tool for contacting experts anonymously.

The Extranet homepages of the WebCT functioned satisfactorily for the survey. All the necessary information could be offered in an illustrative format, and the completion and sending of the questionnaires was simple. However, as WebCT is mainly provided for the education market, some unnecessary instructions and numberings could not be deleted or changed, but according to the respondents' feedback, these minor issues did not cause the participants difficulties. The major challenges of the Extranet were the firewall and browser problems, which should, of course, have been eliminated beforehand for all participants. Because these problems arose unexpectedly, the only solution was to offer HTML, which meant missing all of the other information given on the Internet pages. Apparently, many lost interest because of this and the majority of the respondents who used the hyperlink in Q1 discontinued the survey in Q2. The opportunity for discussion and online debate was not utilized even though the availability of these functions was highly underlined.

The biggest problem to overcome was the participants' lack of time. This reason for not participating was given mainly by the representatives from the pharmaceutical industry. The authorities were mostly willing to participate, but, only one or two participants were gathered from each country. The pharmaceutical schools found the subject interesting, although some doubted their suitability.

As previously mentioned, using e-mail to contact potential participants was insufficient. Of course, some of those who did not participate may also have considered themselves not to be experts in the field and, therefore, excluded themselves. Other probable reasons for not participating after receiving the e-mail request may be because not all people are fully familiar with electronic communication, and the explosion of the quantity of information through the Internet and e-mail has caused a need for filtering information. The latter may be one reason for the reported lower response rates for e-mail Delphi surveys compared with the postal versions. However, in this survey, more participants were willing to continue to Q2 than in many comparable surveys in which the response rates normally fall dramatically in the second and subsequent rounds.

A group size of approximately 30 proved satisfactory to gather overall information regarding pharmaceutical process validation opinions. The group cannot be regarded as very homogeneous because it consisted of experts from 10 different European countries from the three different parties. Thus, the group was representative of the expert population despite the limited total number of participants. As can be seen from Figure 2, all the participants, with one exception, reported that they practiced, taught or controlled process validation in their work, and they can, therefore, be regarded as experts. It is important to note that above a certain threshold, the inclusion of more respondents only contributes to marginal statistical and qualitative improvements.

Conclusion

The Delphi method was found to be a suitable tool for measuring opinion in the pharmaceutical field. It is particularly useful in the pharma-ceutical manufacturing sector where the discussion between the regulated industry and the regulators is often difficult to achieve on a "neutral" basis in face-to-face-meetings. Because of this gap between the two parties, many regulations are accepted by the industry without official criticism and real assessment, and as a consequence, a lot of unnecessary work is performed. The Delphi method offers a perfect tool for this type of situation - it can be organized anonymously and can bring together geographically dispersed experts.

The use of the Internet and electronic communication gives the method clear advantages - the survey can be organized much faster, the group size can be easily increased and a lot of supporting information can be provided. However, in a climate where the quantity of electronic information is ever increasing, there is a high chance that some may be in part ignored; this is a threat to the use of electronic communication. For this reason, the Internet Delphi demands high motivation of the participants; ideally, the method can then be used in situations where the participants clearly see the advantages of participating, and where they can be entitled to participate without the fear of time constraints. If these prerequisites can be granted, the Internet Delphi can be used for systematic assessment of any kind of new technology or methodology in pharmaceutical manufacturing or pharmaceutical quality assurance. There should, however, always be available at least one independent, neutral person to serve as a reporter between the rounds and after the survey. Furthermore, the possible firewall and browser problems have to be taken into consideration before the start of the survey.