Managing Clinical Development Risk: Pharma and CRO Report

The Avoca Group conducted a survey in 2013 to examine sponsor and provider practices and perceptions with respect to the management of risk in outsourced clinical trials (1). The survey specifically explored questions regarding the use of, challenges associated with, and benefits gained from various risk-sharing models, formal risk assessment and management approaches, and approaches to risk-based monitoring. The survey was conducted online, with respondents derived from Avoca’s contact list of industry executives and managers (not necessarily Avoca clients), as well as from links to the survey displayed on industry websites.

The survey was taken by 237 respondents; 113 were employed by 70 sponsor organizations (approximately half in the top 20 companies by revenue), and 124 were employed by 66 clinical service providers (approximately 60% in the top 20 companies by revenue). More than three quarters of each of the sponsor and provider respondents held middle or executive management roles. For questions in which respondents were asked to report their own personal opinions, data reported by all respondents are presented. For questions in which respondents were asked to report on behalf of their companies, Avoca selected one response per company (the most senior title) for analysis and presentation so that the strategies and situations of companies with multiple survey respondents would not be overrepresented.

Risk-sharing models
Sponsors and providers were asked questions regarding the use of the following risk-sharing models:

• Guarantees of continued work/revenue stream in exchange for provider commitments

• Provider penalties for failure to achieve milestones/targets

• Provider bonuses for achievement of milestones/targets

• Provider stake in the outcome of program (e.g., company stock).

The frequency of use of, and success with, these risk-sharing models were examined both overall in each sample, and by the type of relationship between the sponsor and CRO (i.e., transactional, preferred provider, and strategic). Elements of a successful risk-sharing relationship were also explored.

Use of risk-sharing models involving provider stake in the outcome of a program was reportedly uncommon (2%-4% of sponsor and 3%-8% of provider respondents, depending on the relationship types). A majority of respondents from both sponsor and provider organizations reported that their companies had used at least one of the other three types of risk-sharing models addressed. Among sponsors, risk-sharing models were more likely to be employed in the context of strategic partnerships (>40% for each of the first three models listed) than in the context of either preferred provider relationships (30%-33%, depending on the model) or transactional relationships (11%-19%). Both sponsor and provider respondents were more likely to report having had “primarily positive” experiences with the use of guaranteed revenue streams in exchange for provider commitments than with the use of other risk-sharing models, and were highly unlikely to report “primarily negative” experiences with this type of model (see Figure 1). In contrast, reports of “primarily negative” experiences were common for provider penalties. The majority of respondents reporting use of bonus/penalty models reported bonus/penalty values of between 2% and 5% of contract value.

Elements of, and challenges in achieving, successful risk-sharing relationships were explored. Commonly mentioned elements of a successful risk-sharing approach included trust, clear expectations, good communication, and commitment on both sides. Conversely, commonly mentioned pitfalls to avoid in risk-sharing arrangements included unclear and/or unrealistic expectations, poor planning, and poor communication.

Risk assessment and management
The survey collected data regarding sponsor and provider risk-assessment practices, as well as data regarding the outcomes of these practices. Sponsors were asked to report both on their experiences with internal risk assessment and management practices, and on their experiences with the risk assessment and management practices of their clinical service providers.

The majorities of each of sponsor and CRO respondents reported using systematic risk-assessment processes for more than half of clinical trials (see Figure 2). Sponsor respondents from top 20 firms were more likely to report frequent use of systematic risk-assessment processes than were those from non-top 20 companies for both in-house and outsourced trials; in fact, a substantial minority of non-top 20 respondents reported never using formal risk-assessment processes in either context.

When asked who contributed to systematic risk assessments, 47% of sponsor and 39% of provider respondents reported that this task was performed jointly by their own companies and their sponsor/provider partners; those who felt that this task was performed primarily by one party were much more likely to report performance by their own companies than by their partners (see Figure 3). Based on the commentary provided to the question, each group’s tendency to see itself as the primary driver of the process was likely due to the fact that risk assessment was often conducted independently within each company, with each partner not necessarily sharing its risk-assessment process or outcome with the other. Within each type of company, project/program managers were most often said to be involved in the risk-assessment process, though operational team members were also stated by a majority to be included.

Nearly 60% of provider respondents indicated that senior management personnel from their company were involved, consistent with the common practice among providers of including business/reputation/relationship risks among those assessed.

Respondents were asked about the types of risks typically reviewed during the systematic risk assessments conducted by their companies; the results are depicted in Figure 4. As can be seen, most of the risks addressed by the survey were typically assessed by a majority of the respondents’ companies, with patient enrollment risks assessed by the greatest majority of each sample. The majority of respondents reported assessing the probability, severity, ability to proactively reduce, and ability to remediate each risk; only a minority reported assessing each risk’s detectability.

The vast majority of respondents reported at least sometimes changing or refining aspects of clinical trial design, execution, or management based upon information gathered from risk assessments, but most of the possible changes addressed by the survey were made by most respondents only occasionally (see Figure 5). Both sponsor and provider respondents were most likely to report adjusting the capture of performance data and/or the review plan for performance data in response to their risk assessments. Nearly half of the sponsor respondents also used information gathered during the risk-assessment process to adjust their levels and/or types of CRO oversight for a majority of clinical trials (data not shown). This practice was more common among non-top 20 sponsor companies than among larger companies, perhaps reflecting the greater flexibility and/or resource constraints among the smaller companies.

All respondents were asked to rate their levels of satisfaction with their own clinical trial teams’ performance with respect to eight different aspects of risk assessment; sponsor respondents were also asked to rate their satisfaction with the performance of their CROs and other services providers in these respects. As can be seen in Table I, mean satisfaction ratings for the areas examined were generally in the neutral to positive range (i.e., between 3 and 4) for own-team assessments, considerably lower than has typically been seen in Avoca Industry Surveys for other aspects of clinical trial performance. Sponsor respondents reported higher levels of satisfaction with their own teams than with their CROs and other service providers on every attribute, with the latter typically receiving mean ratings in the dissatisfied to neutral range (i.e., means between 2 and 3), again lower than has typically been seen for other performance areas in Avoca Industry Surveys.

Despite the lackluster satisfaction ratings, 52% of sponsor and 79% of provider respondents stated that their risk assessment and management approaches at least sometimes resulted in more efficient use of resources, and 66% and 84%, respectively, stated that these approaches at least sometimes resulted in increased quality. The majority of the remaining respondents stated that it was too soon to evaluate the benefits, since the use of formal risk assessment approaches was fairly new within their organizations.

Risk-based monitoring
Respondents were asked by the survey to report on their use of risk-based monitoring (RBM) both overall and by phase and type of clinical trial. Nearly half of top-20 sponsor companies and approximately one-quarter of smaller sponsor companies reported using RBM for a majority of their in-house and outsourced clinical trials (see Figure 6). Among CROs, 39% reported use of RBM for a majority of clinical trials, likely reflecting the mix of preferences among their sponsor clients. With respect to phase of clinical development, experience with RBM approaches was most common for Phase III trials (60% of sponsors, 72% of providers), and least common for Phase I trials (15% of sponsors, 22% of providers). Only a minority of sponsor respondents had experience with use of RBM for trials involving seriously ill or vulnerable patients (33%), adaptive or complex designs (27%), subjective or complex endpoint assessments (24%), complex or device-dependent treatment administration (24%), sites in developing regions (28%), or investigational products for which limited safety information was available (15%). Not surprisingly given their experience with multiple sponsor companies and development programs, more of the CRO respondents reported having had experience with RBM approaches in such contexts (19%-55%).

A variety of monitoring elements were adjusted by respondents based upon the assessed risks as depicted in Figure 7. The frequency of site visits and aggressiveness of data monitoring were adjusted by the greatest percentages by respondents.

Respondents were asked to rate their levels of satisfaction with their CROs’ (for sponsors) or with their own clinical-trial teams’ (for CROs) experience/expertise in RBM, as well as with their ability to efficiently deliver a quality study using RBM. As was seen for the risk-assessment and risk-management variables, mean satisfaction ratings for these areas were in the neutral to positive range (i.e., between 3 and 4 on a scale of 1 [very dissatisfied] to 5 [very satisfied]). Sponsor ratings of CRO performance were lower than CRO self-ratings for each variable (3.0 [N = 40] for experience/expertise and 3.1 [N = 33] for ability to deliver among sponsors; 3.6 [N = 45] and 3.9 [N = 37] among CROs, respectively). Again these mean ratings were lower than has typically been seen in Avoca Industry Surveys for other aspects of clinical-trial performance.

Conclusion
The results reveal approaches to risk assessment and management within the industry to be in an early stage of evolution, with practices and outcomes varying considerably across companies and no clear and common “best practices” in evidence. The majority of respondents do not see risk assessment and management practices to be performance strengths, either within their own organizations or within their providers (where applicable), and many have not yet put into place the infrastructure, including the partnering practices, to take full advantage of their potential. However, most have seen the potential efficiency and quality benefits to be gained and appear to be “believers” in these approaches, so we can expect a focus on this area, and a demand for increased knowledge and skill, to continue. Close attention will be paid in particular to the potential benefits of RBM, and to the impact of quality by design and other “best practice” initiatives that have become top priorities for many organizations.

Reference
1. The 2013 Avoca Report, Industry Survey Executive Summary: “Sponsor and Provider Perceptions on Managing Clinical Development Risk,” The Avoca Group, Princeton NJ.