Metrics Champion Consortium

Twitter LinkedIn

March 2019

News icon

  Important News

Measuring Success of Sponsor/Vendor Relationships and the Challenges of Finding Staff with Critical Thinking Skills Key Themes at SCOPE Meeting
Linda Sullivan, MCC Executive Director and Keith Dorricott, MCC Ambassador, attended the SCOPE Summit held in Orlando, FL February 18-21. Linda moderated a breakout session on Vendor Performance Metrics and KPIs and contributed to a panel session on Ensuring Harmony Between All Stakeholders – Sponsor, CRO, and Site – When Site Sourcing.

“This year I participated in the Outsourcing track where I was encouraged to see that the work that we are doing in the MCC Vendor Oversight and Specialty Vendor Work Groups are addressing issues discussed at SCOPE – namely, vendor oversight plans, key areas to measure and how to demonstrate oversight activities,” commented Linda. “Keith and I enjoyed meeting people from our various work groups in person and hearing first-hand about the value of participating in MCC’s virtual groups and how they are using MCC metric sets and tools in their organizations.”

Keith attended the Risk-Based Monitoring track. “Great to hear the latest from some of the large sponsors as they work on predictive analytics and hear from smaller sponsors and CROs on how to overcome some of the practical challenges of implementation,” said Keith. “One theme that came through loud and clear was the challenge of finding people with the right skills for the Central Monitor role. This is a topic we’ve discussed in the MCC Centralized Monitoring Work Group and at the MCC Summit in November 2018.”

Central Lab Metrics v3.0 Final Draft Ready for Review
The MCC Central Lab Work Group has reviewed feedback from MCC and non-MCC members on their draft metric set and has finalized the draft metrics.

Work group facilitator and MCC Ambassador Keith Dorricott and representatives from eight different organizations, including sponsors, vendors and consultants, discussed the feedback during the March 5, 2019 meeting. Particular areas of focus and discussion included:

  • How the set-up process is shown as being very linear but activities are normally carried out in parallel, such as how developing the lab manual doesn’t wait until the database is validated. As a result, the process map was updated and is now in the draft metric worksheet.
  • The definition of query rate, in particular how to handle the circumstance where there is more than one query for a requisition. Are queries measured per requisition or percent of requisitions with one or more queries? It was decided that the latter was more useful although, as with all metrics, it will be important to dig down further to try to understand the detail and the root causes.
  • A missing metric related to the quality of samples received by the specialty lab from the central lab. Recognizing that specialty lab samples do not always go via the central lab but when they do, monitoring of quality as well as timeliness is valuable.

MCC members are encouraged to download the draft metrics spreadsheet and provide feedback to the work group facilitator Keith Dorricott by March 26, 2019.  Access the draft metrics by logging into the MCC site, selecting the “Meetings & Work Groups” tab and then “Central Lab Metrics Work Group.”

Vendor Oversight Work Group Measuring Staff Turnover and Finance KPQs
At the March 6 meeting, the MCC Vendor Oversight Work Group took a closer look at one of the newly added quality metrics – staff turnover. The work group reviewed a formula for annualized turnover that takes account of the changing size of project teams before discussing metrics related to finance.

The staff turnover discussion included:

  • The importance of understanding the purpose of the measurement
  • That the metric can be used for both CRO and sponsor teams
  • The reasons for turnover are important to understand – high turnover might be “good” in some circumstances
  • What should the target be?

The final draft of the quality metrics is available for review on the MCC website. MCC members can review the final draft of the metrics for quality oversight by logging into the MCC site, click on the “Meetings & Work Groups” tab and then “Vendor Oversight Work Group.” Please provide any feedback direct to the work group facilitator and MCC Ambassador Keith Dorricott by March 26, 2019.

The work group then moved to metrics related to finance. The group was polled on financial oversight metrics currently used and also on the Key Performance Questions they want metrics to help answer.

Participants also discussed the upcoming MCC Summit on Clinical Trial Risk and Performance Management and topics that might be of interest to the group.

Centralized Monitoring Work Group Discusses Quality Tolerance Limits
The MCC Centralized Monitoring Work Group revisited the topic of Quality Tolerance Limits (QTLs) and reviewed a proposed approach based on the TransCelerate paper Risk-Based Quality Management: Quality Tolerance Limits and Risk Reporting and previous discussions in 2018.

Participants from 15 MCC USA and European member organizations attended the February 28, 2019 meeting. The work group largely supported the proposed approach although one important change was made regarding the need to report breaches of the QTLs in the final Clinical Study Report. ICH E6(R2) states that “important deviations” should be reported but not necessarily all deviations.

Work group participants also discussed whether QTLs could be study specific or whether they should be common across studies for a Therapeutic Area or even a company.

Once the work group comes to a consensus approach, an additional section to the Centralized Monitoring Guidance Document will be developed.

MCC members interested in joining the Centralized Monitoring Work Group can log into the MCC site and use the registration link found under the “Meetings & Work Groups” tab and then “Centralized Monitoring Work Group” to register for the March 20, 2019 meeting.

The comprehensive MCC Centralized Monitoring Guidance Document is also available. This provides insights from work group members on the practical challenges around implementation of a Centralized Monitoring approach. MCC members can click here to download the guidance document.

Save the Date! MCC’s 2019 Summit will be held Sept 4-5, 2019 in Philadelphia
MCC’s 2019 Clinical Trial Risk and Performance Management Summit will take place at the Lowes Hotel in Philadelphia. More information will soon be released. The 2018 summit was a huge success! Don’t miss out on this year’s meeting. MCC members receive a registration discount. Learn more about the 2019 MCC Summit at visit

Surveys icon


New Tool to Link from Issues to Recommended Metrics: We Want Your Input
The Site Contracting and Trial Master File metrics toolkits include a novel tool to help organizations select the right MCC metrics. There has been great feedback on the tool and we are developing it for other metric sets.

With the tool, you can select the issue(s) that you have with the process (or those you want to guard against) and the tool provides recommended metrics for you to consider implementing. When the metrics were developed, the work group – experts in site contracting from across the industry – provided their input into the typical issues they come across. From this, MCC mapped the most relevant metrics and developed the “Issues to Metrics” tool. It helps organizations select the right metrics to help them improve. We are focused on developing a similar tool for Cardiopulmonary (ABPM, ECG, Spirometry), Data Management & Biostatistics, eCOA, and Imaging.

Are you interested in seeing the typical issues others have in these areas? We would like your input on what the common issues are that you face. We can map these to the existing metrics. Where this feedback highlights metric gaps, we will look at developing additional metrics. You can access the surveys below.

Data Management & Biostatistics

The surveys close on April 10, 2019.

Interested in working with others in the industry on these metrics? If your organization is not already a member, contact MCC Membership Director Terry Holland for more information.

Member news icon

  Member News

MCC Membership Increasing as Consortium Welcomes New Members
MCC welcomes the following new members and thanks those for their continued membership.

MCC’s newest members are:

  • Boehringer Ingelheim Pharma GmbH & Co. KG
  • Pivotarc LLC – Dyana Boutwell (MCC Ambassador)

MCC thanks the following members for their renewal:

  • ACM Global Laboratories
  • Daiichi Sankyo
  • Halloran Consulting Group
  • PPD
  • Triumph Research / Triumph Consultancy

Working group icon

  Featured Work Group

MCC Study Quality Trailblazer Work Group to Discuss Next Generation Approaches to Critical Process and Data Identification 
The MCC Study Quality Trailblazer Work Group reconvenes on March 27, 2019. Working with the Central Lab and Centralized Monitoring work groups has further developed the thinking on implementation of section 5.0.1 of ICH E6(R2) – Critical Process and Data Identification.

Third party vendors such as Cardiac Safety, Central Labs, eCOA and Imaging Core Labs are used in studies when the data they are responsible for generating is critical study data. As a result, the processes to setup the services, provide sites with supplies and training, analyze data/samples and transfer data to the sponsor should be considered critical processes.

MCC members have developed process maps and metric sets for many of these processes and will be discussing how to use the new MCC Process Map Library to help organizations move more quickly to effective controls for those critical processes and data. The MCC Study Quality Trailblazer Work Group will consider a further update to the MCC Risk Assessment & Mitigation Management Tool (RAMMT) to include references to using MCC Process Maps, KRIs and metrics in risk-based quality management.

MCC members interested in joining the Study Quality Trailblazer Work Group can log into the MCC site and use the registration link found under the “Meetings & Work Groups” tab and then “Study Quality Trailblazer Work Group” to register for the March 27, 2019 meeting.

Calendar icon

  Upcoming Meetings

MCC Monthly Work Group Meetings
Meetings are open to participants from MCC member organizations. To register for a work group meeting or for a detailed list of meeting dates and times, please log into the member portal or contact Customer Service for assistance. Not a member? Join Today!

 Work Group Meeting
Central Monitoring
3rd Wednesday of the Month at 10AM EDT
Mar 20, 2019
Apr 17, 2019
May 15, 2019
Imaging Performance
3rd Tuesday of the Month
at 10AM EDT
Apr 16, 2019
Study Quality Trailblazer 
2nd Wednesday of the Month
at 10AM EDT
Mar 27, 2019*
Vendor Oversight
1st Wednesday of the Month 
at 10AM EDT 
Apr 3, 2o19
May 2, 2019*
*Meeting deviates from regularly scheduled frequency.

MCC Metric Sets and Tools Implementation Support Group Meetings
These newly created type of online meetings are held on a quarterly basis to provide MCC members the opportunity to ask questions and share experiences related to implementing specific metric sets or tools.

 Metric Set/Toolkit Implementation Q&A
Date & Time
Cardiopulmonary Metric Toolkit v2.1 (ABPM, ECG, Spirometry)
Central Lab Metrics v3.0
Data Management, Biostatistics & Medical Writing Toolkit 2.0
Jun 11, 2019 10 AM EDT
eCOA Metric Toolkit v2.0
Risk Assessment & Mitigation Management Tool v2.0
Jun 13, 2019 11 AM EDT
Site Contracting Metrics Toolkit 1.0
Jun 13, 2019 10 AM EDT
TMF Metrics Toolkit 2.0
Jun 11, 2019 11 AM EDT


Connect with MCC
MCC staff looks forward to having good conversations with you in 2019.

Clinical Trial Innovation Summit
May 13-15, 2019
Boston, MA

Session Title: Connecting CAPA Preventative Actions to Quality Risk Management to Reduce Systemic-Level Risks on New and Ongoing Studies

Session Information:

  • Explore the link between sections 5.0 (Quality Management) and 5.20.1 (Noncompliances) in ICH (R2)
  • Discuss the importance of documentation of CAPA Action Plans and Quality Management decisions and actions
  • Describe resources developed by an industry consortium that support CAPA Management and Quality Risk Management

Session Participation: Linda B. Sullivan, MBA, Executive Director, Metrics Champion Consortium

Session Title: Using Site Contracting Performance Metrics in Site Selection – Do You Have Access to the Right Data to Understand How the Site Performed?

Session Information:

  • Explore why site contracting cycle times vary by country
  • Discuss approaches organizations are using to reduce cycle time results
  • Review the data your organization should be reviewing to gain insights about which sites perform well in the context of local regulatory and ethical review requirements

Session Participation: Linda B. Sullivan, MBA, Executive Director, Metrics Champion Consortium

European Clinical Trials Inspection Readiness Summit
May 16–17, 2019
London, UK

Session Title: Bridging the Gap Between CAPA and Quality Risk Management to Prevent Systemic-Level Issues

Session Information:

  • Examine the link between sections 5.0 (Quality Management) and 5.20.1 (Noncompliances) in ICH E6(R2)
  • Explore toolkits developed by an industry consortium that support Quality Risk Management and align with inspector expectations
  • Discuss the importance of documentation of Quality Management decisions and actions for inspection readiness and approaches to achieve this

Session Participation: Keith Dorricott, Ambassador, Metrics Champion Consortium

DIA Annual Meeting
Tuesday, June 25, 2019 10:30 am - 11:30 am
San Diego, CA

Session Title: Retooling Risk Assessment to Align with ICH-E6(R2) and Connect to Centralized Monitoring and Risk-Based Monitoring

Session Information:

  • This session will explore new approaches to quality risk management implemented by stakeholders since the adoption of ICH E6(R2) and the vital role of critical thinking and metrics in bridging from risk assessment to centralized monitoring


  • Linda B Sullivan, MBA Executive Director, Metrics Champion Consortium


  • Suzanne Lukac, Senior Account Director, CluePoints
  • Ann McCabe, Director, Process Excellence and Risk Management, Daiichi Sankyo Inc.
  • Sherry Merrifield, Senior Director, Global Operations Management, Syneos Health

Ask icon

  Ask the Experts

MCC Answers Your Metric Questions
Q: Does the MCC consider Query Rate (average number of queries per page) an indicator of site quality? Or would you consider this simply to be a reflection of how many queries are being generated, which you could argue is potentially related to the eCRF design?

Meet The Experts

  Keith Dorricott
  MCC Ambassador and Director
  Dorricott Metrics and Process Improvement, LTD

  Linda B. Sullivan
  Executive Director
  Metrics Champion Consortium

A: MCC metric definitions include four types of metrics:  Quality, Timeliness, Cycle Time, Cost/Efficiency. (See Ask the Experts: "Cycle Time vs. Timeliness metrics – which should I use?") Query rate is defined as a quality metric.

We also have different levels of metrics. For query rate, we have:

  • Study level query rate [Ref DM-04aQ-S]. The Key Performance Question (KPQ) that the metric is designed to help answer is, "Is high quality data being entered into the database?"
  • Site level key risk indicator [Ref RBM006]. The KPQ for this metric is, "Is the site delivering high quality, evaluable data?" It compares query rate among sites in the same study to determine if particular sites are outliers, either high or low relative to average performance.

Each metric is designed to answer a different question; one at study level and one at a site level by comparing sites within a study. At study level, high query rate is most likely due to a systemic issue such as eCRF design issues, high protocol complexity, IT programming issues or poor instructions for eCRF completion. Low query rates may also be a concern – perhaps automated and/or manual queries are not being generated when they should be. It is important to recognize that systemic-level issues can’t be addressed by individual sites. But how can you determine whether you have a systemic-level issue or site-level performance issues? Review the site level key risk indicator to determine whether a subset of sites have high query rates or the rates are high among most sites. If you have a subgroup of sites with high query rates, you need to investigate further. You may find that the sites with high rates are early enrollers that have encountered a systemic level issue before the other sites. Alternatively, you may discover site-specific issues such as staff turnover with poor knowledge transfer or training, or other resource issues. MCC metric definitions include suggestions for additional analysis and investigation you could carry out if the metric value is not to target.

The data from these metrics can highlight concerns. But, as with all metrics, further investigation is needed to understand why the data is as it is, whether it is truly a concern, and whether actions are needed to improve the performance.

So, in direct answer to the question, the query rate can be both an indicator of site quality and a reflection of a systemic-level quality issue such as eCRF design. Understanding whether there is an issue at one, some or all sites may point you in a particular direction in terms of possible cause.

Education icon


Does Your Organization Understand Risk and How to Implement Risk Management in Your Clinical Research Program? MCC Risk Management Course is Available – Register Today
MCC is pleased to announce that MCC eLearning Course 301 – Retooling Risk-Based Quality Management Approaches in the Era of ICH E6(R2): Fundamentals of Clinical Trial Risk Management is available for purchase in the MCC store (MCC members receive a 20% discount).

This course is an expanded, upgraded version of a complimentary live webinar presented by Linda Sullivan, MCC Co-Founder and Executive Director and Keith Dorricott, MCC Ambassador in October 2018. Due to the overwhelming response and positive feedback, the webinar content was expanded into a one-hour, hands-on, on-demand training course during which Linda and Keith walk through risk management concepts, and learners apply the concepts by reviewing a case example and completing the risk assessment and control log included in the course material.

At the conclusion of the course, learners are be able to:

  • Explain the concept of risk and its management
  • Describe risk management in clinical research and the relevance of ICH E6(R2)
  • Identify critical data and processes and use process maps to identify areas of risk
  • Develop risk statements and evaluate risk
  • Describe approaches to controlling risks in clinical research
  • Mitigate some common implementation challenges

You have to develop and implement risk-based quality management programs to comply with ICH E6(R2). You owe it to yourself to discover how.

Learn more about the MCC e301 eLearning course.

Join icon

MCC Membership and User Accounts

Access to MCC metric sets, tools and other assets are limited to employees of MCC member organizations. Please visit the MCC website or email MCC Membership Director Terry Holland for membership information.

Setting Up User Accounts to Log onto Member Portal
If you work for a MCC member organization, you are welcome to set up an individual user account to take advantage of the benefits MCC Membership offers. Click HERE to start the process.


Join the MCC LinkedIn Group to learn about industry performance metrics and stay informed of new developments.

WCG Market Intelligence & Insights

Metrics Champion Consortium,  300 N. Washington St.,,  Falls Church, VA 22046,  USA 
Phone (617) 948-5100 | Toll free (866) 219-3440

© Copyright 2021 by Metrics Champion Consortium

Untitled Document