Meet The Experts

  Keith Dorricott
  MCC Ambassador and Director
  Dorricott Metrics and Process Improvement, LTD




  Linda B. Sullivan
  Executive Director
  Metrics Champion Consortium
 
 
 

Q: We are looking at how to set levels on vendor oversight metrics where penalties should be imposed. I couldn’t see anything related to penalties in the MCC metric sets. What levels would you recommend?

A: We are all used to the phrase “what gets measured, gets done” so why not add an extra incentive to what gets measured to make sure it gets done? There are several reasons why this can be detrimental:

1. It will encourage “gaming the metrics” – where the focus is on achieving the metric goal rather than the actual outcome that is desired. For example, high-pressure sales quotas at Wells Fargo bank led to the opening of fake bank accounts to meet the target. In clinical trials, we have come across a situation where site monitors received a penalty for not submitting Monitoring Visit Reports within a set time. This led to site monitors submitting blank reports in the system if they were unable to meet the timeline.
2. Success or failure of processes are rarely down to one party or individual meaning the incentive or penalty is perceived as unfair. This can be detrimental to an open, trusting relationship which most sponsors want with their vendors. For example, the timeliness of central lab tests is not just dependent on the central lab but also on the quality of the samples they receive from the sites, the couriers who shipped the samples, and the sponsor/CRO who provided training to the sites on the requirements. By penalizing the central lab for not meeting the timeliness requirements, you may actually be taking action against the wrong party.
3. The targets are unlikely to take account of changing circumstances. This, again, can be detrimental to an open, trusting relationship. For example, the COVID-19 pandemic led to difficulties in sites entered data within agreed timelines. It is surely unfair to penalize them for this – but making the judgement of an acceptable reason for not meeting a target is subjective. It is likely to be confrontational and so detrimental to an open, trusting relationship.
4. Setting the target level is no longer about the best we can achieve. There are likely to be heated debates about target levels if there are associated incentives or penalties. The target will likely be a compromise rather than focused on what the process can potentially achieve.
5. The focus becomes very narrow rather than on the overall purpose of the process. It drives behavior of having to meet targets rather than improving the process to make it more effective / efficient. Effort is wasted on the narrow focus rather than achieving the overall purpose.

We recommend using metrics to understand processes – what works well, what needs improvement? Along with targets, they are a way of ensuring a clear understanding between parties of what matters. Parties can then work together with the data to understand causes of performance that does not meet (or exceeds) expected levels and then implement actions to improve (or repeat) the performance. MCC metrics for vendors include an indication of which parties’ performance is being measured and this should help with selection of the right metrics and also in metrics use.

We would urge caution in using metrics and associated targets for applying penalties or incentives.