Thinking Through the Numbers

In this month's "In the Field," we note the publication of Pharmaceutical Manufacturing Research Project—Final Benchmarking Report*, the results of a four-year study by Jeffrey Macher (Georgetown University, Washington, DC) and Jackson Nickerson (Washington University, St. Louis, MO).

Douglas McCormick

Two findings were clear. First, organizations that actively use data systems to track and monitor deviations and resolutions perform better than those that don't. Second, organizations that give "decision rights" for resolving deviations to lower-level quality managers rather than bumping everything up to the department head also perform better.

Grappling with complexities

But beyond these clear signals, the report presents complexities that require thought. And that's the reason to read it, with its 60-odd pages of analysis and nearly 400 pages of charts detailing the practices and performance of the 42 participating manufacturing plants. It's bound to be discussed a lot over the next few months, and we owe it to ourselves to understand what it says and what it doesn't say.

Take, for example, two secondary findings that are also highlighted in the executive summary: using process analytical technology (PAT) tools correlates with some poor performance metrics, and contract-manufacturing plants show lower performance scores than in-house manufacturing operations.

Correlation, not causation

Now, Macher and Nickerson warn repeatedly that their statistical analysis shows correlation, not causation. But, people being what we are, we're bound to see these data misstated in blanket declarations like "PAT degrades productivity" and "contract manufacturers perform poorly." And the reason for reading and understanding the report is to inoculate ourselves against this kind of error.

For example, the study does find a relationship between contract manufacturers and the number of failed batches, expressed in terms of coefficient of correlation: 1.00 suggests a direct cause-and-effect relationship; 0 means no connection; and –1.00 indicates a strong inverse relationship. The correlation between failed batches and contract manufacturing is +0.04 among finished-dose manufacturers and –0.05 for API manufacturers. The correlation between contract manufacturing and yield (actual as a percentage of theoretical) is similar: –0.19 for contract manufacturers of APIs (indicating a very weak association with lower yields) and +0.14 for contract finished-dose manufacturers (indicating a weak association with higher yields).

Underlying Reality

But note the words association and correlation. Contracted products are a very different population from branded drugs made in-house. They are, typically, at (or even past) the end of their patent life, under-resourced and being moved out to make room for higher-value products. Or they present special problems that require outside expertise to solve. Or they're being made for small specialty firms that may not have the process-development strength of the majors.

The report's data on PAT deserve equal rumination. The correlation between using PAT tools and batch failure is +0.07 for finished-dose manufacturers. The correlation is positive, though weak, and the authors emphasize that poor process performance is at least as likely to "cause" PAT as PAT is to cause poor performance. And, indeed, our anecdotal experience is that companies roll out the big process-analysis guns mainly when they have especially sticky problems. (The correlation between using PAT tools and improving process yield is +0.04, by the way.)

Macher and Nickerson's data provoke dozens of questions like these, all of them deserving rumination. And it's the process of coming to these realizations, the process of envisioning the realities behind the numbers, that gives their report its real value to industry professionals. It's a jumping-off point for creative thinking and in no way a fixed prescription.

*Available online at www.olin.wustl.edu/faculty/nickerson/results/.

Douglas McCormick is editor in chief of Pharmaceutical Technology, dmccormick@advanstar.com