Reducing False Out-of-Control Signals

Process control is and has been an important aspect of manufacturing, test method operations, and other bio/pharmaceutical processes. FDA’s process validation guidance (1) calls for continued process verification (CPV) as Stage 3 of the guidance. Control charts are important tools for assessing, monitoring, and maintaining process control. Sometimes, organizations resist using control charts due to the perception that too many false out-of-control (OOC) signals are produced. False OOC signals can happen, particularly when the wrong model is used to construct the control chart. This article describes a fix that works in many situations.

A common example of the use of control charts is monitoring the production of a batch of tablets. Depending on the tablet characteristic (weight, hardness, dissolution, etc.) samples of 3-20 tablets are taken by the process operator periodically (e.g., every 30 minutes) during the production of the batch. Using an Xbar control chart, the averages of the samples are plotted versus the order in which the samples were taken. The control limits are computed from within the sample standard deviation.

An example of the resulting chart is shown in Figure 1A. The variation within the control limits, called common-cause variation, represents the variation inherent in the process. Values (points) outside the control limits are said to be due to special-cause variation. The control chart helps detect and reduce the special cause variation, thereby improving the performance of the process. 

When using control charts, typically two types of non-random patterns are observed:

• Sample results outside of the control limits (typically set at the process average ± 3 standard deviations). Such events are referred to as OOC signals.

• Non-random patterns such as trends, drifts, and shifts, up and down. These events are referred to as out-of-trend (OOT) signals and are detected using the Western Electric rules (2).

The process is said to be in a state of statistical control if there are no points outside of the control limits and no non-random patterns (i.e., cycles and shifts) are detected. The OOC and OOT signals provide a basis for human intervention, to identify and eliminate sources of variation, which may require some work (problem solving) to identify. This example demonstrates one of the principles of statistical thinking; namely, understanding and reducing variation provides an opportunity for improvement (3).

Both OOC and OOT signals are based on statistical tests of significance and are included in many statistical software packages (4). OOC and OOT signals can trigger an investigation that can be a wasted effort and organizational anxiety if the signals are indeed false positives. The control chart limits are typically based on ± three standard deviations to provide a good balance between detecting false alarms with missing important signals as recommended by Shewhart (5).

Example 1 shows a process in which 27 samples of 10 tablets each were sampled at hourly intervals across the production of the product and the weight measured on each tablet. Figure 1A is the resulting Xbar chart with the limits based on the within sample tablet-to-tablet standard deviation.

The chart shows no major trends but six OOC signals and one OOT signal are observed. In this chart and others in this article, “1” denotes a point outside the control limits. The numbers 2 through 8 denote different types of OOT points as identified by the Western Electric rules (2).

As seen in Figure 1A, three of the OOC signals are on the low side and three are on the high side. The one OOT signal is associated with a potential process shift (2 out of 3 points more than 2 standard deviations from center line) on one side of the control limits. Ignoring the control limits for a moment, the process appears stable with no major shifts and trends. Is this process really out of statistical control? Is an investigation required? Or are these false positive signals? To answer these questions, it is helpful to review how the control limits should be set.

Setting control chart limits

Variation in the product measurements has three principle sources: manufacturing operations, sampling, and testing. The control limits should be based on short-term manufacturing, sampling, and testing variation as it is the long-term stability of the manufacturing process that is being assessed by the control chart.

Unfortunately, many times the control limits are based on testing variation alone, such as the tablet-to-tablet variation within the sample in Figure 1A. When the short-term manufacturing and sampling variation are real--as they almost always are--and the limits are based on only testing variation, the limits will be too narrow and false OOC and OOT signals will result. The control limits should be based on all the variation inherent in the averages being plotted.

Returning to Example 1, the easy fix is to compute the averages of the different samples and plot the averages on an individuals-moving range (I-MR) chart. The variation of the individual averages is due to short-term manufacturing variation, sampling variation, and test variation. The I-MR chart bases the control limits on the moving range of the averages, which reflects the short-term manufacturing, sampling, and test variation. In essence, the I-MR chart defines average-to-average variation as the variation inherent in the averages and any variation beyond this as special cause variation to be identified and assessed. Figure 1B shows the I-MR chart for the data in Example 1. In Figure 1B, there are no points out of control limits and there is no OOT signals.

Short-term manufacturing and sampling variation have not been taken into account when it is observed that the process appears fairly stable and there are approximately the same number of OOC signals on the high side and the low side. Such a pattern commonly occurs when the control limits are based on an under estimate of the common-cause variation of the averages plotted in the chart.

It is also important to keep in mind that control charts can produce OOC signals when the process is stable and the correct sources of variation have been used to construct the control chart. Setting control limits at ± three standard deviations from the process average and assuming the distribution is normal will, in the long run, produce a 0.3% false positive rate.

Real OOC and OOT events are still detected

The approach of assessing process control by constructing an I-MR chart of the sample averages raises the question: Does this approach limit the ability to detect significant (real) non-random patters such as shifts and trends? The following examples show that this is not the case. 

The control chart for Example 2 (Figure 2A) is for tablet weight of 36 samples of 3 tablets per sample with control limits based on within-sample standard deviation. Figure 2A shows nine OOC signals; some high and some low. Some shifts up and down and a small negative trend (OOT signals) can be seen. Figure 2B shows the I-MR chart of the sample averages. Here, there is now only one sample outside of controls limits. The shifts and negative trend (OOT signals) are still apparent and found to be statistically significant. 

In Example 3, tablet thickness (53 samples of 10 tablets per sample), the Xbar control chart (Figure 3A, control limits based on the within-sample tablet-to-tablet standard deviation) identifies five OOC and five OOT signals.  

The I-MR chart of the sample averages (Figure 3B) shows no OOC signals, but a level shift is detected. This indicates that an investigation of the level shift should be considered, but there is no need to worry about any other OOC signals.

I-MR chart is a generally useful tool

An important question is whether the sample size has an effect on the frequency of OOC and OOT signals. While larger samples will reduce the variation in the sample averages, it doesn’t reduce the variation due to the manufacturing process or the act of sampling. Use of the I-MR chart of the sample averages is still a useful approach.

Don Wheeler points out that the I-MR chart is appropriate when the successive values plotted on the chart are logically comparable and the ranges between the values reflect the local, short-term, routine variation inherent in the values (6). In effect, the process short-term variation is used to create control limits to detect the presence of long-term variation, which indicates process instability at the points in time where the OOC signals appeared.

Understand the sources of variation reflected in the control chart

Control charts that are properly constructed and maintained provide a useful method for monitoring a process. Organizations must understand the sources of variation that are contributing to the variation in the data being evaluated by using the control chart. The proper use of any control chart requires careful thought about the sources of variation that would be considered to be common cause and the sources we would consider to be special cause and want to detect. This will ensure that we only detect the signals that we are interested in.

References

1. FDA, Guidance for Industry, Process Validation: General Principles and Practice, (Rockville, Md., 2011).

2. D.C. Montgomery, Introduction to Statistical Quality Control, 7th Edition (John Wiley and Sons, New York, 2013)

3. R. W. Hoerl and R. D. Snee, Statistical Thinking-Improving Business Performance, (John Wiley and Sons, Hoboken, NJ, 212).

4. L. D. Torbeck. “OOS, OOT, OOC and OOSC”, Pharm Tech, 34 (10), 46-47.

5. W. A. Shewhart, Economic Control of Quality of Manufactured Product (Van Nostrand Co., New York, 1931).

6. D. J. Wheeler, Guide to Data Analysis (SPC Press, Knoxville, Tenn., 2005).