Filtering Out the Noise

Note: This is the fourth of a five-part series on understanding the concept of variation. Knowledge about variation is one of the components of W. Edwards Deming’s System of Profound Knowledge and is fundamental to correctly interpreting data. 

Last month, I discussed the difference between information and knowledge by analogizing the two concepts to data ponds (information) and data streams (knowledge). A key idea in the transformation of information to knowledge is adding the element of time and visualizing the data in a tool called a process behavior chart. Part of the power of the process behavior chart (PBC) is its ability to filter out the noise in our data; the idea of filtering out data “noise” is the focus of this post.

PBCs, Noise, & Signals

The process behavior chart is able to perform a number of important jobs because of how it takes the variation of any set of data into account. The chart defines the Voice of the Process (the performance of a process over time independent of desired outcomes) by displaying the data in time series. Once you have a set of baseline data, ideally at least 12 data points, it gives you a way to understand if you can make predictions about the performance of future data. When the conditions are satisfied for making predictions about future performance, the chart will also define the range of values that you can reasonably expect to see in the near future assuming nothing fundamental changes within your system. If something has changed within your system, the chart will communicate this to you as well. 

One of the primary flaws with the state testing data discussed earlier in this series (see Figure 1 from Data Has No Meaning Apart from Their Context) is the attempt in both the headline and the description to the left of the chart to attach meaning to each year of test data. The process behavior chart approach to this same data instead would focus on the behavior of the state testing data over time. When it comes to filtering out the noise, Dr. Donald Wheeler, perhaps the foremost expert on process behavior chart methodology, offers the following critical advice in his book Understanding Variation (pp. 29-30):

Variation is the random and miscellaneous component that undermines the simple and limited comparisons. The “noise” introduced by routine variation is what confuses and clouds all comparisons between single values. Until you can allow for the noise in a time series, you cannot fully understand just what may be indicated by a single value. Is the current value a “signal” that something has changed, or does the current value differ from the historic average by nothing but “noise”? The answer to this question is the essence of making sense of any value from a time series.

The whole point of the process behavior chart once you have a data set of historical values is to be able to differentiate between “noise” (common cause variation) and “signals” (special cause variation). Being able to make the distinction between noise and signals or between common cause and special cause variation is the foundation for properly analyzing and interpreting data. Wheeler’s first principle for analyzing and interpreting data is: “No data have meaning apart from their context.” His second principle for understanding data is: “While every data set contains noise, some data sets contain signals. Therefore, before you can detect a signal within any given data set, you must filter out the noise.”

Rules for Finding Signals

The power of the process behavior chart lies in its ability to filter out the noise, or common cause variation, through the construction of natural process limits (the red lines in Figure 1). There are a number of rules that can then be used to detect signals, or special cause variation, within the data set. Rule 1 for detecting a signal is indicated by any single data point that falls outside of either the Upper or Lower Natural Process Limit. Rule 2 for detecting a signal is indicated by a run of eight data points on the same side of the central line (green line). Rule 3 for detecting a signal is indicated when three out of four consecutive data points are closer to the upper or lower limit than to the central line. The three rules are illustrated in Figure 1 below.

Figure 1. Rules for Finding Signals (from Mark Graban’s Measures of Success)

It’s worth it here to revisit the idea of Mistake 1 and Mistake 2 introduced in last month’s post when I discussed the two types of variation.  Mistake 1 likely leads to the biggest source of wasted time and resources and occurs when routine, common cause variation is interpreted as meaningfully different from past performance. In other words, this is interpreting noise as if it were a signal. In organizations like schools, Mistake 1 leads to lots of tampering which in turn typically leads to worse results. Much of this tampering in schools is perpetrated by various components of the national, state, and local education bureaucracy as they attempt to improve schools with good intentions but without an understanding of variation. Mistake 2 is the opposite of Mistake 1. It occurs when a significant change has occurred in a process or system, but it goes undetected. This is failing to detect a signal when it is present. 

The process behavior chart allows the user to differentiate between the noise of common cause variation and the signals of special cause variation. The natural process limits allow us to filter out the noise in order to identify the signals or special events in our data. This in turn allows us to chart a proper course of action for any improvement attempts that are made on our system. Deming’s theory of variation makes it possible to understand the source of differences that occur in our data. As he put it:

Statistical theory and methods are creating a science of management and administration through their ability to aid in the discovery of causes. The central problem in management, leadership, and production is failure to understand the nature and interpretation of variation. 

Knowledge about variation in combination with the other three components of Deming’s System of Profound Knowledge-Appreciation for a System, Theory of Knowledge, and Psychology-offer us a powerful map of theory by which to improve schools if we will only choose to follow it.

***

John A. Dues is the Chief Learning Officer for United Schools Network, a nonprofit charter-management organization that supports four public charter schools in Columbus, Ohio. Send feedback to jdues@unitedschoolsnetwork.org.