Production performance ratings can provide crucial insights
Many different metrics or key performance indicators (KPIs) can be used to measure the effectiveness of batch production. These include: batch cycle times; actual results versus expectations; capacity utilization; batch exceptions; conformance to schedule; and attainment to standard [1,2]. Some of these are very general categories — an item like “attainment” can include various criteria, such as critical process-measurement trends, laboratory results for a recipe, and product and labor costs, with the standard defined by an exceptional or “gold” batch.
Of course the ability to repeatedly produce on-specification and saleable product is absolutely essential. Once this is accomplished, however, many opportunities remain to improve production and appropriate KPIs may lead the way to cost savings and capacity increases without large capital expenditures.
With today’s increasing interest in real-time performance management, higher levels of management are accessing batch production KPIs in real-time on dashboards, web pages and in e-mails. One difficult aspect of using multiple KPIs is that they usually do not have an equal importance with regard to overall production efficiency and costs. Compounding this, individual KPIs can give conflicting indications, making it difficult to judge overall performance.
One method to make handling multiple KPIs per batch easier is to calculate a composite KPI that provides a single top-level metric for rating batches. The composite KPI can be refined by giving weights to each input KPI, measuring the variability of input KPIs between batches and normalizing the KPI value from 0 to 100%. This refined composite KPI is called the Production Performance Rating (PPR). While one normalized value cannot express all the subtleties of a batch’s production, it can act as a quick measurement of production performance and serve as a filter for finding top- and bottom-performing batches and products. PPRs for batches can be rolled up to master recipe versions, master recipes and products to enable comparisons regarding their production performance.
Basis for a PPR
The rating’s inputs are individual KPIs based upon meeting targets and specifications as well as KPIs that measure production performance against a batch’s peers, i.e., the other batches based upon the same master recipe version. This combination is weighted to meet the needs of each application and yields an overall percentage that can be used to quickly identify the best- and worst-performing batches.
While the criteria and calculation for the rating will vary with application, the ISA-88 standard provides a basis for a set of application-independent KPIs that are a good starting point for most batch-processing facilities. This set of KPIs can be refined with application-specific criteria and customized calculations can be developed to provide more meaningful results. Some of the standard ISA-88 derived KPIs are:
• cycle time;
• number of times a batch was held;
• percent of time a batch was in hold; and
• number of events associated with a batch.
By themselves these measurements have no context, so they must be compared against a target or their peers.
In some cases, comparison to a target makes sense — for example, specific lab results or process measurements may be required before a product can be released. These KPIs provide a base level for defining an acceptable batch.
However, such KPIs do not give adequate insight into the batch’s production performance. For instance, a set of batches may satisfy requirements for release, but some of the batches may have been produced with much shorter cycle times and less operator involvement than others. These batches should be considered higher performing ones because they met the release requirements at a lower cost (less asset utilization and labor cost). Pure target-based KPIs do not differentiate between the higher- and lower-cost batches. That’s where peer-based KPIs can play a role.
A potent combination
Target-based and peer-based KPIs can be used together as inputs to the PPR calculation (Figure 1). The inputs should be weighted to reflect each application’s needs. For example, when meeting release targets is critical, target-based KPIs can be weighted to force a very low rating, even 0, when any release criterion is not met. When all the release criteria are met, the peer-based KPIs can be used to rate the batch’s production performance against other batches of the same master recipe, or even master recipe version. This provides finer resolution that will enable identification of good and poor production characteristics.
PPRs are normalized values from 0 to 100% to provide an easy-to-use scale. This normalization can require tuning the weighting factors and calculation to achieve meaningful results. Once tuned, the normalized scale allows quick identification of top performing batches, which may be those above 95%, 90%, 80% or other level depending upon the application.
Many operating companies traditionally have used an individual batch as the gold standard. Some have identified one golden batch for each unit recipe and built up a composite golden batch by assembling unit recipes from different batches. PPRs enable the rapid identification of the top performing batches and, if carried down to the unit recipe level, the top performing unit recipes. This permits engineers to move beyond the single golden batch concept to look at what production characteristics the top performing batches have in common — for instance, was it the equipment used, time of day, personnel, or ingredients?
Perhaps more important and offering faster results than chasing the golden batches is focusing on the bottom performing or “brown” batches. Not all of these batches necessarily yielded un-releasable or off-specification product. They may have made acceptable quality product, but perhaps required more re-work, resulting in longer cycle times and more operator involvement. Understanding the characteristics that lead to brown batches may enable corrective actions to decrease production variability and costs.
To ease identifying golden and brown batches, the normalized rating value can be used as a filter criterion in a batch historian when selecting batches for analysis and reports.
Coming up with a rating
Calculations for determining a PPR can vary in complexity. A straightforward calculation ( Figure 2) may find a value for each input KPI based upon a comparison to a target value or the mean and standard deviation of its peers. This value is then weighted to give the individual KPI inputs different influences in the final rating. The calculation is performed using a default percentage as a starting point and the result is normalized to 0-100%. This method can easily be expanded to include new KPIs.
More complex formulas may be used so long as they adhere to three key principles:
1. Compare an input KPI to a target or its peers.
2. Weight each input according to its importance to production performance.
3. Normalize the result, restricting it to 0-100%.
When comparing a KPI to its peers, a simple rule-of-thumb can be used as a starting point. This rule(Figure 3) draws upon concepts from Six Sigma, by valuing low variability. If a batch’s KPI, for example its cycle time, is within the upper and lower standard deviation of all its peer batches, then this is good and should increase the rating. If the cycle time is outside the standard deviation limits, this is poor behavior and should decrease the rating. To reward batches with improvements, those with cycle times below the mean and above the lower standard-deviation limit should have a slightly higher rating.
This rule-of-thumb is intended to reward batches that provide consistent cycle times, with a slightly higher reward for those trending below the mean. If a cycle time is below the lower standard-deviation limit, this should be considered “too good to be true.” Perhaps it is due to a breakthrough in production performance but this decision should be withheld until the shorter cycle time is repeated enough for the mean and standard deviation limits to change sufficiently for these batches to be in the green (best performance) zone.
The number of standard deviations to use depends upon the industry, company and process. In most cases three standard deviations is a good starting point. However, in processes with low variability, this may not provide sufficient differentiation and so a lower number may be desirable.
Peer batches need to be defined for each application. In the strictest sense, all batches based upon the same master recipe version are peer batches. If the master recipe is revised, the batches based upon the new version should be grouped separately from previous batches. In other applications batches from multiple master recipes may be considered peers if the recipes are similar enough.
As more peer batches are produced, the mean and standard deviation values will drift as new data points appear. At some point, the PPR for all peer batches will need to be recalculated to provide a level comparison. Whether this should be done each time a batch completes, periodically or upon demand is an application-specific decision.
Depending upon the process and products involved, it may be necessary to create different PPR formulas for each master recipe or group of master recipes to customize them for differences in the processes or products.
Batch versus unit recipe ratings
So far we have looked at batch-level PPRs. While the batch-level rating is the most visible, the unit-recipe-level rating often can provide a more accurate reflection of key performance areas.
One weakness with batch metrics, such as cycle time, is that one key unit recipe, say, that for the reaction, can represent the true bottleneck or high cost/risk portion of a batch. Other unit recipes such as those for preparation, mixing, post-reaction processing and drying can provide variability that is actually caused by another batch’s reaction or other key unit recipe. If this occurs, calculate unit-recipe PPRs and use them as the primary method to compare batches. Alternatively, a key unit recipe’s cycle time, rather than the batch cycle time, can serve as a KPI input to the batch PPR.
While this could be carried down to the operation and phase level, it may provide diminishing returns. However, such calculations may be valuable for very specific periods of a batch’s execution.
A powerful use of PPRs is the ability to roll up results. Roll-ups can be performed on many levels. The table provides some examples. Rolling up of ratings can provide a quick indication of trends. Figure 4 shows the PPR for batches rolled up by master recipe version and master recipe. This roll-up enables the quick comparison of the rating for each version of Recipe A or B. When rolled up to the recipe level, comparisons between Recipe’s A and B can be made, assuming the same formula was used for both recipes.
A potent tool
PPRs can be used as metrics for dashboards, but also can provide a powerful indexing tool for production analysis. Using a batch’s and unit recipe’s ratings as a filter, it is easy to find top- and bottom-performing batches for a number of criteria such as production lines or units, time of day, products and material lots. These sets of batches then can be analyzed using the KPI inputs to gain a better understanding of what causes production problems and higher costs.
When used with a batch historian, these data can be analyzed for trends over time, not just on a recipe or product basis but also to detect other correlations, such as if certain operations, phase classes or units are commonly associated with high or low ratings.
For instance, an engineer may find that all batches of a product in the last quarter that had performance ratings below 50% and then drill down to determine the root causes. Or the engineer may use the averages and standard deviation for performance ratings for all batches of a product to compare different products to see which have the greatest variability. In this case peer comparisons can be made using batch’s PPRs (Figure 5).
Go for the gold
PPRs provide a composite KPI that can be used to identify top- and bottom-performing batches. The ratings can be used to expand the concept of golden batches from one exemplary batch or unit recipe to a set of excellent top-performing batches, pinpointing common traits and characteristics that lead to the best results. The concept also can be expanded to include “brown” batches, to identify root causes of production problems so that corrective actions can be taken to prevent them.
References
1. Jensen, B., “Six Sigma and S88 Unite for Batch Automation Productivity Improvement,” presented at WBF North American Conference 2001, WBF, St. Louis, Mo. (2001).
2. Luedecke, R., D. Sanders and D.D. Baker, “Throughput Improvement at Henkel Surface Technologies,” presented at WBF North American Conference 2002, WBF, St. Louis, Mo. (2002).
David Emerson is a senior systems architect for Yokogawa in Carrollton, Tex. E-mail him at [email protected].
This article is based on a paper that was originally presented at the World Batch Forum North American Conference 2004 and is copyrighted by WBF.
