Transcript
Today's episode, you and I have discussed how knowledge varies not only from site to site, but operator to operator. So we're going to talk a little bit about that today. On the surface, the task is the same, but discrepancies exist in execution and obviously this needs to be addressed. So let's, right out the gate here, let's talk a little bit about what is the best way to assess knowledge and skill. And then I want to talk a little bit about some work you've been doing with the Center for Operator Performance on this very topic.
Best Ways To Assess Operator Knowledge
Dave: Sure, Traci. So, the skill assessment, knowledge assessment, the best way is some objective measure, something that is not an individual's statement that, oh, they're good or they've got this. But a way to measure performance so that there's a defined criterion and you know, the student knows that, hey, I have successfully completed this. So, if it's a skill, you don't watch them perform a particular task and say, oh yeah, I think I got it. You have a criterion for success/failure on that task. And did they meet that criterion?
Same thing for knowledge. Well, I think they understand this concept, just some subjective assessment about it. Instead, you want an objective way to say, yes, this student has grasped the knowledge because they have met this specific criterion. So, in any kind of assessment of training, again between individuals, between companies, you need some way to put a number to it. There's an adage in the process control community, I can't control what I can't measure. And that's true in training as well. If I can't measure it, I can't control it, meaning I can't improve it or make it better. So, there's a strong reliance on the subjective in a lot of the training that is going on, and you need to move that to the objective so that you can actually measure performance, not simply say, yes, this supervisor thinks this individual's qualified.
Traci: Now I know the Center for Operator Performance has moved this to the objective and designed some tests. Can you talk a little bit about those tests and how they address varying knowledge gaps?
Dave: Sure. So a little background, we had a meeting with the various training managers from the operating companies that are part of the Center for Operator Performance. And one of them brought up a situation where they had an operator who was qualified at one refinery, so he is a qualified operator. He wanted to move to a different refinery within that same company. Fine. And so he goes to the new refinery and they're questioning, well, what do we do with him? He's already qualified. And they just said, well, we'll just put him through the same training program as everybody else. That'll probably just be the easiest thing to do. Well, the problem is he failed. And so they came to this quick realization that uh-oh, our training programs are not producing the same type of operator at these two different locations. One said he's qualified, the other says, no, he is not.
And so the management of this company said, uh-oh, well we need to have a way to assess the training programs at these different locations. Are they doing a good job? In the same group, I asked all the training managers, are you all producing good operators? And they all said yes. So, I'd love to find somebody, and of course, who's going to say I produce bad operators? But the reality is there's going to be differences. And as you said, there are differences between individuals and differences in training programs. But the question is, well, who is doing it well? Because if you know who's doing it well, you can model it. You can say, I want to do what Company X is doing. One of the things that the Center for Operating Performance tried to do was say, well, can we create a means, objective means, to test knowledge on a particular topic, and that can be applied across multiple locations and across multiple operators to find out who knows what, how much variance is there.
And so we contracted with Dr. Beth Blickensderfer of Embry-Riddle Aeronautical University. She had developed a test for general aviation pilots regarding weather and did they understand the various concepts of weather that a pilot would need to help us prepare a test. And we chose distillation. Variety of topics, people said, well, why didn't you choose compressors, or why didn't you choose fired heaters? We just said, well, because we could only choose one, and this is the one we chose. An objective test on distillation.
So, we provided them with the questions and the content for her. And then, she was going to do the statistical analysis. So, this test ended up being, we started 50 questions. We ended up tossing a couple out. So, I think it ended up being a 47-question multiple-choice test. We say in high school, multiple guess test. But so this 47-question test, you could take it on... There was an app, you could take it on your phone. And it would go through and it was broken into various aspects of knowledge about distillation, what causes flooding? How do you control overhead pressure? What's the impact of feed temperature? So there were these 47 questions and it was administered at five different locations or five different companies. Within some of those companies, it was at different locations to say, okay, well what do they know?
So, we got the results. Dr. Blickensderfer analyzed the results and came away with what she would say would meet all the criteria for a validated test on distillation knowledge. And what we found, of course, was that, yeah, what we saw or what we were concerned with is that variation between companies and individuals. We were able to pick that up with this test.
What Can You Learn From Test Results?
Traci: That's interesting. And you had brought up the point of, why didn't you do compressors? Should you test for everything?
Dave: Well, that would be the next potential option. So, we did this as a pilot to see if this would work. Would this give us what we were looking for? And so, we did distillation, but then it was, well, maybe we should do compressors. Maybe we should do fired heaters. So, we were looking for things that were applicable across a portion of the industry and of our member companies. But that would be the next step. So, we'll be discussing that at our face-to-face meeting in October to say, okay, do we want to go on and do this for another particular topic so that we can have it? But one of the real advantages here is you have something that any, if you have distillation, you can give your operators this test, and it's been statistically validated, and now it isn't saying whether they're going to be a good console operator because that's not what this test is doing.
This test is simply saying, do they understand distillation? You have to be careful in any testing situation and understand what it is that you're actually measuring here. So this is measuring distillation knowledge that you would hope or think that your console operators would understand. But that wasn't the intent. The intent was to have this yardstick by which different training programs can measure themselves and use that as a way to get better. So for example, the results that came out from this test, it was interesting that of these five locations, four of them had very similar average scores, and that was in the low to mid-70s. So, 70 to 75%, that was where their average was coming in. So that felt good. But a couple of things out of that, though, one of the companies that scored very low, they don't do a lot of distillation.
So, the fact that their operators didn't have a good grasp of distillation, you would expect. It's like, okay, well, you don't do much of that. So, it's capturing that difference in knowledge and expertise you expect them to have. So, it's like, okay, that's great. Now we know that those people who don't know it very well don't do as well on it. Similarly, the variance in the score, so while these averages across the remaining four companies were very similar, the distribution around that average varied greatly. Some of it was very tight. In other words, they had an average score of 75, but it was pretty much because everybody was scoring somewhere between 65 and 85, and yet you had other companies that were also having an average score of about 75, but they varied anywhere between 40 and 90%. And so that high variance is saying, hey, yeah, some of your people really have this down, but a lot of people don't.
So, you probably want to try to make that as small as possible. You want to cluster them around that mean and then move it up and drive to those upper-level performers. So it can be used by a training department to look at and say, okay, where are we, and do we need to do something, or what is it that we should be doing to try to get the overall knowledge base up? And again, the ability to then within these large refining companies, chemical companies, to look across sites and say, well, gee, this site's doing well. What are they doing that the other sites aren't? And is it the people that they're hiring? Is it the nature of how they train? What can we do to take the best, whoever's doing the best, and have our other sister plants do the same?
Traci: I like that. It not only tests the individual but it also zeros in on who's doing it well so that you can replicate what they're doing and hopefully get great results. My question is, what about those folks that don't really test well? Do these tests take that into factor or is that not an issue?
Dave: Well, so there is going to be a certain inherent bias that some people are very good test takers, others aren't. But when the initial results came in, that's where Dr. Blickensderfer's statistical analysis was looking at various questions that she would look for trends to say, oh, wait a minute, there's a whole cluster around this question that they all seem to get wrong, or this group didn't do particularly well. And that's why I mentioned that we started with 50 and we ended up with 47. So, there were some that she said, no, this is not a good question. Because she was looking at not only the answers to that question but did that relate to their answers to other questions? So, were they just guessing kind of things? And so we threw out a couple of questions because it was like, no, that's not a good question. So while we can't totally eliminate the bias, the statistical analysis has tried to minimize that as much as possible to take into account, well, I'm not good at test taking. We need to do something about that.
We've had this at a particular site location in the past. Obviously, this test requires reading. It's on an app, it's a written test. And that was a problem at one site. Now, back in the late ‘80s, they found a real problem in that a large percentage of the overall operator workforce was functionally illiterate. And so, you can run into things like that.
So, any result that you get, you must look at first and say, well, wait a minute, does this make sense? Or, what's going on here? Or why is this happening? And then take appropriate action and say, okay, well, this test isn't going to work for us because we just don't have enough individuals at that reading level to use it. So, it's not measuring what we want it to measure. And that's always the danger in any of these kinds of tests is that you misapply the results and say, oh, well, they don't know anything. Well, no, they may know a lot. Like you said, they're just not good at taking tests. They're not at the reading level for which the questions are written. Try to take that into account, but nothing's a hundred percent.
Traci: Are these pop quizzes, or can they study for them?
Dave: Well, they can study for them. That was one of the interesting things when Beth had previously worked with the FAA; one of the things she was asking was, how much do you study at home? The pilots in this FAA spent quite a few hours at home studying. The operators it was zero. But you can tell people it's coming because the goal isn't to surprise them. The goal is to say, do you have this knowledge that we think is important for doing this particular job? So, it's not a pop quiz. You can take it whenever you want. You can prep for it. While we were testing it out, one of the member companies had a group of engineers take it, and they were taken aback. They were like, whoa, we didn't think it was going to be this hard. The test takes about an hour to take.
There are some of the questions that are hard that you really have to think about in terms of what you're doing. It’s when it's all said and done, I had the same reaction, and one of the operators who took it had the same reaction, which was, this is interesting because you found out the things you didn't know that somebody else thinks you should. You go, wow, wow, that's an interesting problem or an interesting question for it. But you get a score and it can tell you, well, you should probably be studying more in this particular area or here are the things that you missed that are occurring.
Within this test, some of the questions, not all of them, but some of them, have multiple correct answers. But one of the answers is the best answer. It's really trying to get to that granularity of, well, Yeah, I know what to do. It's like, well, yeah, that would have solved that problem, but there's a better solution to it. And that was the correct answer that often you missed. So again, it's a way to identify both for the company and the individual and say, hey, you really didn't have a good handle on tower flooding and what you needed to do by that. And in this particular case, you would get the results. And said, this is a question you missed and here's why. Here's the correct answer on that particular topic.
Continuing Education
Traci: Giving them the correct answer is beneficial. But what about adding maybe coursework with that so that if you want to really buff up on this, go to XYZ?
Dave: Yes, that would be the ultimate one where you can say... And again, that gets back to the companies that are doing really well. I say, well, what are you using to teach distillation? Oh, well, we have a little distillation model. We take them through and then we have this course material, and we spend a day. So that's where you would want to start linking it into, well, what made this group so successful and this group not as successful? It was one of the things that we did do was we correlated, well, we correlated the results with many of what we would call demographic factors, things about the individual. And what we're trying to do there is just understand is there's something that makes one group better than another so we can understand with that. So we broke it down into, we asked things, typical things.
How old are you? How many years of experience do you have? Are you qualified on more than one console job? Are you familiar with distillation? Have you had simulation-based training? And so, what we were looking for is a correlation that said, ah, okay, these demographics predict that you're going to either do well or not do well. And most of us, I think, would say, ah, well, probably years of experience. That's going to correlate very well. The more experience, the better I'm going to do on this. Well, it turns out that's not the case. There is very little correlation between years of experience and performance. So, people with very few years of experience did very well. Some of the people with a lot of experience didn't do well. So that was not a good predictor of what you would want to do.
One of the things that was one of the few that came out was going through simulation-based training. So, if you have a simulator, individuals who were taught using a simulator did statistically better than those individuals who did not. This is where you may want to say, okay, if I really want to get my people good at this the fastest way possible, it would be to include simulation-based training as part of the curriculum because that's what shows up as the key predictor of overall performance. So, whether it's curriculum, whether it's having simulation-based training, but some of it comes out from this to say, ah, okay, I see this is what is making students perform particularly well on the topic.
Traci: Are these one-and-done type tests, or should they be reoccurring to continually benchmark how they're doing, maybe year to year, to see if they're atrophying in their knowledge or gaining knowledge?
Dave: Well, yeah, so you'd want to reapply it. And again, maybe the average doesn't change, maybe you've reduced that variance that okay, yeah, everybody's at about that same level. We don't have some people that really know a lot and some people that are really struggling. It's something you would want to re-administer that's in there. You may want to change some of the distractor questions just to make it interesting and do it that way. But again, because it is objective, you can do that. You can go year to year and hey, I've got this tool. I've got my yardstick for distillation knowledge. I will go out and measure that knowledge again and see how they come up.
So, because again, we're really looking here at the training programs themselves that you can reapply it and find out, well, how is that working? And do we want to... Are we seeing a difference between our new operators and our older operators in terms of, hey, this group has already been qualified, and we changed the program based on this test? Now let's see how the new and the old are coming into this distribution of scores. Or what do they know? What do they not know?
One of the member companies is actually using it on their process engineers and to find out, okay, I've got a new group of process engineers coming in. They're coming in from all these schools across the country, and I'm the manager. I was like, well, I'm assuming they all know certain things, but if you don't have this standardized objective test, you're just assuming that the engineer from MIT is of equal knowledge. I'll use my alma mater to cast dispersions, equal to the engineer from Wright State University. They both graduated and it's an accredited program, but do they have that same level of knowledge? And so this is a way that you can assess not just the operators, but anybody that is dealing in this particular area to find out, okay, what is their level of knowledge and do we have the same level of knowledge across all the individuals or is additional training needed?
Traci: Well, Dave, thank you for providing yet another yardstick so that we can measure and better control what happens within our facilities. Folks, if you want to stay on top of operator training and performance, subscribe to this free podcast via your favorite podcast platform to learn best practices and keen insight. You can also visit us at chemicalprocessing.com for more tools and resources aimed at helping you achieve success. On behalf of Dave, I'm Traci. And this is Chemical Processing's Distilled podcast, Operator Training Edition. Thanks for listening.
