To CBT Or Not To CBT…It Shouldn't Be A Question

By Joanna Gallant, owner/president, Joanna Gallant Training Associates, LLC

Recently, a colleague sent me a question about whether performance assessments as part of training – verification of a trainee’s ability to perform a task in accordance with predefined criteria – were an industry standard. 

The most common term for this type of training is "competency based training" (CBT), which requires the trainee to demonstrate the skills they learned to perform a task to the level/standard that has been defined as “competent” performance. This implies that someone determines whether the trainee meets the criteria for attaining that competency through a performance assessment of their execution of that task or skill.

Most companies have done on-the-job training (OJT) training for years, which, in theory, confirms the ability of an operator to perform the task through the discretion of a trainer.

However, much of the OJT training I’ve seen over the years doesn't have set assessment criteria, nor does it document what specifically was assessed to confirm the trainee’s ability to perform the task. Sometimes the involved trainer’s level of task knowledge/skill is questionable.

If I’m the supervisor and I have multiple people performing the same task who were trained by different trainers, how do I know all of these individuals are doing the task the same way (using the same techniques, decision processes, evaluation criteria, etc.)?  The answer is: I don’t, because I do not have a consistent standard against which to compare performance. I also don’t have any evidence that the trainee correctly performed the task.

When presented with the idea of CBT, people think it is a good idea – until they’re expected to implement it. 

What Does CBT Involve?

To institute CBT, you must identify the tasks, competencies, and assessments required to assess whether trainees can perform tasks to an acceptable standard that is typically defined in the SOP.

The following provides a high level overview of the development process:

• Start by identifying the tasks performed in the area – the things someone would be assigned to do. This defines where to consider instituting performance assessments.
• Identify the expectations for each task – what those performing the task need to know and be able to do to perform it completely, including the SOPs that will be used to perform the task. This defines the skills to teach during training, as well as the performance levels people need to achieve competence.
• Order those tasks by level — from easiest to most difficult/most complex. This defines your training plan, or the order in which trainees will learn the tasks. Doing so also identifies any prerequisite training they must have completed prior to beginning to learn the new task.
• Author the assessment. Identify the task components and performance criteria – typically contained in SOPs.  From those, define measureable or observable assessment criteria that will evaluate whether or not the trainee can demonstrate the necessary knowledge and skills required to correctly execute the task.
• Once authored, verify accuracy and appropriateness and implement the assessments into the training program, including applying the assessments to currently trained operators.

While in actuality, there isn’t much more to CBT implementation than defined above, it’s usually a larger project than it initially appears, and without the support of a primary management sponsor to drive the project, it may be a struggle to see that the project comes to fruition.

Other critical success factors include involving those people who have the appropriate knowledge of the tasks and support and buy-in from the facility management to make it happen.

Why It’s Important—A GMP/Regulatory Perspective

When regulators visit, they often find people performing tasks incorrectly despite having completed OJT. Incorrect performance of tasks gets cited in a variety of ways in warning letters and 483 observations. Many of these issues relate to aseptic practices and visual inspection because of the direct risk to patient safety, but it’s observed and cited in other areas as well.

‪Taken collectively, the observations ultimately tie back to what people do and how they do it (i.e. performance), raising questions about:

• ‪How task training was developed and whether it covered the variety of conditions the operator would be expected to address and challenged them to identify each condition. For example, if you visually inspect X, Y, Z product configurations, but only train people on X, how do you know they can successfully inspect Y&Z? (observation #14 in this 2009 483 , 2014 483 observation to drug manufacturer)
• ‪Whether people had the knowledge/experience that the tasks/operations required, as without it, their ability to make appropriate decisions is brought into question.  (2010 warning letter, 2014 & 2013 483 observations to drug manufacturers)
• ‪Whether you were using trainers who don’t have experience performing the task, or evidence exists that the trainer is not performing the task incorrectly.  If the trainer can't perform the task, how can they teach someone else to do it correctly? (2013 483 observations to biologics manufacturers)
• ‪Whether the SOPs don't provide enough information for consistent performance of tasks, leading to people performing them differently. This leads to the question of which way is correct, and why were those performing the tasks incorrectly not taught the proper way? (2013 & 2014 483 observations to drug & biotech manufacturer)
• Whether a qualified person observed a trainee’s performance for adherence to gowning and aseptic procedures during a media fill to qualify the operator. How do you know the trainee executed the procedure properly and didn’t compromise the media fill?  (2012 483 observation to a PET drug manufacturer)

‪What Does This All Mean?

Beneath all of these observations is the underlying question of whether we’re preparing personnel to perform tasks correctly, and whether they have what they need to enable correct performance.

‪Eventually, if inspectors continue to see people performing tasks incorrectly, they'll ask whether the person was ever able to do the task correctly, and if there is any evidence proving their abilities.

The only truly acceptable answer to this question will be: “Yes, prior to releasing the person to perform the task independently, we verified their ability to do the task correctly in a documented performance assessment.”

In truth, the question has already been asked — but only at the system level, not about specific individuals. Reference this December 2012 warning letter, in which personnel were not properly performing their assigned tasks. Despite the company’s response that the involved personnel were retrained, the FDA wanted the company to explain why their “training system wasn’t able to recognize, identify, and mitigate these performance lapses.”

Why Is This So Hard For Companies To Do?

One reason is that it may require a fundamental change in how training is currently carried out. 

Many companies still rely heavily on “read & understand” training, because it’s fast and not very resource-intensive, even though we know it’s not a very effective training method. Or, in the situations in which we do OJT, our focus is to get the person up and functioning as quickly as possible. Therefore, training often becomes a question of how quickly someone can be made available, rather than determining whether they can perform to a specific standard. Performance may be assumed, but not actually be assessed.

Another reason for difficulty is that CBT requires performance standards or criteria to be available.

One of the common items found during development of CBT programs is that the task procedures are lacking in detail or missing components of the task, meaning there may not be a set standard for some or all of the tasks being taught.

CBT also brings performance management to the forefront – an area in which many people are not comfortable working. What do you do when someone can’t perform to the standard or meet the criteria? Sometimes coaching works, or the employee simply needs more time to practice the techniques.  However, in some cases, it becomes a performance management issue; the person may not be able to perform the job they were hired to do.

‪Returning back to the FDA’s training system observation, managing personnel performance is really what that observation – and CBT – is all about.

This observation should cause us to ask ourselves whether we are putting in the effort to identify, assess, and correct performance weaknesses in our people. It should also encourage us to consider making appropriate performance of job tasks an assessable item in annual performance reviews.

The Deciding Factor Is Cost To the Business – But In What Form?

The most frequent concern I hear is the amount of time and resources it will take to implement and perform the performance assessments. 

It’s true that it takes longer to implement CBT because it requires defining and documenting performance standards and assessment criteria.  It will very likely require more time on the part of the trainees to come up to speed and be able to perform to the appropriate level. Therefore, training becomes more of an investment, and often, this cost is deemed unacceptable.

But think about the true costs to the business without CBT. If you can’t directly quantify these costs, at least think about them in terms of the organizational investment (people and time):

• How much time is spent retraining after a mistake is made? How frequently does it occur, and how many people are impacted each time?
• How much cleanup work (extra documentation, process delays, deviations, risk analyses, approvals, etc.) has occurred as a result of someone making a bad decision or taking an incorrect action?
• How often does someone fail to execute a procedure accurately, causing deviations, extra work, or process delays?
• How often to you find people in similar positions either operating to different standards (i.e. making two different decisions when presented with the same situation), or performing manual tasks differently, leading to inconsistency in the output of the operations they’re performing? What had to be done to correct those inconsistencies?
• Or worse, how much material and product have been scrapped due to problems that wouldn't have occurred had people’s task knowledge and performance been verified?
• How many observations have pointed to the training and qualification of your people – either directly, or through performance issues identified during an audit/inspection?

Simply considering the cost of implementing a CBT program on the Cost of Quality alone should make it very clear that it’s worth the investment.

Many companies are currently using, are in the process of instituting, or are starting to work towards CBT. Some of these companies are making the leap because of regulatory troubles, but others have chosen to do so because they recognize it provides business value. It is not always an easy sell, even though it is a best practice that will bring business benefit.

So, is CBT an industry standard? No, not yet.

But it should be. The sooner the better.