By Herman F. Bozenhardt and Erich H. Bozenhardt
Our industry depends on a wide spectrum of equipment suppliers that provide our manufacturing systems, utility systems, and environmental systems. Our plants and R&D sites are an organized collection of these delicately interconnected (or not) systems that we depend on. We must be careful to not lose our perspective on what the deliverables should be in terms of a working/workable system. This can be driven by our naiveté and trust in the sales “cycle” that may or may not have disclosed some critical details. If you overlay the COVID-19 supply chain issues, priorities, and travel restrictions, we have reached a tipping point in equipment supply credibility that we all need to be aware of. The purpose of this article is to raise a number of points to discuss with equipment suppliers and your facility engineers to plan for and assure a successful and compliant installation.
Basics Of The Equipment Purchase
Each time we enter an agreement to buy a piece of equipment, we need a rigorous way of making sure we get what we paid for, the vendor delivers it, and it meets our expectations. We are often caught off guard by our own rush to order a “standard off the shelf” parts washer, autoclave, clean steam generator, etc. Even today, we must remind ourselves to produce several documents for our own protection and not just to satisfy those “validation folks”: The documentation “waterfall” is there to help us communicate with and reach an understanding with the supplier. Following up on the “standard” autoclave example, a vendor quote for a standard autoclave starts off straightforwardly, with chamber volume and dimensions, and then it gets into various facility requirements, automation, and process specific details. It is rare that two facilities and processes are exactly the same. The documentation series that defines our needs and deliverables should be as follows:
- User Requirements Specification (URS) – This document covers what the equipment does from a process and operator’s view, but it must indicate what it interfaces with electronically (BMS system and control system) and facility-wise (e.g., vents, water, compressed air, drains).
- Functional Requirements Specification (FRS) – This discusses how the URS will be met with mechanical and electronic systems in a narrative. This is critical for software specification of the vendor’s equipment and location of the various utilities (e.g., ceiling drop, floor penetration, chase routing, wall service panel, etc.).
- Detailed Design Specification (DDS) – This document covers exactly what is being provided and how it is designed and implemented. This is often provided by the equipment vendor in a laboriously long document and drawings. The key is examining the DDS for the install, start-up, and operational details.
- Factory Acceptance Testing (FAT) – This is probably the most key activity available and should never be waived or cancelled. The FAT should be attended by representatives of operations, engineering, and QA, who should carefully watch the testing and ask practical questions. During COVID-19, we have opted for the “virtual FAT,” where the vendor performs the nearly flawless series of tests at its factory while the owner watches via video (or not, between the phone calls and emails). The point of a FAT is not the test protocol but what is needed to make the protocol and process successful on your site. The equipment vendor’s objective is to get the FAT over with, get the equipment off the floor, ship it, and send you the invoice. The vendor is the expert in its equipment, but that doesn’t mean it is an expert in your process or needs. Review the FAT protocol for your needs and communicate what is needed to prove out the equipment operation as much as possible at the FAT. Any problems or concerns at the FAT must be corrected before shipment and never left to be “field engineered.” Once the item is shipped, the success and failure are now up the owner’s project manager and team.
- Site Acceptance Test (SAT) – This often the most confusing time in any project. This usually follows a late delivery, a difficult series of receipts and transports to the process area, and arguments from the utility installer. Typically, the vendor field engineer executes the SAT (similar protocol to the FAT) with operations and QA. This is where the purchasing company discovers that items glossed over in the FAT have become major operational issues. The field engineer has plausible deniability since he or she was not involved in the procurement cycle or FAT. SATs must be scheduled by the purchaser when convenient for the purchaser’s team post install, with all the correct vendor parties in attendance.
- Commissioning, Qualification, and Validation (CQV) – If done correctly, this will use data from the SAT directly and in confirmation with FAT software qualification. At this point in the project, all anomalies should be resolved; regardless, the purchaser now takes the full burden of responsibility. Warranties are nice, but time is never on the owner’s side and all projects are behind schedule by definition.
The following are some of the dangerous or expensive pitfalls buyers can encounter when purchasing equipment.
Find Out Where The Factory Is
In all industries, equipment suppliers are looking for ways to reduce their costs, increase profits, and increase sales. The pharmaceutical industry’s equipment suppliers are no different, especially if they perceive their product has become a commodity. Over the last several years, suppliers have shifted from their original “custom build at their time-tested factory” model to one that looks more like outsourcing. This involves moving a product from the original factory to a “low-cost center” and building the product from a collection of parts sourced globally. This may sound like a great cost advantage, but the low-cost center often does not perform an integrated check of the equipment and offers/provides a discount for no FAT and a “direct shipment.” This should be a danger signal alerting you to a prolonged start-up and having the vendor’s field engineer living at your site. Often, these low-cost centers do not have certifications for UL and may not be able to pass factory mutual (FM) inspections. These problems, remediations, and costs are left up to the purchaser during the start-up.
Schedule The Necessary Inspections
New facilities will undergo many inspections, which are key to getting a building certificate of occupancy (C of O). Your equipment will be directly at the center of those inspections, and you need to be ready for:
- Local mechanical inspection (HVAC and piping interface)
- Plumbing inspections (water and drains)
- Electrical inspection — be careful of breaker sizing. No electrical inspector in the U.S. will pass a facility with a piece of equipment without a UL certificate.
- The fire marshal will inspect only after the electrical is passed.
- The building inspector will inspect after the fire marshal is complete.
In today’s environment with COVID-19, the inspectors are moving quickly among many sites; some are making 20 to 30 visits per day. They don’t have time to negotiate or to listen to you explain why there is no UL certificate and the obscure one you have from elsewhere is just as good. They have standards and their government’s liability on their shoulders. Also, with COVID-19, they will refuse to inspect a site, building, or area if they don’t feel safe. That means you may need to clear the workers/trade personnel out of the area they are inspecting and be considerate of them when they arrive and ensure that any attending personnel are wearing masks and demonstrating safe practices.
Small Clients Can Be At A Disadvantage
To say the business is booming now is an understatement, but it has ramifications if you buy a small piece of equipment from a midsize to large company. A European equipment vendor actually told its client it could not execute the install/SAT and process study at the buyer’s site because it had limited U.S. resources and the client was too small to schedule in. This is a reality if you are buying a one-of-a-kind piece of equipment for a unique application. This vendor allowed the buyer to deduct the service from the final invoice. In this case, the purchaser needs to recognize their position and become their own expert quickly and execute the start-up with a phone call or two.
Watch Out For Inexperienced Suppliers
We are in the best of times when it comes to new technology and we have many new suppliers to work with. In another example, consider a recent purchase of a large piece of unique equipment from a non-U.S. source that is state of the art. What if the company is very young and nobody in it has ever worked with a purchaser before? What if the vast majority of the employees of the supplier have just joined the company in the last few months and this is their very first job? In this case, the SAT was fraught with software “bugs” (operational qualification and the supplier’s computer system validation had many discrepancies), communications contained misunderstandings, and there were equipment surprises due to a lack of expertise on what is needed in a pharmaceutical facility. A critical flaw was no UL certification, required by the building code in that state. A failure of the inspection and a lack of understanding of what UL means caused some friction. The lesson here is that you need to know what you are getting into with the supplier and realize you may have to supplement the supplier’s knowledge and be prepared to adapt the technology to a GMP environment (data acquisition, utility connections, equipment finishes, etc.).
Another reminder is that regardless how much you paid for the equipment, the vendor typically supervises the installation but does not perform the “heavy lifting” and connecting. A full complement of trades and riggers will be needed for this, and these contractors need have some familiarity with working in pharmaceutical facilities. That means the internet-based residential labor company may not be the best source of help for rigging/installing GMP equipment.
Don’t Overlook Start-up And Manual Modes
Prospective buyers often visit many websites to find “standard” equipment, then speak to the sales representatives to select exactly what they want. However, they often look at the final process implementation and neglect the start-up phase and “manual” modes. One piece of utility equipment was ordered assuming full automated operation, but the vendor electronically locked out the manual and circulation modes, thus delaying the CQV activity.
Avoid Cancellation Charges
Make sure during the contract development period your URS is the design basis of the deliverables and if your options or customizations can’t be met exactly you have an escape clause without cancellation charges. Often, equipment vendors will take an order and, upon clarification, if you decide to decline to place the order, they will charge you for “drawings and documents,” which are only their standard drawings with the owner’s name on it. Never pay for that.
Make Sure You Have A Customs Broker
If you are ordering anything manufactured outside the U.S., including an American piece of equipment made outside the U.S. (at the “low-cost center”), there is an entire bureaucracy that must be attended to. This involves the border crossing, NAFTA certification, customs declaration, customs clearance, and duty tax, in addition to the transport costs. This can be a major issue and could delay shipment of the equipment, especially if it arrives by ship into a major U.S. harbor like New York. Busy harbors without a clean tracking entity and timely payment to that entity can cause a shipment to be pushed to the “back of the stack.” At these ports, most equipment vendors stay out of the game, and it is incumbent upon the owner to control the entry and border crossing with special service providers (customs brokers, transportation brokers, etc.).
Ascertain The Vendor’s Capacity
In today’s pharma equipment market, most vendors are taking on more work than they can process, and their shops are full, with every square foot of space occupied. This means the buyer needs to be careful and determine the following:
- What is the delivery schedule contractually promised and will the vendor accept liquidated damages if they slip the schedule?
- What is their current capacity, and can they demonstrate how they will meet the FAT and ship dates?
- If their schedule is tight, how are they mitigating any potential delays — a second shift, obtaining more floor space, renting more machines or tools?
- If the project is an equipment upgrade and they indicate they could do the work on the owner’s manufacturing site, is it feasible? What do they need locally to support this effort from the owner’s side? This should be considered only if it is an easy effort, can be staffed locally, and there is space, power, and machinery to support the work.
When Should I Install The New Floor?
Installation, setup, and startup are such mechanically intensive efforts that, typically, floors are damaged and must be recoated or remediated following the install. So, save the money by having the concrete floor prepped before installation and coated (epoxy terrazzo) after all the heavy lifting and movement is done.
As we discussed above, all virtual checkouts are nearly flawless because the vendor controls the protocol, has all the answers, and has taken all the shortcuts you, the purchaser, can’t take — and you are not on-site to monitor the vendor’s actions. If you can possibly travel, go to the FAT. If not, the SAT becomes the main showcase of functionality.
Data Acquisition And Process Control Interface
All vendor equipment and their systems advertise they can communicate wirelessly and integrate with any standard plant system. The real question is how — via BacNet, Modbus, hardwired, magic? Make sure you include this in the URS and understand how it is implemented in the DDS before approval, because once the equipment is on your site, getting software support is beyond difficult.
In conclusion, acquiring a key piece of equipment is vital to your project’s success; however, you must run the gauntlet of the equipment suppliers. The greater transparency and communication (both ways), the less of a gauntlet it is. The keys are:
- Get your requirements in writing and have the vendor respond to the requirements.
- Control the requirements through all acceptances and approvals.
- Make sure the facility engineer is aware the equipment will be inspected by the local building inspectors, electrical inspectors, and fire marshal. Require that the UL certification of the entire unit (not parts and pieces) is on the unit and the entire system can pass FM inspection.
- Control the delivery to financial payouts.
- If necessary, have a customs broker and include them in the FAT/SAT timing correspondence.
- Include the SAT and final install punch list in the financial payout schedule.
- Make sure your facility is ready to accept the equipment for installation, including all utilities.
- All data communications and control system integration are demonstrated at the FAT.
- Attend the FAT and make a critical review.
About The Authors:
Herman Bozenhardt has 44 years of experience in pharmaceutical, biotechnology, and medical device manufacturing, engineering, and compliance. He is a recognized expert in the area of aseptic filling facilities and systems and has extensive experience in the manufacture of therapeutic biologicals and vaccines. His current consulting work focuses on the areas of aseptic systems, biological manufacturing, and automation/computer systems. He has a B.S. in chemical engineering and an M.S. in system engineering, both from the Polytechnic Institute of Brooklyn.
Erich Bozenhardt, PE, is the lead process engineer for regenerative medicine operations at United Therapeutics. He has 14 years of experience in the biotechnology and aseptic processing business and has led several biological manufacturing projects, including cell therapies, mammalian cell culture, and novel delivery systems. He has a B.S. in chemical engineering and an MBA, both from the University of Delaware.