Challenges when using Vendor Protocols

Background:  

Often when contemplating a GMP application software project it is desirable to use the software vendor’s protocols to minimize cost.  It would be natural to assume that the vendor’s protocols should provide a cost savings, as the vendor’s protocols had previously been executed as part of the testing to release the software for commercial sale.  Or, alternatively were used on previous commissioning / validation projects.  This was the expectation at one recent project; however, the reality of the situation was quite different.

End-user Reality: 

On a recent project, testing protocols were purchased from the software vendor for a validation of an application to be used in a GMP environment.  These protocols were the vendor’s standard test package for the system.  The customer (End-User) executed the protocols; however, they experienced a 100% failure rate.  Examples included:

• Lack of detail in the test setup.
• Missing software functions and features.
• Lack of requirements to capture objective evidence.
• Test scripts that could not be followed by a competent qualified user of the system.
• Significant time required for test error resolution between the end-user and the vendor.

The 100% failure rate created questions in the end-users mind as to the quality of the vendor testing and test protocols.  The high failure rate negatively affected both project budget and schedule to launch a validated application for GMP use.

Investigation and Solution: 

The Performance Validation CSV team was brought in to assist the end-user in resolving these problems.  However, was the issue an application problem or a test protocol problem?  Performance Validation executed dry runs of the vendor’s testing protocols and identified that:

• The test protocols were originally written by the vendor and were written for a high-level user (i.e. someone with a significant implicit knowledge of how the application worked) thus the test protocols did not contain adequate detail in test setup (explicit) information to allow a competent end-user to execute the tests and obtain replicate vendor test results.
• The test protocols were written assuming full functionality of the application; however, the end-user did not require full functionality and was only implementing select portions of the application.  The selection of required testing to demonstrate suitability for use complicated testing and the testing setup.
• The software was configured for automatic updates.  This created a problem, as the software was receiving updates periodically without the test scripts being reviewed, so menu names and windows had been changed and the “Expected Results” of the test scripts were no longer correct, and were not met during execution.
• The individual test scripts did not require collection of objective evidence.

The Results & Benefits:  

While the use of vendor provided testing protocols may provide an opportunity for considerable cost savings it is important to understand the difference in the intended use and of the rigor required for GMP testing versus product release testing.  It is a GMP expectation that testing is repeatable, and that testing can provide reproducible results independent of the individual performing the testing (i.e., any qualified individual should be able to complete the test and obtain the same result).  Additionally, GMP testing is expected to capture objective evidence required to demonstrate satisfactory completion.

Factors that end-users should consider when contemplating use of Vendor test documents:

• Do the test protocols accurately capture the explicit and implicit elements necessary to complete the test?
• Do the test scripts accurately reflect the intended use of the software application (i.e., as purchased, as configured, as used)?
• Do the test scripts require the capture of objective evidence (where appropriate), which demonstrates the satisfactory completion of the test step?
• How will minor/major or application updates be factored into the testing program?

The process of identifying and resolving testing errors is a significant cost and risk to any Computer System Validation project. Because of the identified issues, the end user spent time and money dry running, sending comments back to the vendor and waiting for updated protocols to be returned and that could have been avoided.  Had the testing been maintained and updated after system updates as well as test setup being detailed enough for a less experienced customer, the test protocol execution would have been much smoother and more efficient.

For more information, please, schedule a call with one of our subject matter experts.

Kevin Marcial
Sr. Validation Engineer