Microsoft R Validation Project Summary

The Challenge

A medical device manufacturing client contracted Performance Validation (PV) to assist with creating a Computer System Validation (CSV) package for Microsoft R. Finding evidence of other organizations validating R for GXP compliance did not prove fruitful.  Microsoft R is a free, open source application which uses the S language in statistical methodology. (https://mran.microsoft.com/documents/what-is-r)  The software can handle very large data sets which was important along with functionality which allows the analysis to “learn” over time. Validation was needed since the calculations performed could improve patient care and reduce issues and failures with medical device operation.

The client was using R for a variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, predictive analysis, clustering, etc.) and graphical techniques. The client chose this software for its variety of proven packages and its ease and speed of processing. The client did not have a great deal of validation experience or manpower to write documentation, so PV was given a chance to help. The client requested PV write a Testing Plan, User Requirements, Functional Specifications, Installation Qualifications, Test Scripts, Traceability Matrix, and a Summary Report.

The following validation project considerations were identified for the client:

1. Validating an Open system, and locking down the exact packages of R being used.
2. Wanted to be able to add packages later as time passed.
3. Did not want to duplicate all the official testing already performed by Microsoft.
4. Creating specific calculations, verifying them, and locking them down
5. The software did not possess full electronic signature capabilities, and the audit trail just identified only time and date, not the user ID.
6. A low amount of validation experience from key members, help needed with most validation documents necessary.

The Solution

The PV team on this project leveraged years of experience in GxP compliance creating CSV packages for large scale software applications, and direct experience in compliance with the specific client’s procedures. This aided in the development of validation plans and consultation on the approach toward functional and user requirements.  PV worked on site, collaborating with the client’s team members to ensure successful completion of the project within the established timeline. The PV solution was comprised of the following:

1. To address the system being open, the source code was secured on a local server with access control given only to the client without updates allowed as a change control. User access was only given after official training and users were given a secured link to the exact installation approved by the client. Installation of new Microsoft R packages was disabled and data received from the devices was read-only protected.
2. To address adding packages later, PV created a detailed change control process/procedure for future packages. If a new package was desired it would be installed into a development instance of the system and evaluated by key team members. Then formal testing would be completed prior to being released into production, along with an update to the installation qualification.
3. Following Good Automated Manufacturing Process (GAMP5), the software would fall into the category of 5 for highest risk since the software is customizable. This would mean all the code needs to be validated along with all full life cycle requirements. But Microsoft has run extensive unit testing on the packages available for download.  There was a desire to leverage this testing instead of repeating it.  This led to the decision of breaking the effort into two separate validation plans along with different requirements for each.  The custom code would fall under Category 5 and receive full testing from an outside research company with functional specifications.  The remaining, fully unit tested, packages would fall under Category 4.  This reduced the validation effort to the configurations, and Graphical User Interface from a User Requirements Specification.
4. An independent (Data Science and Predictive Analytics) company was hired to review the Functional Specifications and newly created algorithms (Category 5 portion of GAMP) and compare the results to other statistical software, then sign off that the R code was scientifically valid and accurate.
5. A procedure was developed to detail creating a manual audit trail during production use, and enforcing the user to add their username/date/time as electronic signatures to finished data
6. PV helped draft the following documents using the clients approved forms and formatting preferences:
   1. An installation qualification was created to detail the server structure, minimum installation specifications were met, and verify security.
   2. Test Scripts were written and executed for the intended use of system, the Graphical User Interface (GUI), and the Functional specifications
   3. Traceability Matrix was generated to link the requirements to their respective test scripts.
   4. A Validation Summary Report was generated explaining all the validation deliverables and how the clients’ documentation satisfied regulatory requirements.

The Results

The validation was completed ahead of schedule, meeting the customer’s overall system implementation deadlines.  The testing results were examined/verified by an independent reviewer and all discrepancies discovered were addressed. The business process and related regulatory requirements were equally emphasized and consistently met the client’s needs.  PV provided regular project status updates which provided details on the project deliverables, project budget, and risks to the project and schedule.

 The Benefits

This validation project gave the client a tool they could use to help determine probable service intervals minimize patient risk. They can rest assured their version of Microsoft R does not get automatically updated/changed.  If they need to add another package, a procedure is now in place to guide them, or they can contact PV to support their needs.

The PV Advantage

Performance Validation provided the client validation project planning and execution for their stakeholder’s needs, and the best approach for validating an open source configurable application.  The validation documentation focused on properly addressing regulations and internal procedures while working with Statistics experts, Project management, Information Technology, Quality Assurance, Subject Matter Expert, and the Application owner to ensure the success of the project.

Performance Validation was able to tackle a complex validation by offering suggestions, working with multiple divisions of the team, and adhering to a clients’ detailed documentation structure. PV was able to work on core documents while waiting for windows of time with the client’s Subject Matter Experts to maintain a constant rate of progress. Implementation of the validated software system allows the corporation to rely on Microsoft R for providing predictive service analysis.

For more information please, schedule a call with one of our subject matter experts. 

Kevin Marcial,
CSV Services Manager