Practical Workbook - Vendor Evaluation in IS (Computerized Systems)

The latest regulatory texts highlight the crucial role of IT products and services suppliers in the regulatory compliance of information systems and the need for their preliminary and continuous assessment.

Faced with the complexity and the changes of the current IT landscape (infrastructure outsourcing, software provided in SaaS mode - Software as a Service- ...), how can we best and cheaply evaluate those suppliers whose quality standards are often far from our texts? Regulatory?

An approach based on the respect of the applicable regulations and favoring a risk-based evaluation approach makes it possible to identify a method applicable to all types of suppliers of products and / or IT services: software publisher, solutions integrator, company performing third-party application maintenance (TMA), hosting IT infrastructure or solutions, SaaS software providers, validation service provider ...

The regulatory context
Considering Annex 11 Good Manufacturing Practices (1) As the first repository on this subject, it specifies in its article 3.2 that "The competence and the reliability of a supplier are essential factors to take into account when selecting a product or a service provider. The need for an audit should be based on a risk assessment. "

Then in article 3.4: “Information relating to the quality system and auditing of software suppliers or developers and to the systems installed must be available, at the request of the inspectors of the body responsible for the assessment of GMP compliance.”

Finally, a little further on, in article 4.5, it is specified that: “Users governed by pharmaceutical regulations must take all reasonable measures to ensure that the information system has been developed in compliance with an appropriate quality management system. The supplier must be properly evaluated.”

The PIC / S document (PI011(2)), which is partly responsible for the revision of Annex 11, gives, in turn, the following elements:
"5.1: The assurance of the reliability of a supplier's software is attributable to the quality of the software engineering processes during the course of ... In order to customers confidence in the reliability of the products, they should evaluate the quality methodology of the supplier for the design, construction, supply and maintenance of the software ...

This guide also mentions the certifications available to a supplier without however any major emphasis:
“11.4: Confidence in the structural integrity may be based to some extent on the recognition of relevant certification of a company’s software and hardware development methodology and QMS to ISO 9001 standard, such as (for example) TickIT certification and utilisation of ISO 9000 related guidance.
11.5: However, an assessment of the supplier's QMS and Recognised certification alone is Unlikely to be the final arbiter for critical systems. The certification May very well be inadequate, gold Inappropriate. "

Additional reference documents (3) give interesting insights into the expectations of supplier evaluations:
“29. For vendor-supplied systems it is likely that much of the documentation created during the development is retained at the vendor’s site. In this case, evidence of formal assessment and/or vendor audits should be available at the test facility.
32. Suppliers need not in accordance to GLP règlements, must aim to operate a quality system Documented verified as acceptable by the quality assurance unit of the regulated user. The test facility shoulds-have proper documentation or if Regulate by contract documentation is Kept at the vendor's site. "

This document addresses a strong market trend, namely solutions hosting, and the associated risks:
“33. Hosted services (e.g. platform, software, archiving, backup or processes as a service) should be treated like any other third party service and require written agreements. It is the responsibility of the regulated user to evaluate the relevant service and to estimate risks to data integrity and data availability. The regulated user should be aware of potential risks resulting from the uncontrolled use of hosted services.”

Finally the risk assessment is privileged:
“36. Efforts in evaluating a service provider should be linked to the complexity and criticality of a system (e.g. a LIMS or any bespoke software provided from external sources might need greater attention). It is the regulated user’s responsibility to justify the type of the audit of a service provider or the omission of an audit, based on a risk assessment. An audit that covers technical as well as compliance issues requires the involvement of competent validation personnel (e.g. system owner and/or validation director) and quality assurance.”

Finally, a recent WHO(4) document on data integrity provides valuable guidelines on the contractual aspect and on the required qualifications of auditors:
"To fulfill this responsibility (for the integrity of all the results reported), ... outsourcing organizations should verify the adequacy of comparable systems at the contract acceptor and any significant authorized third parties used by the contract acceptor."
“The personnel who evaluate and periodically assess the competence of a contracted organization or service provider should have the appropriate background, qualifications, experience and training to assess data integrity governance systems and to detect validity issues. The assessment and frequency and approach to monitoring or periodically assessing the contract acceptor should be based upon documented risk assessment that includes an assessment of data processes.”
“The expected data integrity control strategies should be included in quality agreements and written contract and technical arrangements, as appropriate and applicable, between the contract giver and the contract acceptor.”

From these texts, it emerges that the assessment of all types of IT supplier is considered necessary but that in this assessment, account must be taken of the risk and complexity of the system that these suppliers may be involved in at different stages of its life cycle.
The contractual aspect is fundamental and also involves a risk assessment before contracts are signed. This generally takes the form of an audit or due diligence in the most critical cases (infrastructure outsourcing for example).

Assessment processes
Vendor assessment approaches are generally part of an external audit process that is available to most regulated industries; however, the industry standard GAMP 5(5) defines an evaluation procedure with variable dimension according to the level of the considered system:

Furthermore, the document recommends the use of a postal audit, ie sending a short questionnaire which, from the responses, assess the quality of the relevant supplier. This method, if it has the advantage of a lower cost, can not alone be sufficient in the case of a critical system because it is based only on verifiable supplier claims that a site audit.

The ISO 19011 standard(6) proposes a structured and rational approach to the evaluation of management systems in the broad sense; it integrates the audit approach into a process coordinated with the other quality processes of the company and insists particularly on the continuous improvement of the audit program, the necessary competence of the auditors and the assessment procedure based on criteria of audit objectives which must be the reference against which the audit evidence is assessed.

For a regulated business, this evaluation process must first be integrated into the entire life cycle of a computerized system for the solution search phase (Request For Proposal or RFP) to withdrawal ( "decommissioning") of the system.

As shown on the diagram below, the supplier assessment process can be performed at different phases of the life cycle of a system: the invitation to tender, which is generally associated with a specification, can be followed by an on-site audit; audits can be performed during system implementation particularly during the project phases, then on a production system in the event of malfunction; finally additional assessments on some items may prove necessary (in particular for solutions hosting companies) as part of periodic reviews.

So it can be seen that these multiple assessments are driven by different motivations and involve different approaches, but that they all aim to deepen knowledge of supplier practices so as to limit the risks to the systems and processes of the regulated customer.

In practice
One of the difficulties encountered in evaluating suppliers of products and / or IT services is the technical nature of certain services (IT development, infrastructure management, etc.), but also significant differences between the standards known to auditors of regulated companies. (BPx ...) and the standards adopted by these suppliers.

Indeed, these suppliers have several standards with variable geometry whose value relative to the activities carried out remains to be verified. These include:

• NF EN ISO 9001: Quality Management Systems - Requirements (2015 Version)
• FD Z67-910 December 1998: Engineering and Software Quality - Introduction to ISO / SPICE (ISO / IEC TR 15504) and its use for software process quality management
• NF ISO / IEC 2000x -1 June 2012: Information Technology - Service Management
• NF ISO / IEC 2700x January 2015: Information technology - Security techniques - Information security management systems

Finally, these suppliers also have ‘good practices’, some of which can include staff certification (in particular ITIL, PRINCE2) (7).

The approach recommended in this practical guide is therefore to take account of the applicable requirements whilst maintaining the requisite flexibility in assessing the different types of supplier present in the market, their respective levels of involvement in the life of a system, and finally the scale of the assessment, based on a preliminary risk analysis.

The proposed approach therefore proceeds through a progressive and iterative method that aims to progressively enrich the supplier's knowledge; in a first phase, generally when submitting a specification, a quality questionnaire is sent to the various bidders for a quick assessment of their capacity or level of maturity in the different processes for which the client has requested them: software development , support, accommodation ...

This questionnaire based on open questions, ie not indicating a choice or a particular orientation in the answer, allows the supplier to present with a minimum of effort and constraints, the characteristics of its service offer, of its quality system ... etc.

Once the supplier has filled in the questionnaire, it is reviewed and, depending on the answers given and the level of risk associated with the desired product/service, a decision to qualify or disqualify the supplier can be made. If additional information is necessary to validate some answers, it may be decided to hold an audit either on-site or remotely (by videoconferencing).

In the latter case, the process followed is similar to that of a standard audit with the supplier being sent a detailed agenda, specifying the objective, the scope, the personnel involved, the standards used and the chronological sequence of the audit.

The audit preparation involves creating a practical reference for the listener, including the formalization of audit argued criteria that will be the reference for any comments.

These observations must be subject to a criticality assessment in respect of the audit criteria; it is then possible to construct a quality profile of the supplier taking account of an evaluation of the criticality level of deviations, major and minor.

In most cases, a corrective action plan will be proposed by the customer and submitted to the supplier to be actioned. The monitoring of this action plan can be incorporated into the CAPA management of the regulated company or subcontracted to the service provider in charge of the audit.

Subject ID QuestionDocument desiredAnswer from supplier
ProductEL-01Describe the product that is the subject of the proposal: name, proposed version, version history
ProductEL-02Is the product standard (does not require configuration) configurable, modifiable by specific developments?
ProductEL-03Has the product been the property of the company since its creation or has it been integrated following the acquisition of another company?
Product EL-04How much of the turnover does the product represent in the overall turnover of the company?
Product EL-05How does this turnover break down into license revenue (sales, rentals), application maintenance, hosting, services, etc.
Product EL-06Is software development outsourced (outsourcing)?
ProductEL-07What is the typical periodicity for updating the application?
Life cycleEL-08Which general method corresponds to the development method used by the company (cycle V, agile, ...)?
Life cycleEL-09Is this method or life cycle documented? Description of life cycle

This both simple and practical approach enables the gradual development of an increased knowledge of the supplier by means of the various assessment phases to which the supplier will be subject throughout its relationship with the customer.

However, it is advisable to update this understanding regularly through periodic reviews and audits which also allow discussions to be held on the practices and trust necessary for a successful partnership.




Good Manufacturing Practices, Official Bulletin No2015 / 12 bis Special Issue http: //
(5) GAMP® 5: A Risk-Based Approach to Compliant GxP Computerized Systems -
February 2008
(6) ISO 19011: 2011 Guidelines for auditing management systems,
November 2015