Key Considerations in Outsourced “On-Site” Audits as Part of Supplier Qualification

donskarpo/shutterstock.com

Audits of suppliers and service providers are becoming more important in a globalized world. Well-trained auditors will be able to detect potential weak points of the firms that they audit, before such weak points lead to quality problems in routine business with severe consequences (e.g., significant additional costs). 

Thus, supplier audits and qualification have become one of the most important elements in quality assurance and are in the focus of authority inspections. The British Medicines and Healthcare Product Regulatory Agency (MHRA) has published an overview of the most important GMP deviations within a time span of 12 months, and “poor supplier qualification” was ranked sixth in this overview (1). 

In principle, a pharmaceutical manufacturer may have an audit as part of supplier qualification, executed by its own employees; or can authorize a suitable third-party audit provider to perform the audit on its behalf; or can buy an audit report from a commercial provider, for example, the Asociación Forum Auditorías (AFA). 

The main focus of interest lies on the final audit report, regardless of whether the audit has been performed by the pharmaceutical manufacturer itself, by a third party, or whether the report has been purchased from a provider. In any case, the report should be written informatively and distinctly, and should enable the reader to gain a comprehensive impression of the audited firm. This article reports some experiences with the outcome of in-house audits, audits by third parties, and purchased audit reports.

Materials and methods

Within a period of three years, 2012-2015, a total of 126 reports from audits executed by company employees, by third-party audit providers, by “others,” and reports from shared audits bought from commercial providers have been assessed. For the assessment, first a “scale” was issued (see Table I), based on Kettelhoit’s description of the desired format and content of an audit report (2). 

Table I. Audit report assessment as key performance indicator.

The number of achieved points per audit report was divided by the maximum number of achievable points, and the resulting quotient was then multiplied by 100 to achieve the percentage rate, which stands for the report quality. When the final percentage rate was below 50%, the report was regarded to have “room for optimization.” A report was considered “acceptable” if it scored up to 65% and “satisfactory” for scores of up to 80%. In cases of more than 80%, the report was deemed to be very good.  Table II indicates the number of audit reports in each category.

Table II. Quality of audit reports by auditors.

The quantity and quality of the issues (observations) that have been made is another criterion to be considered when deciding whether to share or to buy an audit report rather than executing an audit. For classification, four grades were used: “critical” (very serious observation), “major” (not critical, but serious), “minor” (observation could become a problem if not corrected [in a timely manner]), and “recommendations” (potential improvement). 

Table III shows that there were only a few “critical” observations in this assessment. (In the case of “others,” the relatively high rate of “critical” issues was regarded to be an outlier due to the fact that only very few audit reports from “others” were available, thus one or two “critical” issues within a small total number of issues resulted in a high percentage of critical issues). Approximately 20% were serious or “major” observations, and most observations were classified “minor” or “recommendations.”

Table III. Classifications of issues observed during audits.

As can be seen in Table III, the observed differences of the overall results are not striking. The quantity and quality of observations made are comparable and do not differ too much from each other, with some exceptions in the case of “other” audits, which had a relatively high percentage of “critical” observations, a percentage of “major” observations above the mean value (+ one standard deviation), and only a few recommendations in comparison to the other audit reports; these were considered outliers as described perviously. 

To have another parameter that might show differences in audit report qualities, the mean values of the individual report qualities (percentage rates) were compared with each other. In Table II, the summary of the individual qualities was presented. The respective qualities of the individual reports were summarized and the mean values were calculated and differentiated between “third party,” “shared,” “own audits,” and “others.” The resulting mean values were 73.9%, 71.8%, 79.6%, and 68.1% respectively. The differences in mean audit qualities were not striking, but to see whether the differences measured were statistically relevant, the “t-test” for comparison of mean values (4) was applied (normal distribution of the individual values pre-conditioned). 

As can be seen in Table IV, reports from audits performed by the company’s own auditors turned out to have significantly higher report qualities in all cases, whereas the report qualities from third-party audit providers, shared audits, and others were comparable with each other (no significance difference in any case).

Table IV. Differences in audit report qualities (in brackets: Mean quality of audit reports [%])

Discussion of results

The supply chain is increasingly becomimg an area of interest in authority inspections (1, 5). Supply-chain integrity also includes supplier qualification. An on-site audit is only one but nonetheless an important test of a supplier’s qualification to manage supply-chain control effectively (6). Lack of transparency and the corresponding uncertainty along the supply chain might cause severe quality risks. Often, measures that should assure quality are not sufficiently known, such as the meaning and validity of foreign certificates, quality assurance agreements, analytical controls, and on-site auditing. More suppliers refuse an individual on-site audit by the customer because of several reasons, such as “…the quantities announced do not interest us...”, “...considering that the business for which we will be audited for by you does not proceed…”, “…we have invested a lot of money in our certificates…”, among others. When in cases where an on-site audit by the customer is denied by the supplier, the customer can try to instruct a third-party audit provider to perform the audit on its behalf, or can buy an audit report from a commercial service provider who has perfomed an audit on behalf of more than one contract giver (i.e., a shared audit). 

In the end, a company has to integrate a purchased audit report into its own quality system, and this integration preconditions some precautions (7), as the risk to collect a surrogate (i.e., an audit report of little quality) of an audit report should be avoided by all means.

There are various preconditions that have to be laid down proactively before integrating an external audit report, either purchased or issued by a third party, into a company’s quality management system (7). The auditor who issues the report has to avoid conflicts of interest, be independent, and be qualified to execute the audit. The report issued by the auditor must contain, beside formal aspects, sufficient information of the product-specific processes and workflows. On the basis of such information, the contract giver can decide whether the supplier is sufficiently qualified or not. Moreover, it should be kept in mind that a report of an on-site audit is only one part of the entire supplier qualification. 

To guarantee that the audit report (either issued by the company’s own auditiors, or provided by other sources) meets the qualitative requirements necessary to assess whether the supplier is sufficiently qualified, a scale (Table I) to assess report quality was issued. 

It can be seen from Table IV that the quality of reports from audits being performed by own employees was significantly better than the quality of reports being issued by third parties or being purchased from shared audits through an external audit provider (“other audits” that did not fit in either group mentioned will be left out of the discussion because the number of audits/audit reports from “others” was too small). The differences in quality might be because the scale issued was adapted to the report template for audits being performed by own auditors. To raise the calculable quality of the reports being issued by a third party, a company might consider providing the third party with its own report template; by doing so, the third-party auditor will be more inclined to fill in the template according to the format is expected. As can be derived from Figure 1, “time aspects” are not being met by auditors others than own auditors. 

Figure 1: Differences in quality aspects (audit reports from different sources). (Figure courtesy of authors)

In case of reports from audits performed by others instead of one’s own auditors, the company has to accept the format and content of the report as it is, because there might be little possibility to gain influence. 

It can be concluded that on-site audits are an integral part of supplier qualification and management (1). Given that there might be several restrictions preventing a company’s auditors from perform their own on-site audits (e.g., expert required, cultural restrictions, personnel shortages), one has to consider obtaining audit reports from other sources-whether it’s buying audit reports from a commercial service provider, sharing audits with other companies, or using a third-party audit provider to execute an on-site audit on behalf of the company. Table V lists some examples of criteria to consider when deciding what type of audit to use.

Table V. Decision matrix: Own audits versus audits by third parties versus shared audit/reports purchased*.

One has to ascertain that audits performed by other auditors are qualitatively comparable with the company’s own audits. This practice is also a requirement of official normative guidelines (8) and might be assured by, among other things, administratively integrating a commercial provider for the execution of on-site audits into one’s own quality assurance (QA) system. For example, one might request that the third-party audit provider is certified according to DIN/EN/ISO 17020 (9) or ISO 19010:2010 (10). When being certified according to an international standard, one can have the assurance that the audits performed by such a certified body follow a standardized procedure. 

If a company fails to integrate an audit provider into its own quality system, the company might be in danger of receiving surrogates rather than valuable audit reports that will not be accepted in case of an authority inspection. It is important that requirements that are to be followed by a commercial provider of on-site audits are agreed upon prior to an audit. By doing so, it is more likely to satisfy authorities during inspections that external audits are well under control.

Conclusion

On-site audits play an important role in supplier qualification. Suppliers, however, are sometimes not willing to host on-site audits by their customers for various reasons. In such cases, a customer can either:

• Buy reports from audits being executed at the suppliers’ site, 

• if available. 

• Authorize a commercial audit provider to conduct the required audit on its own behalf.

• Perform audits with other customers of the same supplier who will then share the issued report. 

If the second option is preferred, one has to carefully select the audit provider and integrate the provider administratively into one’s own quality management system. Finally, it has to be assessed by the contract giver whether the quality of the audit being executed by a commercial audit provider is comparable to audits being performed by its own auditors. Quality assessment will be based on the overall quality of the issued report in percent (Table II), the quantity and quality of observations made (Table III), and the statistical comparison of the achieved data (Table IV). The outcome of these assessment and the consequences resulting from it can be seen in Table V. When such conclusions from these assessments are laid down proactively, authority inspectors might be convinced that the supplier qualification process is in order, even if parts of the qualification process have been outsourced.

References

1. MHRA, Deficiency Data Review, April 2011-March 2012.
2. S. Kettelhoit, Pharm. Ind. 72 (2) 242-247 (2010).
3. European Commission, GMP Guide: EudraLex-Volume 4 (Human and Veterinary)-Good Manufacturing Practice (GMP) guidelines, https://ec.europa.eu/health/documents/eudralex/vol-4_en, accessed 12 Dec. 2016.
4. L. Cavalli-Sforza, Biometrie - Grundzüge biologisch-medizinischer Statistik, Gustav Fischer Verlag, Stuttgart (1974).
5. A. Shanley, Pharm. Tech. Eur., 28 (7) 25-27 (2016).
6. S. Barr, Staying Ahead in OutsourcingMeeting the Changing Needs of the Pharma Industry, CheManager Fine and Specialty Chemicals, p. 28-29, July/August 2016. 
7. GMP News, 1 Feb. 2016, www.gmp-navigator.com/dnews_05166_Welche-externen-GMP-Auditberichte-darf-man-verwenden.html, accessed 20 Dec. 2016. 
8. Arzneimittel und Wirkstoffherstellungs-verordnung of 3 Nov. 2006, BGBl, Part I, p. 2523, version of 28 Oct. 2014.
9. DIN/EN/ISO 17020: Conformity AssessmentRequirements for the Operation of Various Types of Bodies Performing Inspections (ISO/IEC 17020:2012), Beuth Verlag, Berlin.
10. ISO 19010:2010, Guidelines for Auditing Management Systems, 
ISO copyright office, Case Postale 56, CH-210 Geneve 20. 

Article Details

Pharmaceutical Technology Europe
Outsourcing Resources 2017 Supplement
Pages: s6-s11

Citation

When referring to this article, please cite it as M. Pfeiffer, "Key Considerations in Outsourced “On-Site” Audits as Part of Supplier Qualification," Pharmaceutical Technology Europe, Outsourcing Resources 2017