Evaluation of data quality in a laboratory-based surveillance of M. tuberculosis drug resistance and impact on the prevalence of resistance: France, 2004

Summary

In France, surveillance of anti-tuberculosis drug resistance is performed by the Azay-Mycobacteria network, representing 30% of all culture-positive cases. We sought to validate administrative and clinical data gathered by the network in 2004 and to produce corrected resistance rates accounting for the observed misclassification. We reviewed a 10% sample of patients' records diagnosed in 2004 and measured the agreement between controlled data and data collected by the network by using the kappa (κ) statistic. A re-sampling bootstrap-based method was used to investigate the impact of bias found on resistance rates. Most of data collected by the network, such as demographic data, and country of birth had an excellent agreement (κ>0·8) with controlled data. The concordance was good (κ>0·6) for HIV status and tuberculosis site. The only variable slightly discordant with controlled data was prior history of treatment (κ=0·52). However, after correcting crude resistance rates for the observed misclassification, all estimated rates were within confidence intervals based on reported rates. This validation study is in favour of a good quality of data produced by the network, even though corrected rates are slightly different from observed rates. Therefore, data collected through the network may be used for policy making and tuberculosis programme evaluation. However, improvement in data collection regarding prior history of treatment should be considered.

INTRODUCTION

The overall prevalence of tuberculosis (TB) in France is low (9/100 000 inhabitants), but the prevalence in French-born cases (52%) is about 5/100 000, while it is 42/100 000 in foreign-born cases. In France, surveillance of TB is the responsibility of health authorities. It is based on a national mandatory notification system (NMNS) where all suspected TB cases treated with anti-TB drugs must be notified, whatever the culture results. Data on drug susceptibility of Mycobacterium tuberculosis have been included only recently in data collected by the NMNS. Therefore, in order to collect data on drug susceptibility, a sentinel network, the Azay-Mycobacteria network, was set up in 1994 by microbiologists of university hospitals in coordination with the National Reference Centre for Mycobacteria and anti-TB drug resistance (NRC). The network was created on a voluntary basis and is not linked to the NMNS. It collects data on all culture-positive TB cases diagnosed in the participating hospital following WHO recommendations [1]. Data are analysed by the NRC and sent to EuroTB and the WHO [2, 3]. The main objective of the network is to accurately describe the annual prevalence and trends of M. tuberculosis drug resistance to first-line anti-TB drugs in new cases (‘primary resistance’) and previously treated cases (‘secondary resistance’) and to stratify results by country of birth [1]. Presently, the network includes 32 laboratories and covers 19 of the 22 regions of metropolitan France.

In regard to drug resistance surveillance, professionals focus almost exclusively on quality of susceptibility tests results and, usually, external and internal quality controls of these tests are regularly organized by the networks. In the Azay network, blind external quality control is performed every other year and the NRC participates in the annual proficiency testing of the susceptibility of M. tuberculosis organized by the WHO. Results of these controls are satisfactory with regard to WHO standards. Besides data on drug resistance, microbiologists collect many other variables including clinical data. These variables are mainly used for stratified analysis of resistance rates, but can also provide useful insights in characteristics of culture-positive TB in a country. The quality of such data is rarely evaluated. In a previous study conducted in Paris area, we compared data on patients' characteristics collected through the Azay network and those collected through the NMNS in 2001–2002 [4]. We demonstrated that the completeness of the network was very satisfactory (96%) and that concordance was good for a majority of data collected by the two sources, although only acceptable when considering the site of TB.

In order to evaluate the reliability of a laboratory-based surveillance of TB and to complete the findings of our first regional evaluation, we conducted a second reliability sub-study on a national scale representative of the culture-positive patients reported by the network by reviewing a sample of data collected in 2004. Furthermore, we corrected resistance rates accounting for the misclassifications observed in the reliability sub-study.

METHODS

The reliability sub-study was carried out on culture-positive TB patients diagnosed in 2004 in the 32 laboratories of the network disseminated throughout France. We measured the agreement between data collected by the microbiologists of the network and data collected by clinicians and available in patients' administrative and medical records.

Because the Azay 2004 database was not complete at the time of the study design, we relied on 2003 data to determine the number of patients to be randomly sampled. A 10% sample of all 2004 TB cases notified by the network, i.e. a total of 175 TB cases was considered satisfactory to obtain accurate estimates of concordance values (typically a standard deviation of 0·2). We used the weighted proportionate cluster sampling method to draw patients from the 32 laboratories [5]. The number of patients in a cluster was determined as six, i.e. 175/32 laboratories. To anticipate medical record losses, two additional patients were added to each cluster for replacement data, resulting in a list of 239 patients out of 1728 culture-positive TB cases. Because the Azay database is anonymous, microbiologists at each centre retrieved the identity of randomly selected patients by using hospital name, initials, sex and age of each patient available in the Azay database. After patient identification, microbiologists or the NRC staff collected information available in patients' medical records by using a standardized form including all variables routinely collected by Azay, i.e. age (in years), sex, place of birth, prior history of treatment by anti-TB drugs, smear results, site of TB, and HIV status.

Data were computerized and analysed by using Stata 8.2 software (StataCorp, College Station, TX, USA). The Mann–Whitney test was used to compare age before being categorized; a discrepancy interval of 1 year in age was seen as no difference between Azay and medical record data. The kappa coefficient (κ) was used to assess agreement between data reported by the two sources [6, 7]. P values are two-tailed and P<0·05 was considered statistically significant.

Estimated prevalence of resistance to anti-TB drugs was computed from the prevalence observed in 2004 in the network for all 1728 patients by using resampling bootstrap-based methods, accounting for uncertainties in demographic characteristics of patients. Specifically, the probability that a patient would actually be in one of the treatment-history categories (previously treated, untreated, or unknown history) or in one of the origin-of-birth categories (French-born, foreign-born, unknown origin), could be estimated from the sample of patients used for quality control, given the previously recorded treatment history or patient's origin of birth. Then, using a random number generator, we could reallocate in a probabilistic way the treatment history and place of birth of each patient according to these probabilities. From the set of 1728 patients (i.e. the size of the original set), a bootstrap sample of the same size was built through random sampling with replacement according to standard bootstrap techniques [8], and prevalence of resistance was estimated. This procedure was repeated 200 times and estimated means and 95% predictive intervals for prevalence of resistance were derived. The basis for this method was that reallocation of patients' characteristics allowed accounting for the impact of erroneous demographic individual data on prevalence estimation and variance, while secondary resampling allowed taking into account the sample size on the variance of the estimates. Both sources of uncertainty could then be taken into account within a unique theoretical framework. Confidence intervals of reported prevalence rates are asymptotic or, whenever necessary, exact intervals.

RESULTS

The sampling method resulted in selecting patients in 21 of the 32 laboratories. A total of 210 cases were included and the mean number of medical records for each laboratory was 10 (range 6–24). To assess the quality of sampling, we compared the distribution of the characteristics recorded in the Azay database for the 210 randomly selected patients to those of the whole Azay population recorded in 2004 (1728 cases). The mean age (47·7 years vs. 47·5 years, respectively), the proportions of men (66% vs. 64%), previous treatment history (9·0% vs. 7·9%), and foreign birth (50% vs. 53%) were not statistically different between both populations, suggesting a good quality of the sampling process.

When reviewing the 210 medical records, 137 (65%) patients were male, and the mean age was 47·8 years (range 0–101 years). Three males and four females had been classified in the opposite gender by the other network. The percentage of agreement (percentage of concordant data over total data) for sex between both databases was 96·7% and the κ-coefficient displayed excellent concordance (κ=0·93, Table 1). The concordance for age as a continuous variable was very good (κ=0·87), the difference between ages of the patients in both databases being ⩽6 years for all patients, with one exception (46 years for a newborn in Azay).

Table 1. Percentage of agreement, κ-coefficient and concordance of characteristics for 210 randomly selected tuberculosis cases out of 1728 patients reported by Azay laboratories surveillance network in 2004

HIV, Human immunodeficiency virus.

* Excluding three patients with missing values in control data.

According to the medical records, 83 (39·5%) cases were French-born, 119 (56·7%) foreign-born, and the country of birth was unknown for the eight (3·8%) remaining cases. The percentage of agreement regarding patients' origin with the Azay database was 90·0% (κ=0·82). It was slightly lower when region of birth was used for stratification (κ=0·79). A major discordance (French-born for foreign-born, or vice versa) was observed for five (2·4%) of the 210 patients, and a minor discordance (unknown for French-born or foreign-born) was observed for 16 (7·6%) patients. Two patients born in North Africa and three in sub-Saharan Africa according to the medical records were reported as French-born by Azay, and 13 of unknown origin according to Azay were classified as French-born (n=2) and foreign-born (n=11) by the medical records.

A total of 18 patients (8·6%) were previously treated by anti-TB drugs, 179 (85·2%) were never treated, and 13 (6·2%) had unknown history of treatment according to the medical records. In contrast in Azay, 19 (9·0%) were initially previously treated, 173 (82·4%) were new patients, and 18 (8·6%) had unknown history. A major discordance (treated for untreated, or conversely, untreated for treated) was observed for 12 (5·7%) of the 210 patients, and a minor discordance (unknown for treated or untreated) for 17 (8·1%), a majority (n=9) being considered as never treated according to medical records review but recorded as unknown in the Azay database. The percentage of agreement was 86·2% (computed κ=0·52).

The smear result was positive for 92/210 patients (43·8%) according to both databases, and negative for 117 (55·7%) patients in medical records, and 116 (55·2%) in Azay, resulting in 91·9% agreement (κ=0·84).

The HIV status was positive in 21 (10%) patients, negative in 142 (67·6%) and undefined in 47 (22·4%) according to medical records. These numbers were 17 (8·1%), 131 (62·4%) and 62 (29·5%) in the Azay database, respectively. In the latter 62, four were in fact HIV-positive and 17 HIV-negative according to medical records. There was no major discordance, i.e. quoted negative for positive, or vice versa. The percentage of agreement was 87·1% (κ=0·75).

The clinical site of TB was pulmonary for 127 (60·5%) patients, extrapulmonary for 45 (21·4%), and combined (pulmonary and extrapulmonary) for 37 (17·6%) according to medical records. These numbers were 142 (67·6%), 43 (20·5) and 24 (11·4%) in the Azay database, respectively. Major discordance (pulmonary for extrapulmonary disease or vice versa) was observed in nine (4·3%) cases, and minor discordance (combined disease for pulmonary or extrapulmonary only) in 17 (8·1%) cases. The percentage of agreement was 86·7% (raw κ=0·75).

Data on M. tuberculosis drug resistance in 2004 were available for 1709 of the 1728 cases diagnosed by Azay laboratories (Table 2). Reported prevalence of resistance to at least one of the first-line drug was 9·0%. As expected, resistance rates were higher in previously treated cases and foreign-born patients. For instance, primary resistance to isoniazid was 1·8% in French-born patients, and 6·0% in foreign-born patients (P<0·01). Although the rifampicin resistance rate in foreign-born patients was almost double that in French-born patients (1·4% vs. 0·8%, respectively), the difference was not statistically significant (P=0·45). When comparing secondary resistance rates in foreign-born patients to those observed in French-born patients, they were twofold higher for isoniazid (23·9% vs. 12·7%, P=0·16), and fourfold higher for rifampicin (14·9% vs. 3·6%, P=0·06), respectively. These observed resistance rates were ‘corrected’ to account for misclassification in treatment history and place of birth by using data controlled in medical records as ‘gold standard’. Estimated resistance rates are presented in Table 2 with predictive intervals. In new cases, the estimated prevalence of resistance to any drug was slightly higher than observed rates in French-born patients (8·1% vs. 6·7%, respectively), and foreign-born patients (10·8% vs. 9·2%, respectively), but estimated rates were within the confidence intervals based on reported rates. Estimated resistance rates to isoniazid and rifampicin were exactly similar to reported rates, although confidence intervals were slightly wider. In previously treated patients, estimated resistance rates were slightly lower than observed rates for all drugs, but as for new cases, estimated rates were within observed ranges.

Table 2. Reported and estimated prevalence of resistance to first line drugs in France in 2004 in new, and previously treated cases* by country of birth†

CI, Confidence interval; PI predictive interval for the prevalence of resistance; INH, isoniazid; RMP, rifampicin; EMB, ethambutol; SM, streptomycin.

* Treatment history unknown for 132 cases.

† Country of birth unknown for 113 cases.

DISCUSSION

Evaluation of a surveillance system in order to improve quality and usefulness of the surveillance is highly recommended [9]. However, it is rarely performed, except for quality control of drug susceptibility testing of antimicrobials, but results of validation studies are seldom taken into account to quantify uncertainty of raw results. We focused our study on the reliability of demographic and clinical data collected by the Azay-Mycobacteria network, and especially those data related to M. tuberculosis drug resistance [1]. We showed that all data reported by the network are highly concordant with those found in medical records, with the exception of prior treatment history. However, the impact of the latter misclassification on drug resistance rates was moderate.

There was a good correlation for demographic data, and site of TB between both sources. We showed in a previous study that there was a slight tendency for Azay laboratories to report pulmonary involvement more frequently than clinicians [4]. At that time, we hypothesized that there may be confusion between the type of specimen and the site of the disease. Current results are more satisfactory and may result from intervention after our first evaluation study.

The major risk factor for drug resistance is a previous history of treatment. Such history is difficult to collect and answers may vary according to many variables. In our study, agreement between Azay data and medical records was reasonable (κ=0·52). Usually, microbiologists collect data through clinicians in charge of patients. Therefore, discordant results may occur because information initially transmitted to microbiologists may not be reliable or unavailable and is corrected afterwards. Microbiologists may not be aware of such corrections. To lower misclassification, microbiologists may re-collect information at the time they communicate drug susceptibility results to clinicians. If this is not possible, it may be useful for microbiologists to abstract information on prior treatment in a sample of patients' medical records, especially in case of drug resistance.

The second well-known risk factor for anti-TB drug resistance is the place of birth. According to our study, misclassification was mainly due to patients being considered as French-born while they were in fact foreign-born. In a country where TB is due in a large proportion to immigrants, there may be confusion between nationality and place of birth. In that case, rates observed in so-called French-born patients may be over-estimated because of higher resistance rates in most countries of origin of foreign-born patients.

To our knowledge, we present the first attempt to take into account misclassification when computing stratified estimates of resistance rates to anti-TB drugs. Probabilistic corrections have been used to quantitatively assess the bias and uncertainty introduced by classification errors [10]. In new cases, adjusted rates are very similar to crude rates observed by Azay, especially for resistance to isoniazid or rifampicin, indicating a very limited impact of the misclassification. Therefore, public health policies or management of new patients is not altered by misclassification, either for French-born, or foreign-born patients. For example, it is acceptable to omit ethambutol in the intensive phase of the recommended standardized treatment regimen for HIV-negative new cases without cavitary pulmonary TB when isoniazid primary resistance is lower than 5% [11]. According to our result, this is the case for French-born new cases, and the predictive interval for the isoniazid resistance rate is far below 5%. In previously treated cases, adjusted rates are slightly but constantly lower than crude rates, but always included in observed confidence intervals. Therefore, the conclusion drawn from the adjusted rates does not differ from those drawn from crude rates interpreted with their confidence interval. A similar methodology may be conducted in other networks of surveillance of drug resistance, when resistance rates are stratified on well known risk factors. In addition, the NMNS should become a leading source of information on TB drug resistance in the near future. It will be of major interest to compare data collected by both sources to evaluate misclassification and correct observed rates. However, it should be borne in mind that errors may occur at different levels in a network for surveillance of drug resistance. First of all, we did not take into account errors due to drug susceptibility testing. Most guidelines recommend the organization of external quality controls [12] but none give recommendations on methods to be used to take measurements errors into account. Second, errors may occur while abstracting data from medical records [13] or other sources, e.g. administrative databases. Because it has been shown that data gathered prospectively are more accurate than information obtained retrospectively [14], prospective data collection, as in our network, should be encouraged. Finally and as in our study, statistical modelling could be used to account for measurement errors or abstractor agreement after the validation sub-study [10]. Despite limitations, these modelling methods have the advantage of providing a quantitative estimate of the uncertainty of the main outcome that may be more satisfactory than the measure of agreement. However, they have the disadvantage of requiring statistical expertise and sub-studies that may not be easily repeated over the years.

In conclusion, our results suggest a good quality of data produced by the network. However, it is of major importance to draw the attention of microbiologists and clinicians to the patients' treatment history for TB surveillance and management. Thorough interview of patients by using standardized detailed questionnaires should be emphasized and become routine. However, the observed misclassifications had no major impact on resistance rates and data collected through the network may be used for policy making and TB programme evaluation.

APPENDIX

The following members of the Azay-Mycobacteria network actively participated in the study:

M. Chetaou (Angers), G. Couetdic (Besançon), F. Jaureguy (Bobigny), J. Texier-Maugein (Bordeaux), B. Malbruny (Caen), L. Lebrun (Clamart), L. Deforges, (Créteil), M. Chomarat, M. de Montclos (Lyon-Sud), D. Terru (Montpellier), M. Dailloux (Nancy), P. Bemer (Nantes), L. Landraud, D. Sicard, P. M. Roger (Nice), C. Pierre, R. Ruimy (Paris – Bichat), S. Coignard (Paris – Hôtel-Dieu), C. Truffot-Pernot (Paris – Pitié-Salpêtrière), V. Lalande (Paris – St-Antoine), J.-L. Hermann (Paris – St-Louis), A. Rossier (Paris – Tenon), A. Bourgoin (Poitiers), M. Pestel-Caron (Rouen), P. Lanotte (Tours).