A new surveillance indicator identifying optimal timeliness and accuracy: application to the Korean National Notifiable Disease Surveillance System for 2001–2007

Summary

Although immediate notification of a case is crucial for epidemic control, clinicians may delay notification due to uncertainties in diagnosis, reflecting a trade-off between timeliness and the accuracy of surveillance. We assessed this trade-off for four epidemic-prone diseases that require immediate notification of suspected cases: shigellosis, typhoid fever, paratyphoid fever, and cholera in the Korean National Notifiable Disease Surveillance System data for 2001–2007. Timeliness was measured as the time to registration (T_R), being the time interval from symptom onset to notification by the clinician to the local public health centre. We introduced a new index, ‘time-accuracy trade-off ratio’ to indicate time saved by clinical vs. laboratory-based notifications. Clinical notifications comprised 34·4% of total notifications, and these showed a shorter median T_R than laboratory-based notifications (1–4 days). The trade-off ratio was greatest for shigellosis (3·3 days), and smallest for typhoid fever (0·6 days). A higher trade-off ratio provides stronger evidence for clinical notification without waiting for laboratory confirmation.

INTRODUCTION

Timely notification is crucial for effective surveillance and control of global epidemics of infectious diseases. Since it takes time to confirm cases, suspected cases are designated as notifiable in order to save detection time. Although most communicable diseases require immediate notification of suspected cases, physicians still delay notification until laboratory confirmation. For example, although gastrointestinal or foodborne infections are prevalent notifiable diseases of great importance in the UK, notification remains low despite regulations that require notification of suspected cases. There has even been a suggestion that the current system of notification of suspected cases should be revised or abandoned [1]. Uncertainty of diagnosis is one of the major reasons why clinicians do not notify cases [1–4].

This non-compliance is encountered in routine surveillance as well as during epidemics. The core of the problem involves the trade-off between the timeliness and positive predictive value (PPV) of reporting. For example, a delay in a clinician's notification to health authorities hinders a rapid public response and increases the risk of subsequent outbreaks. In contrast, a case notified without laboratory confirmation could prove to be a false-positive result that leads to unnecessary public health measures such as investigation, isolation, and public concern and confusion.

Several important attributes of surveillance functions have been proposed [5]. For early detection and a rapid response to an epidemic, timeliness and diagnostic accuracy are the two main factors challenging the clinicians responsible for reporting communicable diseases. In the clinical setting, these factors are directly linked to the attributes of the surveillance function.

Timeliness as a system attribute is measured by the interval between any two steps within a surveillance system. It is related to the system's ability to take timely action. This enables us to assess the time lag or bottleneck phenomenon in a surveillance step. Diagnostic accuracy in clinical reporting is related to two system attributes, sensitivity and PPV. Sensitivity is usually impractical for routine measures because of the difficulty in estimating the true number of cases in a population. In contrast, PPV is readily available in practice, since it is calculated by the proportion of confirmed cases among all notifications. It is useful for assessing the adequacy of resource utilization, since low PPV leads to misdirection of surveillance resources [5, 6]. Therefore, timeliness and PPV are two important practical measures that link clinicians’ decision making to surveillance performance.

Generally, timeliness is more important than accuracy for contagious diseases with high infectivity or virulence, because any delay can lead to large outbreaks or severe results [7]. However, there are different levels of balance between timeliness and PPV in terms of notification on the basis of clinical judgement vs. notification after laboratory confirmation. Clinical notification before laboratory confirmation is faster, but includes more false positives (FPs). On the contrary, laboratory-based notification involves fewer FPs, at the cost of delay. A more scientific approach is necessary to optimize notification guidelines for clinicians.

The Korean National Notifiable Disease Surveillance System (KNNDSS) provides excellent data by which to analyse the trade-off between timeliness and PPV. The Korea Centers for Disease Control and Prevention (KCDC) began collecting individual data that were notified to the national surveillance system from August, 2000. As of January 2010, this system covers 50 infectious diseases. Clinicians notify infectious disease cases to their public health centre (PHC), which initiates a series of public health responses, including epidemiological investigation and intervention to minimize any outbreak [8]. Using data from the KNNDSS, we investigated the nature of the trade-off between timeliness and PPV, and introduced new measures to help quantify the optimum balance between these two factors, thereby facilitating scientific decision making in surveillance.

METHODS

Data collection

We used the KNNDSS database from 2001 to 2007, which was released without personal identifiers for research purpose. Four diseases were selected for this study from those classified as Group I by the Contagious Diseases Prevention Act of Korea: shigellosis, typhoid fever, paratyphoid fever, and cholera. Group I is defined as the diseases that require immediate control measures by law, thus it is appropriate to investigate the balance between timeliness and accuracy. The other two diseases in Group I, Enterohaemorrhagic Escherichia coli and plague, were not selected because of insufficient information. The enterohaemorrhagic E. coli surveillance system had been changed from passive to active methods in 2003 so the information is not comparable for the current analysis, and no case of plague has been reported. Four selected diseases are mandatory for notification of both suspected and confirmed cases, and all reported cases were recommended for laboratory confirmation and epidemiological investigation. Laboratory confirmation of all four diseases depends on identification of pathogens by conventional culture methods.

Definitions

Data classification

Figure 1 shows the notification algorithm and the scheme for data classification. Clinical notification implies notification based on clinical suspicion according to KNNDSS guidelines. These guidelines require that physicians notify suspected cases without waiting for laboratory results. Physicians are also asked to update laboratory results for the notified cases according to the follow-up laboratory testing for confirmation. We classified these cases by laboratory results as: true positive (TP), false positive (FP), or unclassified (UC). Despite the guidelines, physicians often wait to report cases until after confirmation of the laboratory result. These cases were classified as laboratory-based notifications.

Fig. 1 [colour online]. Notification algorithm of data classification and T _R (Korean National Notifiable Disease Surveillance System 2001–2007). T _R, time to registration (from symptom onset to clinician's notification to a public health centre); TP, true positive; FP, false positive; UC, unclassified (no laboratory confirmation).

Time lag

Timeliness was measured as the time to registration (T _R), being the time interval from symptom onset to notification by the clinician to the local PHC. This time interval indicates the delay until the first recognition by the surveillance system that enables public health action to be initiated [8].

Figure 2 shows the cumulative distributions of T _R. The x-axis is the T _R, and the y-axis is the cumulative proportion of notifications within the given T _R. For example, about half of the cases were notified within 10 days. Use of this graphical method has been published previously [8]. In the graph (Fig. 2 a), the area over the curve (AOC) represents the sum of T _R for all reports divided by the number of reported cases, i.e. the average T _R. The shape of the AOC provides additional insights into how the delay of reporting is distributed, or how it can change with different policies. For example, Figure 2 b illustrates that laboratory-based notification has, on average, a longer T _R than clinical notification. Furthermore, the shaded area between the two curves indicates that the difference in reporting is largely distributed between 5 and 15 days in this hypothetical case (Fig. 2 b).

Fig. 2 [colour online]. Conceptual model for estimating T _R reduction caused by early notification (Korean National Notifiable Disease Surveillance System 2001–2007). T _R, time to registration (from symptom onset to clinician's notification to a public health centre); CD, cumulative distribution of T _R (proportion of cases notified within the given time (days); AOC, area over curve (person-days needed for all notifications to take place); AOCdiff, difference in AOCs.

New measures

Based on the conceptual framework described above, we developed new measures for analysis of the time-accuracy trade-off, as in Table 1. ‘Time reduction per true positive (▵₀)’ is the index of improvement in timeliness that shows how much time is on average saved by clinical versus laboratory-based notifications for a TP case. Our time reduction measure only utilizes TP reporting, regardless of the distribution of FP reporting times. This definition is used primarily because the actual reduction of a potentially infectious period would only involve TP reporting, but it also has the advantage of not being influenced by the distribution of FP cases. Multiplying ▵₀ by PPV leads to ‘effective time reduction (▵)’, an index of the actual improvement in timeliness for an average notification, reflecting the probability of being a TP that may be <1. Multiplication of ▵ again by the total number of notifications provides ‘total time reduction (Σ)’, the overall reduction in infective person-days in all notifications, to which only TP cases contribute. This can also be calculated by multiplying ▵₀ by the number of TP cases. Finally, ‘time-accuracy trade-off ratio (ρ_ta)’ is defined as the ratio between the effective time reduction and the probability of a FP, i.e. ▵/p(FP). This ratio involves the odds of TPs and FPs in the total notifications, as a penalty weight for ▵₀. It also has an implication for the cost-benefit ratio of T _R reduction at the cost of an additional FP case. This ratio is not defined when PPV = 1. In this situation, there is no FP and the shorter T _R is always better, without any trade-off. One useful interpretation of the ‘time-accuracy trade-off ratio’ measure is in terms of total time reduction, as ρ_ta = Σ/n(FP). For example, ρ_ta = 3 implies that clinical reporting without waiting for laboratory confirmation can effectively reduce 3 person-days of infectious period at the cost of one additional FP reporting.

Table 1. Measures for time-accuracy trade-off analysis

Statistical analyses

We analysed the time lags and AOCs of TP clinical and laboratory-based notifications. Differences in AOCs for each disease are shown graphically. Two-sample Kolmogorov–Smirnov (KS) tests were used to compare the cumulative distributions of time lags between TP and laboratory-based notifications. This provides a general non-parametric test of the differences derived from two independent samples of empirical cumulative distribution functions in both location and shape [9]. All other analyses were performed using the SAS software package, version 9 (SAS Institute Inc., USA), and Microsoft Office Excel 2003.

RESULTS

For the four diseases studied, a total of 7436 cases were identified (Table 2). Shigellosis (58·7%) comprised the largest proportion of cases, followed by typhoid fever (27·2%), paratyphoid fever (10·9%), and cholera (3·2%).

Table 2. Number of cases (%) notified by clinical or laboratory-based notifications (KNNDSS, 2001–2007)

KNNDSS, Korean National Notifiable Disease Surveillance System; PPV, positive predictive value [calculated as A/(A + B) in this table]; TP, true positive; FP, false positive; UC, unclassified (no laboratory confirmation).

Table 2 shows the number of cases and proportion by case classification of notified data. Clinical notifications comprised 34·4% of total notifications, with the highest for typhoid fever (39·9%) and the lowest for paratyphoid fever (25·6%). However, the proportion of TP clinical notifications in all notified cases was highest for shigellosis (18·4%), whereas it was lowest for typhoid fever (7·2%). Similarly, PPV was highest for shigellosis (0·68) and lowest for typhoid fever (0·24).

Median T _R varied by disease (Table 3). Shigellosis showed the shortest median T _R (6 days), followed by cholera, typhoid fever, and paratyphoid fever. All four diseases showed a shorter median T _R for TPs than for laboratory-based notifications, with the difference ranging from 1 to 4 days.

Table 3. Median T_R in days by data classification (KNNDSS, 2001–2007)

T _R, Time to registration (from symptom onset to clinician's notification to a public health centre); KNNDSS, Korean National Notifiable Disease Surveillance System; TP, true positive; FP, false positive; UC, unclassified (no laboratory confirmation).

Figure 3 illustrates the cumulative distributions of T _R for the two curves for TPs of clinical and laboratory-based notifications. For each disease, the curve for laboratory-based notifications was always below that for TPs, and the overall differences between the two curves were significant by two-sample KS test (Table 4). These figures are more informative than median T _R values because they clearly illustrate the gaps between the two curves. Furthermore, they also show that the gaps are present from the beginning and persist throughout the time span. The areas between the curves (i.e. AOC differences) represent the transmission time that can be eliminated by prompt clinical notification. The shapes of AOC difference vary by disease. For shigellosis, the difference is distributed relatively uniformly throughout time, whereas for paratyphoid fever, a relatively larger difference is evident after 10 days.

Fig. 3 [colour online]. Cumulative distributions of T _R (Korean National Notifiable Disease Surveillance System, 2001–2007). T _R, time to registration (from symptom onset to clinician's notification to a public health centre); TP, true positives (in clinical notifications).

Table 4. AOC, PPV, and time-accuracy trade-off (KNNDSS, 2001–2007)

AOC, Area over curve; PPV, positive predictive value; T _R, time to registration (from symptom onset to clinician's notification to a public health centre); KNNDSS, Korean National Notifiable Disease Surveillance System; TP, true positives (in clinical notifications).

* Kolmogorov-Smirnov two-sample test.

† A and PPV obtained from Table 2.

Table 4 shows the results of a time-accuracy trade-off analysis. Time reduction per TP (▵₀) was greatest for paratyphoid fever (4·3 days) and smallest for shigellosis (1·6 days). Effective time reduction (▵) reflects PPV, and was therefore smaller than ▵₀ for all four diseases, ranging from 1·6 days for paratyphoid fever to 0·4 days for typhoid fever. Total time reduction (Σ) during the study period for all TP notifications became greatest for shigellosis by 1250 days. Finally, time-accuracy trade-off ratio (ρ_ta), ▵/p(FP), became greatest for shigellosis (3·3 days) and smallest for typhoid fever (0·6 days). The trade-off ratio showed that shigellosis had the greatest gain in timeliness for a loss in accuracy of one additional FP.

DISCUSSION

Brief summary

We investigated the timeliness and PPV between clinical and laboratory-based notifications for four epidemic-prone diseases. Clinical notification always showed timeliness that was superior to that of laboratory notification. We suggested four new measures to analyse the trade-off between timeliness and PPV. These indices give us not only general information for surveillance but also the tools to compare the characteristics of different notification protocols for local and global epidemics.

Properties of new measures

The new measures we propose in this study provide information on different aspects of time-accuracy trade-off, when two notification policies are compared. First, time reduction indices, for a TP (▵₀) and average notification (▵), measure the efficacy of improved timeliness by clinical notification without waiting for laboratory confirmation. When there are FPs in notified cases, the efficacy is influenced by PPV. This is reflected by effective time reduction (▵). For example, typhoid fever had a small ▵ mainly because of a low PPV. In addition to these two measures of time reduction, it is informative to examine the shape of differences in AOC, because time reduction gained during an earlier stage of an epidemic is more beneficial.

Second, total time reduction (Σ) shows the overall effectiveness of clinical notification, as compared to laboratory-based notification, in terms of actual improvement as measured by a reduction in infective person-days. Both incidence and PPV have decisive effects on this index. Shigellosis had the greatest total time reduction, because of the large number of TPs despite the relatively low ▵₀. On the contrary, paratyphoid fever showed a relatively low level of total time reduction because of the small number of TPs, despite having the highest ▵₀.

Third, the time-accuracy trade-off ratio (ρ_ta) reflects the efficiency of clinical notification in the actual context, considering the loss in accuracy. This ratio can be used to assess the balance between the benefit in time reduction and the cost of additional FPs. For example, the T _R of true shigellosis cases was reduced by 3·3 days by clinical notification, at the cost of one additional FP. In contrast, typhoid fever showed 0·6 for the ratio, much lower than that for shigellosis. According to this analysis, promoting clinical reporting of shigellosis is a high priority. However, given the low efficiency for typhoid fever, improving the PPV of reporting may be more important. This may be achieved by reviewing the current criteria for clinical reporting or case definitions (see Appendix Table 1).

Factors influencing the trade-off

The trade-off between timeliness and accuracy may be influenced by several factors. First, public awareness and clinical sensitivity are likely to increase timely reporting for a given PPV. For example, notification of more common, severe diseases, especially in epidemic situations, is typically performed quickly [4, 8, 10, 11]. In the present study, shigellosis is the most common and epidemic-prone of the four diseases studied, and has typical symptoms including diarrhoea and abdominal pain. Therefore, patients may seek medical help faster, and clinical suspicion and reporting would be processed more rapidly. In contrast, typhoid fever and paratyphoid fever are less frequent, and exhibit longer periods of non-specific general symptoms, such as a high fever. In general, promoting public and clinical awareness is an important strategy to obtain better outcomes in the trade-off. Second, as shown in our data, the effect of total time reduction depends on the number of cases in the population. A greater number of cases in the population is of concern in itself, but also requires emphasis on timeliness because of the larger effect in terms of total time reduction. Third, the timeliness and PPV observed in practice may depend on the disease-specific goals of surveillance. For vaccine-preventable diseases, timeliness is usually emphasized more. It is possible to reduce disease incidence by vaccination, but the remaining susceptible population pocket can lead to a large outbreak unless timely detection and response to a new case is assured [12]. On the other hand, when the surveillance goal is mainly for monitoring trends rather than early detection of epidemics, PPV attracts greater attention in practice. In Finland, tuberculosis is closely monitored for multidrug resistance. Even though clinically suspected cases are notifiable, physicians tend to report later in the clinical process, resulting in a surveillance PPV of 99% [13].

Potential applications

For clinicians, our new measures can provide empirical data to support decision making for notification when assessing suspicious symptoms. For public health practitioners, the measures will indicate where to improve control activities in both routine and epidemic situations. For policy makers and researchers, the indices will be useful for evaluation of surveillance system performance.

Surveillance systems must produce appropriate data for decision making with regard to the trade-off between timeliness and PPV. Assessment of T _R is based on the dates of disease onset and notification. PPV is estimated from laboratory results for a representative sample of clinically notified cases. Additionally, the date of laboratory confirmation is informative for monitoring the time delay between clinical suspicion and laboratory confirmation.

Limitations and further issues

Our study has several limitations. First, the clinical features of cases notified clinically, and those notified based on laboratory results, might be different. For atypical or uncertain cases, clinicians might postpone diagnosis until they obtain laboratory confirmation. If they notify such cases by clinical suspicion alone, more FPs would be included in clinical notifications. Thus, our estimation of the trade-off ratio might have been, to some extent, an overestimate.

Second, we used T _R (time from the symptom onset to notification) as the measure of timeliness, instead of the time from clinicians’ diagnosis to notification (T ₂). When the time from symptom onset to clinical diagnosis (T ₁) varies widely across diseases or populations, reduction in overall T _R (T ₁+T ₂) by clinicians’ notification practice (which only affects T ₂) may be relatively small [8].

Third, the level of completeness of notification might influence the measures we propose. Completeness for the selected diseases in the KNNDSS ranges from 64·0% for typhoid fever to 77·8% for shigellosis according to research by medical record review for 57 general hospitals selected nationwide in 2005 [8, 14]. This level of completeness is comparable to other countries with well-established surveillance systems [15]. Relatively severe cases with distinct clinical features are notified more frequently; therefore, higher levels of completeness might result in inclusion of less-clear cases, which can lead to more FP results. In such contexts, the trade-off ratio might become smaller.

Finally, we did not measure the actual benefit of improved timeliness or cost of FPs in comparable economic units. The actual economic benefit and cost are likely to depend on the nature of the disease and also on public health measures that, themselves, vary widely by disease. The measures that we propose need to be combined with broader epidemiological and economic evaluations for policy decision making.