Improving selection of markers in nutrition research: evaluation of the criteria proposed by the ILSI Europe Marker Validation Initiative

Abstract

The conduct of high-quality nutrition research requires the selection of appropriate markers as outcomes, for example as indicators of food or nutrient intake, nutritional status, health status or disease risk. Such selection requires detailed knowledge of the markers, and consideration of the factors that may influence their measurement, other than the effects of nutritional change. A framework to guide selection of markers within nutrition research studies would be a valuable tool for researchers. A multidisciplinary Expert Group set out to test criteria designed to aid the evaluation of candidate markers for their usefulness in nutrition research and subsequently to develop a scoring system for markers. The proposed criteria were tested using thirteen markers selected from a broad range of nutrition research fields. The result of this testing was a modified list of criteria and a template for evaluating a potential marker against the criteria. Subsequently, a semi-quantitative system for scoring a marker and an associated template were developed. This system will enable the evaluation and comparison of different candidate markers within the same field of nutrition research in order to identify their relative usefulness. The ranking criteria of proven, strong, medium or low are likely to vary according to research setting, research field and the type of tool used to assess the marker and therefore the considerations for scoring need to be determined in a setting-, field- and tool-specific manner. A database of such markers, their interpretation and range of possible values would be valuable to nutrition researchers.

Introduction

A biomarker has been defined as ‘a characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or pharmacologic responses to an intervention’^{( 1 )}. Thus, biomarkers are measurements that reflect biological processes and they can be various sorts of data, such as physiological measurements; analyses of tissues, blood or other body fluids; metabolic data; genetic data; or measurements from bio-images. In recent years new technologies have enabled the simultaneous measurement of genetic sequences, messenger RNA, peptides, proteins, or metabolites resulting in patterns (or ‘signatures’) as biomarkers. Biomarkers have relevance to medical practitioners and other healthcare professionals, researchers, the general public, patient subgroups, industry, healthcare funders, regulators and policy makers. It is important to distinguish between biomarkers, risk factors and endpoints. Biomarkers are biological characteristics that are measured and evaluated. As a consequence, they are subject to measurement quality issues such as accuracy, precision, reliability, reproducibility, and the need for standards and quality control. Risk factors are variables that are related to an increased probability of developing a disease or injury; they may include biomarkers but also social and environmental factors. Endpoints are clinical outcomes or events. Surrogate biomarkers are substitutes for clinically meaningful endpoints and are expected to predict the effect of a therapy^{( 2 )}, but not all biomarkers predict risk or function as endpoints.

Biomarkers, risk factors and endpoints are all very relevant to nutrition research and are widely used. However, nutrition researchers are often interested in a broader range of exposures and outcomes. These may include food, nutrient and non-nutrient intake from the diet; behaviour in the context of food or nutrient exposure; psychological as well as physiological outcomes; and well-being. Fig. 1 depicts the relationship between dietary exposure, nutrient status, and the impact of nutrition on growth, development, behaviour, and psychological and physiological function, which in turn influence health, wellbeing and disease risk. A concept that defines biomarkers, bio-indicators and public health indicators as types of measures in nutritional assessment has recently been introduced^{( 3 )}. Hence, the term ‘markers’ is used herein to distinguish this broader range of nutritional interests from the narrower focus upon physiological ‘biomarkers’ (Fig. 1); note that ‘markers’ will include ‘biomarkers’.

Fig. 1 Contexts of markers in nutrition research. There is a relationship between dietary exposure, nutrient status, and the impact of nutrition on growth, development, behaviour, and psychological and physiological function, which in turn influence health, wellbeing and disease risk. Nutrition research requires validated markers for each of these levels. Note that the same measure may serve as both a marker and an outcome, depending upon the context.

The conduct of high-quality nutrition research requires the selection of appropriate markers as outcomes, for example as indicators of food or nutrient intake, nutritional status, health status or disease risk. The selection of suitable markers will allow a research question to be robustly addressed, but such selection requires detailed knowledge of the markers, and consideration of the factors that may influence the measurement of these markers, other than the effects of nutritional change. A framework to guide selection of markers within nutrition research studies would be a valuable tool for researchers in the field. In this context, a key conclusion of the European Commission-funded project PASSCLAIM (Process for the Assessment of Scientific Support for Claims on Foods), coordinated by the European Branch of the International Life Sciences Institute (ILSI Europe), was that there is a lack of adequate markers in nutrition sciences and that there is a high need for such markers^{( 4 )}. ILSI Europe therefore launched an activity, ‘Marker Initiative on Nutrition Research’, with the aim of identifying and reviewing criteria for validation of markers. It was envisaged that this would be a multi-step process, as illustrated in Fig. 2. The first step was the identification of those criteria that have been used to assess a broad range of markers in nutrition research. This was followed by a Workshop, ‘Obtaining consensus on the criteria for evaluating markers in nutrition research’, held in June 2012 in Lisbon, Portugal, comprising step 2 of the process (see Fig. 2). During the Workshop, participants established a preliminary list of consensus criteria for marker evaluation for nutrition research^{( 5 )}:

1. (1) The marker should be validated according to recognised methods;

2. (2) The marker should reflect an endpoint (there should be a significant association between the marker and an endpoint in a target population and the marker should change consistently with a change in the endpoint);

3. (3) The marker must respond to a dietary intervention.

Fig. 2 The International Life Sciences Institute (ILSI) Europe Marker Initiative on Nutrition Research: a stepwise approach towards criteria for the evaluation of markers in different fields of nutrition research.

The next step in the process (step 3), the current activity, was to assess the use of these criteria, using a range of different possible markers reflecting the breadth of nutrition research possibilities in order to test whether the criteria were fit for purpose, and, if not, to propose alternatives. The current activity was performed by a multi-disciplinary Expert Group, members of which discussed all aspects under consideration until consensus was reached. This article conveys the result of those discussions. One outcome of the current activity is a revised list of criteria, incorporated into a template. A second role of the current activity was to consider the development of methods for scoring markers against the pre-specified criteria, and to develop a template for this purpose. Review of ‘nutritional (bio)markers’ themselves was outside the remit of the Expert Group, the overall aim of which was to (re)consider the process by which such markers can be evaluated.

Testing the proposed criteria

It was considered that the best way to test the criteria was to complete a template based on the criteria established by de Vries et al. ^{( 5 )} (see Table 1) using examples of markers that reflect:

1. (1) A broad range of interests in nutrition research (see Fig. 1);

2. (2) The use of different tools, including both questionnaires and laboratory tests;

3. (3) Both long-established and newer markers and tools;

4. (4) Commonly used and not commonly used markers.

Table 1 Template to aid the evaluation of candidate markers for their usefulness in nutrition research according to previous step 2 of the Marker Initiative on Nutrition Research (de Vries et al. (2013)^{( 5 )})

Hence, the markers selected are not necessarily well validated or widely accepted. Markers covering the fields of nutrient intake, nutrient status, physiological function, metabolism, cognitive function and disease risk were all evaluated (Table 2).

Table 2 Markers used to assess the proposed criteria according to their specific field of application

FADS1, fatty acid desaturase 1.

Individual members of the multidisciplinary Expert Group completed the draft template (Table 1) using the criteria proposed in step 2 (de Vries et al. ^{( 5 )}) (henceforth ‘proposed criteria’, see Fig. 2) for each of the markers selected and then the completed template was discussed amongst all members and modifications made until consensus on the utility of the criteria for each marker was reached. One completed template is included (Table 3), while the completed templates for each of the thirteen markers are included in the online Supplementary material.

Table 3 Example of a completed template: use of response to vaccination as a marker of immune competence

Refining the criteria and developing a new evaluation template

As experts evaluated the proposed criteria and during the subsequent discussions, a number of pertinent points emerged regarding the ease of use, utility and relevance of the criteria and also the exact wording used to describe/define criteria. Although several of the core components that form part of the criteria are clearly defined (for example, sensitivity, specificity), others are not (for example, robustness), making it difficult to address these less well-defined criteria. Furthermore, in the absence of clear definitions, different individuals interpret the meaning of these terms or criteria differently, resulting in a less robust and less reproducible (from individual to individual) evaluation. It was recognised that, despite that fact that some markers are widely used, they fail to meet some of the criteria; for example in some cases assays may be poorly standardised. Furthermore, some measures are often used as endpoints themselves rather than as markers of other endpoints (outcomes); for example verbal memory, a marker of cognitive function, is often reported as an endpoint in its own right. Also, some markers have remained in use over a very long period of time, perhaps because they become validated or well accepted or, in some cases, because they are easy to use. On the other hand, some markers cease to be used after a period of time while new markers can emerge. Thus, there is a certain level of ‘turnover’ of markers. This has been hastened by the emergence of new technologies, typically ‘omics’-based, that have enabled the simultaneous measurement of clusters or patterns of markers. Some of these seem likely to replace existing single-measurement markers, although validation of the patterns, access to the technology and cost remain barriers. An example of how the marker field is developing is the proposal of a composite biomarker called the ‘vascular health index’ based on the integration of a meticulously selected cluster of biomarkers all related to vascular health measured with different types of analytical techniques^{( 6 )}.

Many markers are evaluated in a static setting, for example in fasting blood samples. This separates the sample and its component markers from the reality of human physiology, which is the need to respond appropriately to ‘daily stressors’ which may be metabolic, immune, physical (for example, exercise, temperature) or psychological. Thus, it may be desirable to include a challenge test in a study protocol and to evaluate the dynamic change in the marker in response to the challenge. Some of the examples of markers used to test the criteria do include a challenge test (for example, response to vaccination); again it seems that in the future more studies will incorporate challenge models to simulate events which could occur in the natural environment^{( 7 , 8 )}. One area where challenge models have become widely used in recent years in order to study the dynamic change in a marker is the evaluation of inflammation. For example, both high-fat and high-carbohydrate meals induce an acute elevation in a number of biomarkers of inflammation^{( 9 )} and such challenges have been used to assess the effect of including fibre^{( 9 )} or vitamin C^{( 10 )} in the meal on the acute inflammatory response that is elicited. Exposure of the skin to UV irradiation induces inflammation and controlled exposures have been used to assess the effects of including n-3 fatty acids^{( 11 )} or green tea catechins^{( 12 )} in the diet on a range of biomarkers of inflammation. Intramuscular injection of bacterial endotoxin has been used to assess the effect of dietary n-3 fatty acids on inflammation^{( 13 )}.

It also emerged that a marker may not be equally useful across different applications. For example, a marker that is informative in the controlled setting of a small intervention study may be much less informative, or even unfeasible, in the setting of a large observational study.

The proposed criteria included ‘Must respond to a dietary intervention’. However, upon further discussion it was considered that whether a measure (i.e. a marker) is sensitive to nutrition does not make it a better or worse marker. This will depend on what the marker is designed to measure. Further, when considering whether a marker is influenced by a dietary or other intervention, then the extent of the effect seen (or foreseen) needs to be taken into account. This poses a challenge because studies are typically powered to show a statistically significant change in the marker being used. Even if that marker has an association with an endpoint, a statistically significant change in the marker may not be of clinical significance or even biological significance. Conversely, a change that is clinically significant may not be statistically significant in a study setting. Thus, it would seem prudent when planning a nutrition study to consider both clinical and statistical significance of the change being sought. Such considerations of study design, including effect size, are discussed elsewhere^{( 14 )}. It was concluded that the two criteria listed in the section ‘Reflect/mark an endpoint’ were essentially addressing the same point: that a relationship exists between the marker and an endpoint of interest. It was difficult for experts to adequately complete the section on ‘Analytical aspects’ because the different criteria asked about in this section were not well separated. Thus, by completing the draft template, based upon the proposed criteria, a number of the components of the proposed criteria were identified for change or improvement. It was also identified that providing information on normal values or ranges in different population subgroups and thresholds used to make different conclusions would be very valuable and was not explicitly requested in the draft template. It was also felt that sections in a new template to add other relevant information, for example to record inconsistencies in the literature, to make a clear conclusion about the usefulness of the marker under consideration, including any important limitations, and to record references used would all be valuable.

The above considerations led the Expert Group to conclude that the proposed criteria could be improved upon and therefore the draft template (Table 1) was revised to produce a new template reflecting these improvements (Table 4). This retains the general features of the proposed criteria (as described in the draft template), but the template is formatted in a way that is easy for the end-user to complete with a clearer indication of the nature of the information that is required for each section. The section ‘Methodological aspects’ (previously termed ‘Analytical aspects’) explicitly separates the most important components (validation; sensitivity; specificity; technical aspects other than sensitivity and specificity; biological variation), providing an opportunity to consider these individually. The section ‘Reflects the biological purpose of the marker’ (previously ‘Reflects/marks an endpoint’) combines the two previous criteria (‘Significant association between marker and endpoint in a target population’ and ‘Marker changes consistently with a change in the endpoint’) into a single reworded criterion (‘A change in the marker is linked with a change in the endpoint in one or more target population(s)’). This is because the two previous criteria address the same point, both stating that a relationship exists between the marker and an endpoint of interest. The section ‘Relevance to nutrition research’ (previously termed ‘Must respond to a dietary intervention’) expands upon and presents a change in focus from the proposed criteria. Now information on the normal range of values can be entered and the requirement that the marker must respond to a dietary intervention is replaced by a question seeking the evidence that nutrition can influence the marker and, if so, the extent of the reported effect. The reason for this change in focus is that whether a measure is sensitive to nutrition or not does not make it a better or worse marker, although it may make it more or less attractive to researchers and other stakeholders. Finally, in this section a question about other factors that might affect the marker is now included. A section ‘Other relevant information’ is included and there are cells for ‘Conclusion’ and to record ‘References’. It is considered that these changes will make the criteria and associated template more useful and more robust.

Table 4 Refined template to aid the evaluation of candidate markers for their usefulness in nutrition research

* Appropriate is used here to indicate that the required sensitivity and specificity of measurement may differ between study contexts, for example between a large epidemiological study and a much smaller randomised controlled trial.

It is anticipated that once a particular marker has been assessed using the criteria and the associated template, the information will be used as the basis for scoring the marker in order to determine its usefulness as a research tool. Such scoring requires a suitable methodological approach (see next paragraph).

Towards developing a marker scoring system

Examples of scoring systems may be found in Albers et al. ^{( 15 , 16 )} where markers of immune function were evaluated by scoring them against a range of predetermined criteria; in Albers et al. ^{( 16 )} these related to clinical relevance, biological sensitivity and feasibility. Table 5 is the generic marker scoring template now proposed. Researchers would score any marker under consideration according to the different criteria listed in the upper section of Table 4 as proven (+++), strong (++), medium (+) or low (0). Additionally, an arbitrary marker score would be based on subjective expert judgement on the usefulness of a marker based on carefully considered evaluation of individual criteria. This would enable researchers to evaluate and to compare different candidate markers within the same field of nutrition research in order to identify their relative usefulness. The criteria for a ranking of proven, strong, medium or low are likely to vary according to the research setting (for example, epidemiology, intervention, mechanistic investigation), the research field (for example, immune function, cognitive function, metabolism and metabolic dysfunction), and the type of tool used to assess the marker and therefore the ranking criteria need to be determined in a setting-, field- and tool-specific manner. Examples of such criteria and their use in evaluating immune markers may be found in Albers et al. ^{( 16 )}. It is expected that researchers would develop scoring definitions and then score and rank potential markers as part of study planning.

Table 5 Generic scoring system to evaluate and compare candidate markers within the same field of nutrition research

Summary and conclusions

An Expert Group set out to test proposed criteria (see Table 1) designed to aid the evaluation of candidate markers for their usefulness in nutrition research and subsequently to develop a scoring system for markers. The criteria were tested using a total of thirteen markers selected from a breadth of fields of nutrition research (Table 3 and online Supplementary material). The result of this testing was a modified list of criteria and a template (see Table 4). It is considered that these changes will make marker assessment easier and more robust. Subsequently a system for scoring a marker and an associated template were developed (Table 5). This system would enable researchers to evaluate and to compare different candidate markers within the same field of nutrition research in order to identify their relative usefulness. The ranking criteria of proven, strong, medium or low are likely to vary according to research setting, research field and the type of tool used to assess the marker and therefore the criteria need to be determined in a setting-, field- and tool-specific manner. Examples of such ranking criteria for immune function markers may be found in Albers et al. ^{( 16 )}. It is anticipated that defining the scoring system and then using this to score possible markers would be done by researchers as a part of their study planning. These activities and the development of the templates described in Tables 4 and 5 complete step 3 of ILSI Europe’s marker initiative programme (see Fig. 2). The next step is to use the evaluation criteria and scoring system to evaluate markers. It is anticipated that ILSI Europe will hold an open access ‘library’ of completed evaluations, using the templates developed in this activity that will be available to the nutrition community for use, comment, modification and updating. Besides applying the evaluation tool in study planning, researchers would complete templates with scenarios of marker applications to populate such a library with evaluated markers from various fields of nutrition research. It is important to note that many of the markers considered here and in the future are of interest to research communities beyond the field of nutrition and, as such, the library of ILSI Europe marker evaluations will be a valuable resource for a wide research community and for other relevant stakeholders (for example, industry, regulators, medical practitioners).