Historic public health scares (e.g. in the UK following the Phillips inquiry into variant CJD/new variant CJD, the human prion disease caused by the BSE crisis in 1996) contributed to a general commitment for transparency, openness and evidence-based (information indicating whether a belief or proposition is true or valid) decision making in food-related science and policy( Reference Lofstedt, Bouder and Wardman 1 ). This resulted in the formal separation of science and policy activities and adoption of the risk analysis framework( 2 ).
The risk analysis framework was first adapted for use in the food area following a joint FAO/WHO expert consultation in 1995( 3 ). The framework has been comprised of three interconnected activities: (i) risk assessment (scientific evaluation of health effects); (ii) risk management (policy decisions to minimise risk); and (iii) risk communication (exchange of risk information in/outside the framework). A principle has been to maintain the separation of activities while recognising the interaction between activities (e.g. separation can vary from different organisations to different groups/tasks within the same organisation responsible for different activities)( 3 ).
Central to implementation of the risk analysis framework has been the encouragement of transparency and disclosure of uncertainty in the evidence underlying scientific advice (risk assessment) or political decision making (risk management)( 2 , 4 ). The transparent handling of uncertainty may refer to ‘explicit recognition of any uncertainty either in the current state of knowledge or in the adequacy of the available data’ (e.g. accuracy/quality or degree of incomplete/quantity of data)( 5 ). Transparency and openness have been promoted to achieve greater accountability and credibility during risk analysis, improved communication between stakeholders (e.g. science, policy, industry, health practice) and safeguard the abuse of public organisation power or resources( 2 , Reference Bailar and Bailer 6 – Reference Schreider, Barrow and Birchfield 8 ).
Policies of transparency can be found across various public health disciplines responsible for risk analysis activities, such as the European Food Safety Authority (EFSA), the National Institute for Health and Care Excellence (NICE) and FAO/WHO( 2 , 4 , 9 ). Implementation of these policies requires the process and outcome of a risk analysis activity to be evident and obvious, i.e. documented and understandable (transparent), as well as accessible (open)( 7 , Reference Schreider, Barrow and Birchfield 8 , 10 , Reference Turilli and Floridi 11 ). This may be related to the organisation responsible for a risk analysis activity (e.g. disclosure of organisation role and responsibility, membership, confidentiality signed agreements, declaration of interest) or the technical content of the activity (e.g. disclosure of non-published requests, methods, findings and conclusions via websites, reports and manuscripts, meeting minutes and agendas, open consultations and open meetings, release of data and study results, etc.).
A commitment to transparency and openness may seem the correct thing to do in view of due diligence towards scientific practice( Reference Schreider, Barrow and Birchfield 8 ). Nevertheless, the implementation of transparency per se may be limited or represent certain challenges( Reference Turilli and Floridi 11 ). Detailing and providing access to all elements of the risk analysis process may be challenging in terms of resource, technical, confidentiality or wider trust issues (e.g. articulating the nature of risk assessment or inherent uncertainty of scientific endeavour; detailing contradictory views during commonplace expert discussions while maintaining unanimity in the final scientific advice)( Reference Bal, Bijker and Hendriks 12 , Reference Palmer and Hardaker 13 ).
Nutrition risk assessment to derive micronutrient Dietary Reference Values (DRV) is an area where greater transparency and openness has been encouraged( Reference Dhonukshe-Rutten, Bouwman and Brown 14 ). DRV are developed in the first two steps of the risk assessment process. The identification of a nutrient-related hazard in a food/food group is established in step one (nutrient-related hazard identification). The qualitative/quantitative evaluation of adverse health effects associated with a nutrient is established in step two (nutrient-related hazard characterisation). This constitutes the development of DRV. In a full risk assessment, DRV (steps one and two) would be combined with a nutrient intake/exposure assessment (step three) to assess exposure in relation to the DRV (step four)( Reference Baines, Cunningham and Leemhuis 15 ). Once completed, the scientific advice from the risk assessment is used to inform development of recommendations in risk management. Thus, the development of DRV can be considered a precursor to the development of micronutrient recommendations. Recommendations are defined as population targets of micronutrient intake necessary for adequate growth, function and health throughout the human lifespan. These are widely used in monitoring and evaluating population intakes as well as in developing public health policies, interventions and dietary risk communications (e.g. food-based dietary guidelines to help the population meet recommended intakes)( Reference Dhonukshe-Rutten, Bouwman and Brown 14 , 16 ).
The widespread use of DRV (directly or indirectly via recommendations) belies the complexity and uncertainty with which they are developed. The first stage of DRV development has been referred to as the ‘nutrition problem formulation stage’. During problem formulation risk managers and assessors are required to establish a shared understanding of the problem and the purpose of the risk assessment( 10 ). Previously, the primary focus of DRV development was to devise intakes related to nutrient deficiency or overconsumption. More recently the remit of DRV has expanded to also encompass intakes for health benefits (including intakes to delay the onset of disease). Yet, the evidence base surrounding the role of nutrition on health has been far from certain. Hence, prevailing scientific knowledge is used to handle limitations and uncertainty in the evidence (unknown, unreliable or indefinite evidence), develop plausible assumptions and complete the DRV development process( Reference Dhonukshe-Rutten, Bouwman and Brown 14 , 17 ).
Guidelines have been produced to assist with the development of DRV (e.g. the Institute of Medicine’s Dietary Reference Intakes (DRI) for the USA and Canada; the EFSA Panel on Dietetic Products, Nutrition and Allergies’ Scientific Opinion on principles for deriving and applying DRV)( 16 , Reference Taylor 18 ). Yet, no standardised approach or agreed best practice has been used to set DRV. Furthermore, disparity has been observed in DRV developed by different national or international bodies (e.g. vitamin D DRV). The lack of agreed best practice and disparity in DRV has led to confusion among inter/national policy decision makers, health professionals, the food industry and consumers( Reference Doets, de Wit and Dhonukshe-Rutten 19 ).
An open and transparent DRV process has the potential to assist with understanding why values differ between countries by clearly detailing what, why and how decisions were made, particularly regarding the degree of uncertainty in the evidence and how this was handled or influenced the strength of the final risk assessment conclusions/resultant reference values( Reference Doets, de Wit and Dhonukshe-Rutten 19 , Reference Timotijevic, Barnett and Brown 20 ). This would inform the debate on DRV development best practice. Furthermore, increased documentation, understanding and accessibility to the DRV process could contribute to the responsible use of DRV throughout risk management and risk communication activities( 2 , Reference Dhonukshe-Rutten, Bouwman and Brown 14 ).
The European Commission-supported Network of Excellence EURRECA (EURopean micronutrient RECommendations Aligned, FOOD-CT-20006-36196, 2007–2012) was tasked with reviewing the methodologies used to derive DRV, assessing the reasons for value disparity and the potential for methodological alignment across national and international DRV development. Early EURRECA findings highlighted the importance of the nutrition problem formulation stage to ensure risk assessment activities remained achievable while also fulfilling the requirements of risk managers( Reference Timotijevic, Barnett and Brown 20 ). Further research suggested DRV disparity was unlikely to be explained by differences in concepts, definitions or defined population groups. Instead it was considered likely to be due to different interpretations or assumptions taken while accounting for uncertainty and limitations during evidence evaluation( Reference Doets, de Wit and Dhonukshe-Rutten 19 ). Finally, research identified variation in the transparency of different types of bodies in relation to how risk managers integrated DRV with other types of advice to develop recommendations( Reference Timotijevic, Barnett and Brown 20 ). The current study built upon these previous findings. Transparency and openness during the DRV risk assessment process was explored, with a particular focus on how uncertainty in the evidence base was handled. Three areas of interest, likely to demonstrate handling of uncertainty in the evidence base, were identified during DRV development: (i) the request to develop DRV; (ii) the process of DRV evidence evaluation; and (iii) the integration of evidence to develop final DRV.
Method
Design
Qualitative in-depth case studies explored the handling of uncertain evidence and the transparency of vitamin D and folate DRV development in six countries/regions: (i) Australia and New Zealand (ANZ); (ii) the Netherlands (NL); (iii) the Nordic countries (Denmark, Finland, Iceland, Norway and Sweden; NOC); (iv) Poland (PL); (v) Spain (ES)( Reference Garcia and Majem 21 ); and (vi) the United Kingdom (England, Northern Ireland, Scotland and Wales; UK).
Each country/region produced a descriptive report based upon semi-structured interviews with at least two advisory committee members together with desk research (Table 1). A common protocol was followed throughout. The interview schedule, desk research and final case study report all centred upon the transparency and handling of uncertain evidence in three predefined areas: (i) the request to develop DRV, e.g. source of request, scope of request, reasons for the request; (ii) the process of DRV evidence evaluation, e.g. selection and evaluation of evidence; and (iii) integration of evidence to develop final DRV, e.g. formation of quantitative numbers and advisory committee report.
Table 1 Country/region, number of interview samples, response rate, and references and associated updates for the folate and vitamin D DRV
DRV, Dietary Reference Values; ANZ, Australia and New Zealand; NL, Netherlands; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); PL, Poland; ES, Spain; UK, United Kingdom; NHMRC, Australian National Health and Medical Research Council; HCN, Health Council of the Netherlands (Gezondheldsraad); NNR, Nordic Council of Ministers’ Nordic Nutrition Recommendations; SENC, Spanish Society of Community Nutrition; FESNAD, Federación Española de Sociedades de Nutrición, Alimentación y Dietética; COMA, Committee on Medical Aspects of Food Policy; SACN, Scientific Advisory Committee on Nutrition.
* Recommendations under development at time of study data collection.
Transparency and openness was qualitatively judged via the availability of accessible, understandable documentation on the DRV process (e.g. downloadable/upon request meeting agendas, minutes and key discussion documents; ability to attend open meetings and consultations and documentation surrounding DRV; documented risk analysis policy, organisation role and responsibility, membership, declaration of interest; stakeholder involvement; etc.).
Vitamin D and folate were selected due to their prioritisation as micronutrients of interest in Europe( Reference Cavelaars AEJM and Dhonukshe-Rutten 22 ). Countries/regions were originally selected from Europe with a subsequent opportunity taken to also collect data in Australia and New Zealand. The countries/regions sampled represented diversity in the age and method of developing DRV as well as in geographical location, socio-cultural factors and institutional infrastructure. Data sets were collected from December 2010 to June 2011.
Interviews
Interviewees were recruited due to their involvement in the advisory committee and/or development of vitamin D or folate reference values. This included members of the advisory committee and/or members of relevant advisory committee working subgroups. The response rate ranged from 33 % (UK) to 100 % (ES and PL; Table 1). Consent was obtained from each participant, with interviews recorded and later transcribed verbatim. All data were made anonymous and held in accordance with the local data privacy laws (e.g. UK Data Protection Act)( 23 ). The semi-structured interview schedule was piloted and devised by the research team with questions focused upon the transparency and handling of uncertainty in the aforementioned three areas of DRV development.
Desk research
Sourced documents related to the development of vitamin D or folate DRV in each country/region. Keyword searches were conducted of advisory committee websites, publication websites (e.g. PubMed Central) and search engines (e.g. Google). Documents referred to during the interviews and manual searches of references elicited several additional documents. Further information was obtained in the UK case study via observation at two open advisory committee meetings (14 February 2011 and 7 June 2011). Desk research key search terms included variants of the nutrient name (e.g. folate and folic acid), variants of the word DRV (e.g. micronutrient recommendations, nutrient intake values, nutritional objectives) and/or the advisory committee name (e.g. Health Council for the Netherlands). Excluded documents were not relevant to vitamin D or folate DRV, the specific country/region, human nutrition or healthy populations or were duplications from previous searches. Desk research was led and organised by questioning the transparency and handling of uncertainty in the aforementioned three areas of DRV development.
Analysis
Initial interview and desk research analysis was conducted in the native language using template analysis and a skeleton coding structure created and modified by partners during preliminary analyses( Reference Boyatzis 24 ). Case studies consisted of an English-translated summary of interview- and/or desk research-identified themes with illustrative quotes organised by the three areas (value request, evidence evaluation and final values). Construct validity was upheld by limiting the subjective collection of data and performing traceable literature searches. Internal validity was upheld by encouraging several interpretations of the data to be considered. Multiple sources of information (triangulation between interview, desk research and DRV reports where possible) were used to cross-reference (corroborate/dispute findings). The uniform framework of case study reporting allowed further analysis where cross-case synthesis was used to describe and identify any differences or similarities across countries/regions in the transparency and handling of uncertainty( Reference Yin 25 ). One research team member conducted the cross-case synthesis. The qualitative software NVivo version 9 (2010) was used to organise case study data. A subsection of the data was cross-coded by another research team member. One fewer theme was identified by the second coder. This was subsequently dropped from further analysis. Agreement on remaining themes and sub-themes was 91%. Interpretation of cross-case analysis results was reflected back to case study authors and amended where necessary.
Data presentation
Interview quotes have been displayed in the tables using double quotation marks and italics. Verbatim quotes have been modified in publication for improved readability. Case study/desk research text has been displayed in the tables using single quotation marks. The term ‘micronutrient recommendation’ was used in the interview schedule and throughout the present study. This term could be translated and understood across the cases sampled as pertaining to either micronutrient DRV or the use of values to make recommendations.
Note that quotes have been unaltered in this respect and differences in the terminology can be seen as cases refer to their respective micronutrient recommendations, e.g. ‘The term “Nordic Nutrition Recommendations” refers to a set of dietary reference values (DRVs) for essential nutrients that includes the average requirement (AR), recommended intake (RI), upper intake level (UL), lower intake level (LI), and reference values for energy’( 26 ).
Results
Transparency and openness of the DRV process (i.e. documentation, understanding and accessibility) varied across the countries/regions (cases) and different areas of the DRV development process studied. Results have been presented below in the three areas studied: (i) the request to develop DRV; (ii) the process of DRV evidence evaluation; and (iii) the integration of evidence to develop final DRV.
Request to develop DRV
In ES, where multiple sets of DRV were developed by different individuals, teams and organisations (Spanish Society of Community Nutrition (SENC), Federación Española de Sociedades de Nutrición, Alimentación y Dietética (FESNAD), Universidad Complutense de Madrid), minimal documentation could be found to clarify details on the source, scope or reason for the request to develop DRV (meeting agendas/minutes, discussion documents, press releases, website pages, final report details, journal publications). However, the Spanish interview data were more informative and interviewees openly discussed details surrounding the request for previous DRV. In contrast, greater information (via desk research and interview data) on the request to develop DRV was available in countries/regions with an official set of DRV and a clear body tasked with their development, such as in ANZ (National Health and Medical Research Council (NHMRC)/Ministry of Health (MOH)), NL (Health Council of the Netherlands (HCN)), NOC (Nordic Council of Ministers (NCM)), PL (National Food and Nutrition Institute (Instytutu Żywności i Żywienia; IŻŻ)) and UK (Committee on Medical Aspects of Food Policy (COMA)/Scientific Advisory Committee on Nutrition (SACN); Tables 1 and 2).
Table 2 Request to develop DRV
DRV, Dietary Reference Values; ES, Spain; SACN, Scientific Advisory Committee on Nutrition; UK, United Kingdom; PL, Poland; MHWS, Ministry of Health, Welfare and Sport; NL, Netherlands.
“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.
Regarding the source of the request to develop DRV, Spanish interviewees cited the majority of activity was undertaken by independent and academic institutions. On the other hand, in the five cases of ANZ, NL, NOC, PL and UK the requests were predominantly sourced from the government or within the advisory committee as a set programme of work.
Across all six cases the reasons provided for the request to develop DRV ranged from acknowledged discrepancy between local DRV and those of other countries, DRV age, the emergence of new data, a set time cycle for the programme of work, information provision to assist policy option selection and triggers from monitoring practices, such as clinical health outcomes or markers of chronic disease risk (e.g. neural tube defects, LDL). However, details on the final trigger to initiate a request for particular nutrient DRV to be reviewed or developed were not always transparent, i.e. the influence of lobbying (professional, academic, commercial or special interest group) on initiating or suppressing any development of DRV. Similarly, the criteria for judging when DRV were ‘too old’, when there was ‘sufficient’ new evidence to initiate renewing DRV or at what point disease incidence constituted a public health issue to address were not always transparently documented.
The scope of request differed between cases. In the ANZ, PL and ES cases the scope of the request centred upon the utilisation and adaptation of international DRV to each respective country/region (predominantly the US/Canadian DRV in ANZ and PL; various international DRV in ES). This was clearly detailed in ANZ. The NHMRC was asked to assess the body of evidence used to establish the US/Canadian DRV (and any relevant literature that had subsequently been released) and provide an opinion on one of four courses of action: whether they should adopt, adopt with minor changes, adopt with substantial changes or reject the US/Canadian DRV in ANZ.
The remaining three cases (NL, NOC and UK) all referred to the importance of clarifying the scope of the request and iterative interactions between government representatives and advisory committee members during the nutrition problem formulation (e.g. to demarcate risk assessment/risk management activities or develop values for deficiency/adequacy/optimal/toxicity). Problem formulation was deemed necessary to ensure the request would be both appropriate and achievable (e.g. scientific limits of knowledge and resource constraints such as expertise, finance, time). Transparency appeared to be the greatest regarding the current UK vitamin D DRV development process. Here, the public was allowed to attend an open meeting where the scope of the request was discussed (also known as ‘terms of reference’ in the UK) and online access was made available for downloading detailed meeting minutes and discussion documents surrounding request clarification.
Process of DRV evidence evaluation
Several discussions were undertaken on how to evaluate and interpret evidence that contained methodological and theoretical uncertainties (e.g. folate assay method, folate equivalents, uncertainty factors, bioavailability factors, vulnerable population groups, mechanisms of action, multiple nutrient interactions). These discussions were present in the cases which adopted/adapted existing international DRV as well as those which developed their own DRV (Table 3). For example, the ANZ, PL and ES cases made reference to the evidence evaluation previously conducted in the original DRV as well as the additional evidence evaluation required to ensure adapted DRV were up-to-date and relevant to their respective country/region.
Table 3 Process of DRV evidence evaluation
DRV, Dietary Reference Values; NHMRC, Australian National Health and Medical Research Council; ANZ, Australia and New Zealand; ES, Spain; CBO, Dutch Institute for Healthcare Improvement; SIGN, Scottish Intercollegiate Guidelines Network; NL, Netherlands; NNR5, Nordic Council of Ministers’ Nordic Nutrition Recommendations, 5th edition; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); SACN, Scientific Advisory Committee on Nutrition; UK, United Kingdom.
“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.
To increase the transparency and scientific rigour of the evidence evaluation process, four cases (ANZ, NL, NOC, UK; Table 4) used protocols to guide evidence evaluation, address uncertainty in the data and limit interpretation bias. These evidence evaluation protocols differed slightly. Yet, all four cases provided guidance on the systematic reviewing of literature (including meta-analyses), the assessment of individual study quality/risk of bias and accounting for uncertainty when weighting or deciding the strength of the evidence base.
Table 4 Protocols used to assess study quality/risk of bias across cases (PL/ES unknown)
PL, Poland; ES, Spain; ANZ, Australia and New Zealand; NHMRC, Australian National Health and Medical Research Council; NL, Netherlands; CBO, Dutch Institute for Healthcare Improvement; SIGN, Scottish Intercollegiate Guidelines Network; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); NNR5, Nordic Council of Ministers’ Nordic Nutrition Recommendations, 5th edition; UK, United Kingdom; SACN, Scientific Advisory Committee on Nutrition; RCT, randomised controlled trial.
ANZ, NL and NOC used a series of checklists to assess study quality and a number of categories to differentiate between results from different study designs. The UK also utilised a series of checklists but discouraged the use of numerical grading. It was clearly acknowledged that a degree of subjective judgement was still required with the application of these protocols, especially when moving from the quality assessment of each study to collating the strength of the total evidence. Nevertheless, the following of a protocol for evidence evaluation and reporting of results was seen by these four cases as important steps towards a standardised approach to the evidence evaluation decision-making process.
Integration of evidence to develop final DRV
Interviewees across all cases acknowledged that evidence evaluation did not constitute the end of the DRV development process. Evidence evaluation findings needed to be appraised in the context of the original request to form the final DRV. Expert judgement was required to take account of the underlying certainty of the evidence, decide the strength of evidence and conclude with specific reference values. This stage of the process appeared to be less standardised with few transparent protocols available to aid the expert deliberation and consensus seeking decision-making process (Table 5).
Table 5 Integration of evidence to develop final DRV
DRV, Dietary Reference Values; NRV, Nutrient Reference Values; NHMRC, Australian National Health and Medical Research Council; ANZ, Australia and New Zealand; NNR, Nordic Council of Ministers’ Nordic Nutrition Recommendations; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); ES, Spain; NL, Netherlands.
“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.
The four cases which employed evidence evaluation protocols (ANZ, NL, NOC, UK) appended these protocols to reports and used them to guide descriptions of the strength and degree of certainty in the evidence underlying each DRV. In addition, prior to the publication of DRV reports, the standard practices of peer review (NL, NOC, PL) and invited or open consultation (ANZ, NL, NOC, PL, UK) provided an element of transparency and credibility to the process. Nevertheless, from the interviews it was clear that considerable discussions occurred between the evidence evaluation stage and the creation of the final DRV, even in relation to the wording or terminology used to describe DRV concepts (particularly PL) or the strength of evidence/degree of uncertainty (particularly UK). However, the details from many of these discussions were rarely communicated in the final report.
Two interviewees (ES, NL) specifically mentioned that the exact discussions undertaken by those developing DRV were not commonly reported. Reasons cited for this included a concern for the confidentiality of those who developed the recommendations as well as a belief that it was unnecessary to report every aspect of discussions that occurred during the deliberations before a consensus was reached. An exception was observed in NL when a disagreement of an advisory committee member could not be resolved during the formation of the folate DRV( 27 ). Consequently, a footnote was added to the Dutch report detailing a minority position of disagreement regarding the expression of folate v. folic acid as well as folate equivalent selection.
Discussion
Transparency and openness of the DRV process was not universally adopted across countries or areas of the DRV development process studied. Implementation of transparency policies may be limited by a lack of dedicated resources and best practice procedures, particularly to assist with the latter stages of reference value development.
Findings aligned with previous research regarding transparency in risk management activities( Reference Timotijevic, Barnett and Brown 20 ). The countries with dedicated advisory committees for risk assessment activities (ANZ, NL, NOC, PL and UK) demonstrated greater transparency of the DRV process than ES, where coordinated action at a national level was in its infancy and DRV had been developed by dedicated individuals. Arguably, NL, NOC and UK appeared to display the most transparency (also potentially ANZ, although the underpinning US/Canadian DRV setting process was not studied). Similarly to the above, this could be explained by a greater legal and clarified role of the relevant advisory committees and their responsibilities (demarcation between risk assessors, risk managers and risk communicators), a higher political priority or greater dedication of resources for nutrition and a longer tradition of DRV development. Furthermore, both NOC and UK were currently updating DRV at the time of study. Thus, the nature of transparency observed may have reflected the current increased calls for transparency and been associated with the most recent micronutrient DRV setting processes.
All cases demonstrated differences in transparency (in general and specific to the handling of uncertain evidence) across the three a priori areas studied (value request, evidence evaluation and final values). Based upon these findings, the status quo for current best practice regarding transparent and open handling of uncertainty in nutrition risk assessment has been discussed below.
Regarding the request to develop DRV, transparency was increased via documentation and accessibility to the iterative discussions between risk assessors, risk managers and others in problem formulation. These discussions acknowledged the uncertainty and limitations of the evidence base available to ensure the feasibility of the risk assessment task. This transparent detailing of the source, scope and reasons for any DRV request corresponded with guidance seen in a number of international risk assessment procedural documents (e.g. the nutrition problem formation stage before nutrition risk assessment activity in CODEX Alimentarius; defining the problem in the EURRECA framework for setting micronutrient recommendations)( 10 , Reference Dhonukshe-Rutten, Bouwman and Brown 14 ).
Transparency in risk assessment evidence evaluation is designed to minimise the probability of bias( Reference Lichtenstein, Yetley and Lau 28 ). Best practice for transparent handling of uncertainty in the cases studied referred to the documented protocols designed to guide and standardise the process of sourcing and interpreting evidence. Bias was reduced by employing an independent body to conduct the review and reproducibility was increased by employing standardised systematic literature review processes. Difficulties were still seen regarding the handling of uncertainty during evidence evaluation and communicating how this influenced the quantification/qualification of risk (e.g. the merits of grading different types of evidence). In addition, whether restricting a review to evidence from randomised controlled trials or meta-analyses limited the scope of the review or the utility of the final advice/reference values. However, the constant updating and disclosure of evidence evaluation procedures in the most recent cases reflected an ability to detail the overall strength of evidence reviewed.
Few protocols were available to assist with the articulation of, or guide the transition from, evidence evaluation to the integration of evidence and development of quantitative values/final DRV. This has been highlighted previously in the health-care area, where difficulties have been observed translating evidence evaluation into scientific advice for use in recommendation/guideline development( Reference Sackett, Rosenberg and Gray 29 , Reference Michie, Berentson-Shaw and Pilling 30 ). These findings support the requirement for further initiatives such as the EFSA consultation exercise in 2013 on how to conduct consistent procedures for expert elicitation during risk assessment( 31 ).
Cases demonstrated that consensus seeking was an essential stage of the DRV development process, designed to increase validity and reliability of decisions rather than relying upon an individual’s judgement. However, it was difficult to identify the best practice for transparently developing quantitative DRV; values that require a certainty not always apparent in the underlying data. The production of a transparent DRV report documenting the complexity of evidence-based expert decision making, the strength of advice and the areas of uncertainty remained a challenge. This had the potential to pose subsequent problems for the interpretation of DRV by those outside the advisory committee (e.g. responsible for setting micronutrient recommendations/relevant toxicology risk assessment activities)( Reference Innvaer 32 – Reference Dobrow, Goel and Lemieux-Charles 34 ).
The present study was based upon qualitative case studies. Therefore, the selection of micronutrients, countries and the three a priori defined areas of study may have limited external validity and the degree to which these findings can be transferred outside the sample studied. The authors recognise the study would have benefited from including analysis of other countries that develop micronutrient DRV, e.g. the USA/Canada. The exploratory nature of the present study justified the use of a qualitative design and steps were taken to limit biased interpretation. A common protocol and the three a priori defined areas of study were employed to maximise study rigour via clarity of the research goal and the consistent method of data collection, analysis and reporting. This also enabled the combination of data across countries. Construct validity was upheld by limiting the subjective collection of data, regular monthly research team meetings and performing traceable literature searches. Internal validity was upheld by encouraging several interpretations of the data to be considered. To maintain the cultural context and authenticity of the data, the majority of qualitative interpretation was conducted in the native language and information from interviews, observation or desk research provided the ability to cross-reference findings. The case studies (data collection, analysis and reporting) were led by the three a priori defined areas and did not represent the totality of the DRV development process. Nevertheless, they have provided an insight into the nature of transparency, in particular regarding the handling of uncertain data, available in an evidence-based decision-making process across multiple countries/regions.
Debate shall continue regarding the best practice for nutrition risk assessment, particularly handling uncertainty in the evidence surrounding nutrition, diet, lifestyle and health and rating the overall evidence underpinning DRV( Reference Russell 35 ). Future discussions may focus on adaptations to the risk analysis framework to facilitate increasing requirements for nutrition risk–benefit assessment in the development of DRV( 26 , Reference Tijhuis, de Jong and Pohjola 36 ). Furthermore, how to meet the challenge of incorporating developing evidence bases into risk assessment, such as those related to individual differences (e.g. metabolomics), non-randomised controlled trial study designs or whole-diet approaches (e.g. epidemiology)( Reference Baines, Cunningham and Leemhuis 15 ).
It may not be possible or necessary for all countries/regions to follow an aligned DRV risk assessment procedure nor for all reference values to be identical. However, transparency, as well as increasing the accountability and credibility of DRV development, can facilitate the sharing of best practice to inform the evolution of nutrition risk assessment. Therefore, the recent transparency initiatives from the EFSA, such as proposals to promote public access to risk assessment technical data, are to be welcomed( 37 ).
Conclusion
Implementation of transparency policies may be limited by a lack of dedicated resources and best practice procedures, particularly to assist with the latter stages of reference value development. Challenges remain regarding the best practice for transparently communicating the influence of uncertain evidence on the final reference values. Resolving this issue may assist the evolution of nutrition risk assessment and better inform the recommendation setting process.
Acknowledgements
Acknowledgements: The authors would like to acknowledge the contributions made to study design, data collection and analysis by Wojciech Roszkowski (WULS-SGGW). Financial support: The work herein was carried out within the EURRECA Network of Excellence (http://www.eurreca.org), financially supported by the Commission of the European Communities, Specific Research Technology and Development (RTD) Programme Quality of Life and Management of Living Resources within the Sixth Framework Programme, contract no. 036196. The Commission had no role in the design, analysis or writing of this article. This article does not necessarily reflect the Commission’s views or its future policy in this area. Conflict of interest: M.M.R. joined SACN in 2011, after the results in this paper were analysed. No other conflicts of interest have been declared by the authors. Authorship: The contributions of authors were as follows: K.A.B., L.d.W., L.T., A.-M.S., L.L., N.B.G., M.J.-B., E.S. and M.M.R. were responsible for the study concept and design. K.A.B., L.d.W., A.-M.S., N.B.G., M.J.-B., E.S. and A.N.M. conducted data collection, analysis and the composition of individual country summary reports. K.A.B. coordinated data collection and analysed the data across countries. K.A.B. drafted and wrote the manuscript. All authors were involved in manuscript revisions. Ethics of human subject participation: Ethical approval was not necessary for this body of work. However, this study was conducted according to the guidelines laid down in the Declaration of Helsinki. Written/verbal (witnessed and formally recorded) informed consent was obtained from all respondents, with all data made anonymous and held in accordance with the local data privacy laws.
