How reliable are randomised controlled trials for studying the relationship between diet and disease? A narrative review

Abstract

Large numbers of randomised controlled trials (RCT) have been carried out in order to investigate diet–disease relationships. This article examines eight sets of studies and compares the findings with those from epidemiological studies (cohort studies in seven of the cases). The studies cover the role of dietary factors in blood pressure, body weight, cancer and heart disease. In some cases, the findings from the two types of study are consistent, whereas in other cases the findings appear to be in conflict. A critical evaluation of this evidence suggests factors that may account for conflicting findings. Very often RCT recruit subjects with a history of the disease under study (or at high risk of it) and have a follow-up of only a few weeks or months. Cohort studies, in contrast, typically recruit healthy subjects and have a follow-up of 5–15 years. Owing to these differences, findings from RCT are not necessarily more reliable than those from well-designed prospective cohort studies. We cannot assume that the results of RCT can be freely applied beyond the specific features of the studies.

Randomised controlled trials (RCT) are a research tool of major importance for investigating the effects of nutrients and diets on the risk of disease. This encompasses primary prevention as well as treatment. RCT are often referred to as the ‘gold standard’, implying that the findings are highly reliable.

Cohort studies (prospective longitudinal studies) are an alternative method in terms of generating reasonably reliable information on the relationship between diet (and other lifestyle factors) and disease risk. Other types of epidemiological studies, such as case–control studies and cross-sectional studies, can also be used for investigating the relationship between diet and disease risk, but these are generally less reliable (i.e., they are more likely to generate false and misleading findings)^{( 1 )}.

In the hierarchal approach to the evaluation of evidence, a key feature of evidence-based medicine, RCT are widely judged as being of higher value than cohort studies^{( 2 )}. There is little argument that RCT are enormously valuable in many areas of medical research, such as the testing of drugs. However, it is debateable whether RCT are indeed as reliable as often assumed for carrying out research in the area of the relationship between diet and disease risk^{( 3 )}. It is also open to debate whether RCT are inherently more reliable than cohort studies for this type of research (see Table 1). The present study is a narrative review that aims to explore these questions.

Table 1 Advantages and disadvantages of randomised controlled trials (RCT) and cohort studies

This is especially the case in the area of determining the factors associated with disease causation. The findings of cohort studies have provided a wealth of information regarding how many lifestyle-related diseases can be prevented.

This review examines eight cases where RCT have been carried out in order to investigate a diet–disease relationship, either for prevention or for treatment. The findings from RCT are compared with those from epidemiological studies, of which seven out of the eight are cohort studies. The cases selected are those where there is sufficient evidence to allow inferences to be made with reasonable confidence. A critical evaluation of this body of evidence allows important lessons to be drawn regarding the reliability of RCT. The selection of cases and the evaluation of the evidence were based on the author’s knowledge and judgement rather than a systematic approach.

There is plenty of other evidence regarding the relationship between diet and risk of the diseases discussed here in addition to the findings from RCT and cohort studies. For example, many studies have investigated the mechanisms of disease aetiology that may help explain diet–disease relationships. However, that evidence is excluded from this article as it is usually too complex to provide useful insights into the questions explored in this study^{( 4 )}. For that reason, there is no discussion on topics such as the effect of diet on immune system functioning, on the processes involved in carcinogenesis and on the various hormones believed to be involved in appetite regulation, and thence obesity.

Studies of the relationship between diet and disease risk factors

In this section, the reliability of RCT is assessed with a focus on studies of the relationship between diet and either disease or risk factors for disease. Details of the RCT and of the comparator epidemiological studies (mainly cohort studies) are described in Table 2. The text provides a brief description of the key findings, which are then summarised in Table 3.

Table 2 Design of randomised controlled trials (RCT) and epidemiological studies and key findings*

BP, blood pressure; FU, follow-up; DASH, Dietary Approaches to Stop Hypertension; HR, hazard ratio; SSB, sugar-sweetened beverages; RR, risk ratio.

* In almost all cases, multivariate analysis was carried out on the data from the cohort studies. Data were adjusted for numerous possible confounding variables.

Table 3 Comparison of findings from randomised controlled trial (RCT) and epidemiological evidence

DASH, Dietary Approaches to Stop Hypertension; BP, blood pressure; SSB, sugar-sweetened beverages

* Epidemiological studies were all cohort studies except for the study of Na and BP, which was a cross-sectional study.

Case no 1: the Dietary Approaches to Stop Hypertension diet and blood pressure

The Dietary Approaches to Stop Hypertension (DASH) diet has been formulated with the goal of being a treatment to lower elevated blood pressure (BP). The diet includes relatively high contents of fruits and vegetables, moderate amounts of low-fat dairy products, fats and oils and relatively low contents of meat, snacks and sweets^{( 5 )}. RCT carried out on persons with elevated BP demonstrate that the diet brings about a significant lowering of BP^{( 5 )}. These findings are consistent with those from a cohort study that was conducted on 84 000 healthy women^{( 6 )}. The risk of developing hypertension was significantly lower in persons whose diet had a high score for its similarity to the DASH diet. However, the DASH score included Na intake, which is not a component of the DASH diet. The subjects in the cohort study had normal BP at baseline and were followed-up for 14 years, whereas those in the RCT had elevated BP and were given the DASH diet for only 7 weeks.

Case no. 2: sodium and blood pressure

Evidence has been steadily building for decades that the high salt content of the Western diet is causally associated with elevated BP. This has been investigated many times in RCT but with mixed results. The weight of evidence strongly indicates that lowering the intake of salt significantly reduces BP, provided that the duration is sufficiently long (weeks not days)^{( 7 , 8 )}.

These findings are supported by plenty of epidemiological evidence, which also demonstrates a strong association between salt intake and BP. An especially large cross-sectional study was recently carried out on 102 000 people residing in eighteen countries. These findings revealed a positive association between Na intake and BP^{( 9 )}. This study was notable in that the subjects were highly varied in their BP, body weight, ethnicity and the income level of the country in which they resided.

Case no. 3: sugar-sweetened beverages and body weight

There has been much interest in recent years concerning the possible role of sugar, especially when consumed as sugar-sweetened beverage (SSB), in the epidemic of obesity. Numerous RCT have been carried out that have investigated whether SSB do indeed contribute to excessive body weight. In recent reviews, it was reported that increasing the intake of SSB led to more weight gain in adults^{( 10 , 11 )}. Conversely, reduced intake in children led to less weight gain^{( 10 )}. These findings are reasonably consistent with those from the numerous cohort studies that have been carried out; these have mostly reported that for both adults and children a relatively high intake of SSB leads to more weight gain^{( 10 , 11 )}. The cohort studies had a much longer duration than the RCT (mostly several years in the cohort studies v. a few weeks or months in RCT on adults or 6–18 months in RCT on children).

Case no. 4: whole grains, cereal fibre and body weight

Dietary fibre has long been viewed as the ‘opposite’ of dietary fat in that fibre is believed to enhance satiety, and thereby help protect against excessive energy intake and thence obesity. Numerous RCT have been carried out that have investigated whether an increased intake of cereal fibre (e.g., by adding whole grains to the diet) aids loss of body weight. Findings have been mostly negative^{( 12 )}. One aspect of the experimental design that may have been responsible for the failure of cereal fibre to reduce weight is that many studies had a relatively short period of follow-up (≤16 weeks). In several studies, fibre was added to an energy-reduced diet, and as in the case of dietary fat this may have masked an effect of fibre. Findings from three cohort studies suggest that a relatively high intake of whole-grain cereals (and cereal fibre) is associated with less weight gain over the next several years^{( 13 – 15 )}. This comparison of the two sets of findings reveals a complete lack of concordance.

In all, four cases have now been considered that looked at studies of diet and health outcomes. The findings from RCT appear to have generated reliable results in the first three cases but not in the final one. The first three cases included two that looked at BP and one at body weight, whereas the final case was on body weight. The explanation for these inconsistencies is far from clear; one factor that may be important is the duration of the RCT. The RCT considered here mostly had duration of a few weeks. This appears to be sufficient time for the BP to respond to the modified diet but may not always be long enough for the altered diet to affect body weight.

Studies on the relationship between diet and disease

In this section, the reliability of RCT is assessed with a focus on studies on the relationship between diet and occurrence of actual disease.

Case no. 5: multi-vitamin supplements

There has been much debate regarding whether healthy adults should take a one-a-day, multi-vitamin, multi-mineral supplement. RCT carried out on well-nourished adults suggest that supplementation provides no benefits in terms of improved health or the prevention of disease^{( 16 )}. Consistent with this, findings from several cohort studies demonstrate that persons who consume such supplements do not have a reduced all-cause mortality^{( 17 – 21 )}.

Case no. 6: selenium and cancer

The possible preventive action of Se against cancer has been under investigation since the early 1970s. Several RCT have been carried out in which subjects have been given supplemental Se. A meta-analysis of nine RCT reported a reduced risk of cancer^{( 22 )}. The RCT included both primary and secondary prevention interventions. The follow-up period in the larger trials was 5–6 years; one of the largest RCT included in the analysis is of particular interest. Supplemental Se failed to prevent the development of prostate cancer in the SELECT study (Selenium and Vitamin E Cancer Prevention Trial), a primary prevention trial^{( 23 )}. However, the baseline serum level of Se was significantly higher in that study than in the other RCT. It has been proposed that supplemental Se prevents cancer but only in persons with a relatively low dietary intake of the mineral^{( 22 , 24 )}.

Several cohort studies have reported an inverse association between Se status (such as the level of the mineral in blood or toenails) and risk of cancer, especially for cancer of the lung and prostate^{( 24 , 25 )}. These findings indicate that a relatively low dietary intake of Se increases the risk of cancer. This comparison of the findings from RCT and cohort studies reveals quite good concordance, but only if one accepts the above-stated explanation for the negative results from the SELECT study.

Case no. 7: dietary fibre and colorectal cancer

The hypothesis has been debated for 40 years that dietary fibre is protective against colorectal cancer. Several RCT have been carried out^{( 26 )}. In each case, the reported outcome was the recurrence of colorectal adenomas in persons with a history of the disorder. Wheat bran was the most commonly used source of fibre but other sources were also used (as well as additional dietary changes in some trials). These RCT have failed to detect any degree of protection by supplemental sources of fibre. Evidence from cohort studies points to fibre as being protective against colorectal cancer. The evidence is strongest for total intake of fibre as well as for cereal fibre and intake of whole-grain cereals but is much weaker for fibre from fruit and vegetables^{( 27 )}.

It appears, therefore, that the results from RCT are inconsistent with the findings from cohort studies. However, both sets of findings should be viewed cautiously. There are several possible explanations for the negative findings seen in RCT; in particular, the follow-up period is far shorter than in cohort studies (2–4 v. 5–17 years), the subjects had a history of colorectal adenomas and the end point was recurrence of adenomas rather than cancer. The negative findings from RCT are best interpreted not as the ‘gold standard’ that disproves the evidence from cohort studies but rather as a lesson that RCT should be evaluated cautiously; it may often be unjust to extrapolate the findings beyond the specific features of the studies. The cohort studies are probably best characterised as indicating that a high-fibre diet (rather than fibre per se) is protective against colorectal cancer. A relatively high intake of fibre is associated with a healthy diet and a generally healthy lifestyle^{( 14 )}. For that reason, we cannot ignore the possibility that residual confounding may provide a partial explanation.

Case no. 8: fish, fish oil and heart disease

The preventive benefit of fish, especially fish with high fat content, against the development of heart disease has attracted much attention. It is commonly believed that long-chain n-3 fatty acids are mainly responsible for this benefit^{( 28 )}. The possible value of fish or fish oil has been tested numerous times in RCT but the findings indicate that the reduction in risk of cardiac death is fairly small (approximately 9 %) and of borderline statistical significance^{( 29 )}. In contrast, a large body of evidence from cohort studies provides consistent and compelling evidence that persons who regularly eat fish are at significantly reduced risk of cardiac death (approximately 16–21 % lower) than are people who seldom eat fish^{( 30 )}.

There are several possible explanations for these contrasting results. The most plausible ones are as follows: (1) most subjects in the RCT had a history of cardiac disease, whereas the cohort studies recruited only healthy people (2) the follow-up period was far shorter in RCT (median of 2 years) than in cohort studies (mostly 12–20 years) and (3) most RCT provided fish oil rather than fish. It is possible that fish contains cardioprotective substances that are not present in fish oil.

A similar conclusion that was made in the previous case therefore also applies here. The findings from RCT are best interpreted not as the ‘gold standard’ that demonstrates that fish oil has only a weak protective effect against heart disease, and not the much larger protection indicated by cohort studies, but rather as a lesson that RCT should be evaluated cautiously; it may often be unjust to extrapolate the findings beyond the specific features of the studies. James et al.^{( 31 )} recently made a detailed analysis of the likely reasons that explain the findings from RCT.

Table 3 summarises the findings from the eight cases.

Possible limitations of randomised controlled trials

Why are findings from RCT often contradicted by the results of cohort studies? We look at several possible explanations in this review. Cohort studies are prone to certain types of error^{( 1 )}. Misleading findings from cohort studies are therefore the likely explanation for some of the cases. However, there are other possible explanations that merit serious consideration. Some of these are considered below.

1. 1. The persons recruited to cohort studies have typically followed a fairly stable dietary pattern for most of their adult lives. RCT, by contrast, usually last only a few weeks or months, although some last a few years. Therefore, if a dietary agent prevents a disease but only if given for 10 or 20 years, this will probably be seen in cohort studies but not in RCT. The possible importance of this variable was demonstrated by the Trials of Hypertension Prevention (TOHP)^{( 32 )}. In these two RCT, subjects with pre-hypertension were advised to follow a Na-reduced diet, either alone (TOHP I) or combined with weight reduction (TOHP II). The interventions led to a significant decrease in risk of CVD, but this required follow-up for at least 8 years.

2. 2. Cohort studies generally recruit healthy people. However, the subjects recruited to RCT are typically at relatively high risk of the disease under study. For example, the subjects recruited to cancer studies may be smokers or have other characteristics that place them at relatively high risk of cancer. Most RCT on heart disease have been carried out on people who already have heart disease. Therefore, if a dietary agent prevents a disease but only when given at an early stage of disease development, positive findings will be seen in cohort studies but not in RCT.

3. 3. Supplements of a nutrient may only achieve positive results in RCT when most subjects in the study have a poor intake of the nutrient. In several studies a likely explanation for a lack of benefit of supplementation is that only a small minority of subjects had a low intake. As mentioned earlier, this is the likely explanation for the failure of supplemental Se to prevent prostate cancer in the SELECT study. Likewise, the very low success of supplements of vitamins C and E and of β-carotene at preventing disease is most likely because the large majority of subjects in these RCT had sufficiently high intakes of these nutrients, and as a result supplements provided very few health benefits. Another example is that of folic acid and stroke. RCT indicate that supplements of folic acid prevent stroke but only in populations where food is not fortified with the vitamin^{( 33 )}.

On the basis of these considerations, we should be cautious before concluding that the findings from RCT disprove the findings from cohort studies. Instead, the findings from a RCT may often not be generalisable beyond its specific design features.

Ideally, RCT should more often use healthy subjects and have a relatively long period of follow-up. Unfortunately, such studies are prohibitively expensive. In particular, it takes several times more healthy subjects to generate enough clinical end points than when RCT are carried out on persons at high risk of the disease under study. The enormous cost of well-conducted RCT is demonstrated by the Women’s Health Initiative. This study recruited more than 160 000 women and investigated the risks and benefits of hormone treatment at menopause, supplements of Ca and vitamin D and of low-fat diets. The cost of the study was $625 million^{( 34 )}. In addition to cost, carrying out RCT on healthy subjects and using a long period of follow-up can also create other challenges, notably a high dropout rate and poor compliance.

Limitations of the study

The findings reported here are based on eight cases where RCT have been carried out in order to investigate a diet–disease relationship. The cases were selected where there is sufficient evidence from both RCT and epidemiological studies to allow comparison of the two sets of findings and for inferences to be made with reasonable confidence. However, the selection of the cases and the evaluation of the evidence were carried out on the basis of the author’s knowledge and judgement rather than a systematic approach; because of these methodological limitations and also because of the small number of cases examined, the conclusions should therefore be seen as tentative.

Discussion

The evidence reviewed here indicates that we need to show much caution before assuming that the results of RCT can be freely applied beyond the specific features of the studies. In particular, the health/disease history of the subjects and the length of follow-up may have a major impact on the outcome. Very often RCT recruit subjects with a history of the disease under study (or at high risk of it) and have a follow-up of only a few weeks or months. Cohort studies, in contrast, typically recruit healthy subjects and have a follow-up of 5–15 years. For these reasons, therefore, it is a mistake to assume that the findings from RCT are more reliable than those from cohort studies. There is a need for more in-depth investigation of the reliability of RCT that investigate diet–disease relationships.

Blumberg et al. ^{( 3 )} also argued that RCT are inherently flawed when used for nutrition research, especially for studies of individual nutrients. They pointed out that RCT are well suited for use in drug trials but that studies of nutrients pose very different challenges. It follows, therefore, that decisions that have an impact on public health, such as developing policies or presenting advice to the public, should not place heavy reliance on RCT, especially those of short duration carried out on subjects with a history of the disease under consideration.

The findings reported in this study suggest improvements that could be made in the reliability of RCT and cohort studies when used for investigating the effects of nutrients and diets on the risk of disease. RCT are clearly a very valuable tool. Indeed, several of the cases looked at in this study are best interpreted as indicating that the findings are accurate, especially those cases where the findings from RCT are strongly consistent with those from cohort studies. However, the problems highlighted in this study point to the need for several improvements, both in methodology and reporting. RCT would be of greater value if their design more closely resembled the actual health problem that they were investigating. Thus, if an RCT is testing whether nutrient X prevents disease Y, then, ideally, the subjects should not already have disease Y and the follow-up period needs to be sufficiently long. However, as pointed out earlier, the cost of such studies can be prohibitive. The reporting of RCT needs to be improved; specifically, the abstract and discussion sections of papers should stress that the findings can only be interpreted within the design features of the study. Therefore, for example, if the study subjects had already been diagnosed with disease Y and were then treated with nutrient X, it should be stressed that the results may not be relevant as to whether nutrient X can achieve primary prevention of disease Y. Likewise, systematic reviews and meta-analyses need to apply the same approach in the interpretation of findings from RCT. In brief, the findings from RCT should not be extrapolated beyond what is justified by the design features of the study.

Findings from cohort studies have been used in this paper as a useful comparator for assessing the reliability of the findings from RCT. However, it is stressed that cohort studies also have their own inherent sources of error.

A major limitation of cohort studies is the problem of confounding. This is caused by the fact that many dietary and lifestyle factors occur in close association. For example, intake of both cereal fibre^{( 13 )} and fish^{( 35 )} are associated with a healthy diet and a generally healthy lifestyle. Adding to this problem many dietary components that may play an important role in preventing disease are present in the same foods. For example, many types of fruit and vegetables are rich in vitamin C, folate, K and various carotenoids. This problem is addressed by multivariate analysis, but whether this completely solves the problem of confounding is not known. For that reason, residual confounding may be a significant source of error. This problem is illustrated by the following example. A survey of cohort studies reported that one-third of them failed to include any measure of socio-economic status (SES) in multivariate analyses^{( 36 )}. Moreover, where SES was included in the analysis, this was usually done using education as the indicator of it, whereas income and social class were seldom determined. Whether this is a source of significant error is not known.

Another important concern with cohort studies is the accuracy of diet recalls. It is well recognised that these have sources of significant error. For example, estimates of alcohol and energy intakes are usually substantially underestimated in heavy drinkers and obese persons, respectively. The most suitable methodology for diet assessment has been much debated^{( 37 )}; one issue is the use of repeated measures of usual dietary intake, say every 4 years. This can provide a more accurate assessment of the usual diet of subjects over the many years between baseline assessment and the end of follow-up. Stringhini et al. ^{( 38 )} found that repeated measures of diet explained much more of the association between socio-economic status and mortality than was the case when diet was assessed only at baseline. The accuracy of dietary assessment can often be enhanced by use of biomarkers. For example, the Se content of tissues such as toenails has been used as an indicator of the long-term intake of the mineral. This approach is potentially more accurate than diet assessment as it does not rely on the subject’s memory.

Despite these limitations of cohort studies, there is little doubt that they are of great value. This is especially the case in the area of determining the factors associated with disease causation. The findings of cohort studies have provided a wealth of information regarding how many lifestyle-related diseases can be prevented.

On the basis of the above considerations, RCT should cease to be regarded as the summit of the evidence hierarchy in the area of diet–disease relationships. Instead, a new approach is needed that bases conclusions on the overall body of evidence. Synthesis of evidence may be compared with a three-legged stool. The three legs are (i) RCT; (ii) cohort studies (combined with case–control studies) and (iii) other evidence (such as cross-sectional studies, ecological evidence and historical evidence). It is emphasised that overall conclusions must consider the entire weight of the evidence, paying close attention to the perceived reliability of each component.