INTRODUCTION
Over the last few decades, patients and their family members are increasingly being involved in care decisions (Colligan, Metzler, & Tiryaki, Reference Colligan, Metzler and Tiryaki2017; Kane, Halpern, Squiers, Treiman, & McCormack, Reference Kane, Halpern, Squiers, Treiman and McCormack2014). To be able to make informed decisions, patients need sufficient cognitive abilities and self-awareness, but these functions can be disturbed in patients with brain disorders (e.g., Day et al., Reference Day, Gillespie, Rooney, Bulbeck, Zienius, Boele and Grant2016; Kerrigan, Erridge, Liaquat, Graham, & Grant, Reference Kerrigan, Erridge, Liaquat, Graham and Grant2014; Sunderaraman & Consentino, Reference Sunderaraman and Consentino2017). More specifically, evidence emerging from patient studies, imaging research, and behavioral paradigms suggests that different components of executive functioning (EF) play a crucial role in decision-making and that impairments in EF seem to be associated with risky and ambiguous decision-making (Del Missier, Mäntylä, & Bruine de Bruin, Reference Del Missier, Mäntylä and Bruine de Bruin2010; Gleichgerrcht, Ibáñez, Roca, Torralya, & Manes, Reference Gleichgerrcht, Ibáñez, Roca, Torralya and Manes2010; Schiebener, Wegmann, Gathmann, Laier, Pawlikowski, & Brand, Reference Schiebener, Wegmann, Gathmann, Laier, Pawlikowski and Brand2014). Executive functions include several higher-order cognitive processes that enable people to control and regulate their own behavior. Key executive functions are inhibition (i.e., deliberate overriding of dominant responses), cognitive flexibility (i.e., shifting between different tasks or mental sets), and working memory (i.e., constant monitoring and updating of retained information) (Diamond & Ling, Reference Diamond and Ling2016; Karr et al., Reference Karr, Areshenkoff, Rast, Hofer, Iverson and Garcia-Barrera2018; Miyake & Friedman, Reference Miyake and Friedman2012). With respect to decision-making, executive functions are necessary for, among others, assessing probabilities, categorizing options, selecting decision-making strategies, and using feedback to revise strategies if necessary (Schiebener et al., Reference Schiebener, Wegmann, Gathmann, Laier, Pawlikowski and Brand2014).
Impairments in EF are among the most pronounced cognitive deficits in patients with primary brain tumors (Gehrke, Baisley, Sonck, Wronski, & Feuerstein, Reference Gehrke, Baisley, Sonck, Wronski and Feuerstein2013; Meskal, Gehring, Rutten, & Sitskoorn, Reference Meskal, Gehring, Rutten and Sitskoorn2016; Talacchi, Santini, Savazzi, & Gerosa, Reference Talacchi, Santini, Savazzi and Gerosa2011; Van Kessel, Baumfalk, Van Zandvoort, Robe, & Snijders, Reference Van Kessel, Baumfalk, Van Zandvoort, Robe and Snijders2017). Impairments in EF can have a negative impact on patients’ everyday lives but also on the lives of the people in their environment (Aaronson et al., Reference Aaronson, Taphoorn, Heimans, Postma, Gundy, Beute, Slotman and Klein2011; Nugent et al., Reference Nugent, Weimer, Choi, Bradley, Bender, Ryan, Gardner and Sherwood2014; Sterckx et al., Reference Sterckx, Coolbrandt, Dierckx de Casterle, Van den Heede, Decruyenaere, Borgenon, Mees and Clement2013). It is important to identify problems with EF at an early stage, so we can refer to intervention programs in time, including, for example, Goal Management Training (Richard et al., Reference Richard, Bernstein, Mason, Laperriere, Maurice, Millar, Shultz, Berlin and Edelstein2019).
From research and clinical practice, it is known that patients who experience cognitive complaints do not necessarily show lower scores on neuropsychological tests. And vice versa, if patients demonstrate lower test performance, they do not always experience cognitive complaints in their daily life (Gehring, Taphoorn, Sitskoorn, & Aaronson, Reference Gehring, Taphoorn, Sitskoorn and Aaronson2015; Hutchinson, Hosking, Kichenadasse, Mattiske, & Wilson, Reference Hutchinson, Hosking, Kichenadasse, Mattiske and Wilson2012). Therefore, both neuropsychological tests and self-report questionnaires on EF are often used to get a full picture of a person’s EF. When patients are unable to complete assessments, due to, for example, language problems or paralysis, information from informal caregivers can be of added value. When using proxy-report as an addition to, or as a substitute for self-report, it is crucial to know how these measures are related.
A few studies in brain tumor patients evaluated patient–proxy agreement with respect to patients’ quality of life. Overall, moderate to high levels of concordance were observed, if patients did not suffer from cognitive dysfunction (Brown et al., Reference Brown, Decker, Rummans, Clark, Frost, Ballman, Arusell and Buckner2008; Ediebah et al., Reference Ediebah, Reijneveld, Taphoorn, Coens, Zikos, Aaronson, Heimans, Bottomley and Klein2017; Giesinger et al., Reference Giesinger, Golser, Erharter, Kemmler, Schauer-Maurer, Stockhammer, Muigg, Hutterer, Rumpold and Holzner2009; Sneeuw et al., Reference Sneeuw, Aaronson, Osoba, Muller, Hsu, Yung, Brada and Newlands1997). Also, high congruence was found in the study of Armstrong et al. (Reference Armstrong, Wefel, Gning, Acquave, Vera-Bolanos, Gilbert, Cleeland and Mendoza2012) on a brain tumor symptom checklist MD Anderson Symptom Inventory – Brain Tumor Module that was administered in 115 brain tumor patients and their caregivers. Additionally, a study in 60 brain tumor patients showed high level of agreement between patients and proxies on the Dexamethasone Symptom Questionnaire-Chronic (Agar et al., Reference Agar, Koh, Gibbs, Barnes, Hovey, Livingstone, Sawkins, Chye, Lovell, Clark, Vardy and King2016). By contrast, the study of Rooney and colleagues demonstrated substantial disagreement between glioma patients and their proxies in the evaluation of depressive symptoms (Rooney et al., Reference Rooney, McNamara, Mackinnon, Fraser, Rampling, Carson and Grant2013), with proxies reporting more depressive symptoms than patients themselves. These authors suggested that patients and proxies generally seem to agree on objective signs and overt behavior of patients, but that agreement is limited on subjective, internal symptoms, such as mood and emotional functioning (Rooney et al., Reference Rooney, McNamara, Mackinnon, Fraser, Rampling, Carson and Grant2013).
It is important to know how brain tumor patients perceive their EF and how this relates to the experience of their informal caregivers and to performance-based measures of EF. In the current study, the main objective was to evaluate the level of agreement between patient-report and proxy-report of patients’ EF. Second, associations of reported measures of EF with performance-based measures of EF (i.e., neuropsychological tests) were examined, as well as the influence of these performance-based measures on the level of patient–proxy agreement. Since from previous studies it is known that brain tumor patients suffer from executive deficits, we hypothesized that this may also influence their self-assessment, potentially resulting in disagreement among patients and proxies. We also expected that lower patients’ scores on performance-based measures would be associated with greater patient–proxy discrepancies.
METHOD
Design and Procedure
Data of this study were gathered at preintervention assessments in a feasibility study (Van der Linden, Sitskoorn, Rutten, & Gehring, Reference Van der Linden, Sitskoorn, Rutten and Gehring2018a) and a randomized controlled trial (RCT; Van der Linden, Sitskoorn, Rutten, & Gehring, Reference Van der Linden, Sitskoorn, Rutten and Gehring2018b) on cognitive rehabilitation in brain tumor patients, initiated at the Elisabeth-TweeSteden Hospital Tilburg, The Netherlands. This research was conducted in accordance with the Declaration of Helsinki (World Medical Association, 2013). The cognitive rehabilitation study (i.e., feasibility study and RCT) was approved by the local ethical review board (METC Brabant: NL 51152.028.14) and registered with clinicaltrials.gov (NTC03373487) and the Dutch Trial Register (NTR5392).
Before and 3 months after surgery, patients underwent neuropsychological assessment (NPA) as part of usual clinical care in the hospital. Directly after the 3-month NPA, patients who participated in the RCT were randomized to an intervention or control condition. During an appointment with the researcher 3 months after surgery, participants and proxies completed the adult version of the Behavior Rating Inventory of Executive Function (BRIEF-A).
Participants
Adult patients who were scheduled for resective surgery for a meningioma or low-grade glioma were invited to participate in one of the cognitive rehabilitation studies. Patients undergoing only biopsy were not eligible. Exclusion criteria were tumor resection in the last year; chemotherapy or radiotherapy in the last 2 years; presence/history of progressive neurological disease; severe psychiatric disorder or substance abuse; diagnosis of acute neurological or mild psychiatric disorders in the last 2 years (e.g., cerebrovascular accident); multiple (>1) tumors; lack of basis proficiency in Dutch; Karnofsky Performance Score below 70; IQ below 85 or (very) low cognitive skills; and insufficient reading skills, visual impairment, or motor impairment. Patients were also excluded if they had severe surgery-related complications or if they were referred to formal cognitive rehabilitation.
Study participants were invited to involve an informal caregiver (e.g., spouse, family member, or close friend) to the study. No exclusion criteria were applied to the proxies. If patients agreed, informed consent was obtained from the proxies. Only participants who involved an informal caregiver were included in the present analyses.
Measures
The BRIEF-A assesses problems in an adult’s EF as experienced in his or her daily life (Roth, Isquith, & Gioia, Reference Roth, Isquith and Gioia2005). The self-report and informant-report versions are available, and both questionnaires consist of 75 items. Patients and proxies were asked to assess the extent to which certain behavior of the patient occurred during the past month. The informant version of the BRIEF-A uses a proxy–proxy perspective: proxies need to report on how they think the patient is functioning. Answers are given on a three-point scale (i.e., never = 1, sometimes = 2, often = 3). Two index scores, Behavioral Regulation and Metacognition, can be calculated based on 30 and 40 items, respectively. Behavioral Regulation contains four subscales (i.e., Inhibit, Shift, Emotional Control, and Self-monitor), and Metacognition has five subscales (i.e., Initiate, Working Memory, Plan/Organize, Task-monitor, and Organization of Materials). Three validity scales (i.e., Negativity, Infrequency, and Inconsistency) were checked, and in case of invalid responses, patients were excluded from analyses. Cases were also excluded if there were ≥5 missing answers. Missing values were handled in accordance with the manual (Scholte & Noens, Reference Scholte and Noens2011) and scores of “1” were imputed in case of missing values. Raw scores on the indices were converted into T scores, using representative norms of the Dutch/Flemish population (self-report: n = 1600, informant-report: n = 1082) (Scholte & Noens, Reference Scholte and Noens2011). These standardized T scores were converted into Z scores and also reversed, to coincide with performance-based scores, so that lower Z scores indicate lower subjective EF. Z scores ≤−1.5 were considered as low (Scholte & Noens, Reference Scholte and Noens2011). Psychometric properties of the Dutch version of the self-report and informant-report are good, with Cronbach’s α above 0.90 and intraclass correlation coefficients (ICCs) between 0.73 and 0.81, for the two indices (Scholte & Noens, Reference Scholte and Noens2011).
Three neuropsychological tests from the 3-month assessment were included to measure different aspects of performance-based EF. Response inhibition and cognitive flexibility were measured, respectively, with the Stroop Test and the Shifting Attention Test (SAT) of the computerized test battery Central Nervous System Vital Signs (CNS VS LCC, Morrisville, NC, USA). Regarding validity and reliability, Gualtieri and Johnson (Reference Gualtieri and Johnson2006) concluded that psychometric characteristics of the CNS VS’ tests were comparable with the conventional tests on which they were based (Gualtieri & Johnson, Reference Gualtieri and Johnson2006). Additionally, previous studies demonstrated sufficient sensitivity of the CNS VS in the detection of (mild) cognitive deficits and change in cognitive function in patients with brain tumors (Meskal et al., Reference Meskal, Gehring, van der Linden, Rutten and Sitskoorn2015; Rijnen et al., Reference Rijnen, Meskal, Bakker, De Baene, Rutten, Gehring and Sitskoorn2019; Van Loenen et al., Reference Van Loenen, Rijnen, Bruijn, Rutten, Gehring and Sitskoorn2018). Furthermore, the Digit Span Backward of the Wechsler Adult Intelligence Scale was used to assess working memory (Wechsler, Reference Wechsler2015). Patients’ scores on the neuropsychological tests were compared to representative, recently collected Dutch norms (Rijnen et al., Reference Rijnen, Meskal, Emons, Campman, Van Der Linden, Gehring and Sitskoorn2017; Rijnen, Van der Linden, Emons, Sitskoorn, & Gehring, Reference Rijnen, Van der Linden, Emons, Sitskoorn and Gehring2018; CAR study A, ClinicalTrials.gov, reference nr. NCT02953756). Patient test scores were corrected for sex, age, educational level, and practice effects and converted into Z scores, with lower scores indicating worse EF performance. Again, Z scores ≤−1.5 were considered as low.
Data Analysis
Descriptive statistics of the sample were calculated and compared with patients who were not included in the current study (i.e., patients who did not include a proxy in the cognitive rehabilitation study) using independent-samples t-tests and Chi-square tests.
Mean scores of patients and proxies were compared with normative values [M = 0, standard deviation (SD) = 1] using two-tailed one-sample z-tests. The standardized mean differences between patients and the normative values, and those between proxies and normative values, are interpreted as Glass’ delta effect sizes, with 0.20–0.49 indicating small effects, 0.50–0.79 medium effects, and ≥0.80 reflecting large effects (Glass, McGaw, & Smith, Reference Glass, McGaw and Smith1981). Subsequently, means of the patients were compared with means of the proxies using two-tailed paired-sample t-tests, to investigate potential systematic differences, and effect sizes [Cohen’s d (Cohen, Reference Cohen1988)] were calculated.
To examine the level of concordance between patient- and proxy-reports, Lin’s concordance CCCs (Lin, Reference Lin1989, Reference Lin2000) were calculated for the two indices of the BRIEF-A. CCCs were interpreted as follows: 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.81–1.00 almost perfect to perfect agreement (Altman, Reference Altman1991). As described in the manual of the BRIEF-A (Scholte & Noens, Reference Scholte and Noens2011), in order to be able to consider patient-report and proxy-report as interchangeable, ICCs of at least 0.60 are required. Bland–Altman plots were generated, plotting the difference between measurements (i.e., patient score – proxy score) against the mean of the two measurements [i.e., (mean patient score + mean proxy score)/2] (Bland & Altman, Reference Bland and Altman1999). By visualizing these differences, patterns of agreement, types of bias, and outliers can be further identified. We computed 95% confidence envelopes around the mean difference (i.e., mean ± 1.96 × SD of difference scores), which shows the range of difference scores that covers 95% of the observations. Patient and proxy scores can be used interchangeably if the range is small given the application envisaged. Furthermore, patient and proxy scores on the two indices were dichotomized denoting low (Z score ≤ −1.5) or normal (Z score > −1.5) EF. Percentages of cases below the cut-off were calculated, and percentages of dyadic agreement about the presence or absence of reported EF impairment were determined.
Furthermore, performance-based measures of EF were examined, with standardized mean scores of the patients being compared to normative values (M = 0, SD = 1) with z-tests. The relationship between performance-based measures of EF and reported EF was explored by calculating Pearson’s product moment correlation coefficients between standardized performance scores and index scores of both patients and proxies. Correlation coefficients between 0.10 and 0.29 were considered as small, 0.30–0.49 were considered as medium, and 0.50–1.0 reflected large correlation coefficients (Cohen, Reference Cohen1988).
Finally, the association between objective EF and the level of agreement was examined using multiple regression analysis. The observed difference scores (patient self-report – proxy-report score) on the two BRIEF-A indices served as the dependent variables. All independent variables (standardized scores on the Stroop Test, SAT and Digit Span Backward of the 3-month NPA) were entered at the same time for each of the two analyses. Assumptions were evaluated using Durbin–Watson tests (Durbin & Watson, Reference Durbin and Watson1951), scatterplots and histograms of residuals, variance inflation indices (not exceeding 0.80 and 10; Field, Reference Field2009), and Cook’s distances (≤1; Cook & Weisberg, Reference Cook and Weisberg1982).
Statistical analyses were conducted using SPSS Statistics (version 24.0, IBM corporation, Armonk, NY, USA). All statistical tests were performed at an α level of 0.05.
RESULTS
Participants’ Characteristics
Three months after surgery, 47 out of 75 participants (63%) of the cognitive rehabilitation studies chose to involve a proxy. Regarding age, years of education, sex, tumor histology, and patient-reported EF, no significant differences were observed between patients who did include a proxy and those who did not (p-values > .05). Included patients (n = 47) had a mean age of 51 years and 26 patients (55%) were female. Twenty-eight patients (60%) were diagnosed with a meningioma and 19 (40%) with a low-grade glioma. In 26 patients (55%), the tumor was located in the frontal lobe. In the large majority (94%), the relationship of the proxy to the patient was partner. Three patients involved their mother, sister, or brother.
All BRIEF-A questionnaires met the validity criteria. There were no questionnaires with ≥5 missing values. In this study, Cronbach’s α of the indices ranged between 0.90 and 0.96 in patients and proxies, indicating good internal consistency (Gliem & Gliem, Reference Gliem and Gliem2003).
Patient-Report and Proxy-Report of Patients’ EF
Patients scored on average significantly lower on the Metacognition Index in comparison with normative values (M = −0.47, SD = 1.04, z = −3.25, p = .001). Glass’ delta effect size of 0.47 suggested a small effect. On the Behavioral Regulation Index, the average patient score did not significantly differ from normative values (M = 0.02, SD = 1.01, z = 0.13, p = .896). There were no significant differences between proxy evaluations of patients’ EF and normative values for both indices (Table 1).
Table 1. Mean levels of reported EF: Comparison with normative values and differences between patients (n = 47) and proxies (n = 47)
a Lower scores indicate lower reported EF/more experienced problems.
b Sample means compared to normative values (Mean = 0, SD = 1).
* p < .05
When comparing mean patient scores with mean proxy scores using paired-sample t-tests (Table 1), a significant mean difference was observed on the Metacognition Index. Patients reported significantly lower EF (i.e., more concerns) than their proxies (t(46) = −2.40, p = .021, Cohen’s d = −0.35), with Cohen’s d indicating a small effect. On the Behavioral Regulation Index, no significant mean difference was observed between patients and proxies (t(46) = −0.83, p = .703, Cohen’s d = −0.06).
On the Metacognition and Behavioral Regulation Index, 23% and 9% of the patients scored below the cut-off (Z score ≤ −1.5), respectively. According to the report of the proxies, 13% and 6% of the patients would score below the cut-off of the Metacognition Index and Behavioral Regulation Index, respectively.
Level of Agreement Between Patients and Proxies
Lin’s CCCs, listed in Table 2, indicated moderate to substantial agreement among patients and proxies (CCCs of 0.57 for Metacognition and 0.61 for Behavioral Regulation). Figure 1 shows the Bland–Altman plots. The lower and upper limits were −2.11 and 1.47 for Metacognition and −1.76 and 1.67 for Behavioral Regulation. The Bland–Altman plots showed that 93.6% of the data points (i.e., 44/47) lied within these limits of agreement. Inspection of the plots revealed no particular trends in agreement with respect to the level of reported EF. When the cut-off for reported EF impairment (i.e., Z score ≤−1.5) was applied, congruence was observed in 94% (44/47) dyads on Behavioral Regulation and in 77% (36/47) dyads on Metacognition (Table 3). Based on the dichotomized data of the Metacognition Index, 8 proxies of the 11 nonagreeing dyads did not rate the patient’s EF as impaired, whereas the patient did (Table 3).
Table 2. Level of agreement between patient-report and proxy-report of EF
a CCC = Concordance correlation coefficient. 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect to perfect agreement (Altman, Reference Altman1991).
b Limits of agreement: Meandiff ± 1.96*SDdiff.
Fig. 1. Bland–Altman plot for agreement on the (a) Behavioral Regulation Index, and (b) Metacognition Index. Note: Dashed lines represent the upper and lower limits of the 95 % confidence intervals.
Table 3. Dyadic agreement on the presence or absence of reported EF impairment
Note: Z scores ≤ −1.5 were considered as low (Scholte & Noens, Reference Scholte and Noens2011).
Performance-Based Versus Reported EF
Compared to normative values of the general population, patients scored significantly lower on the Stroop Test (M = −0.91, SD = 1.71, z = −6.25, p < .001), the SAT (M = −1.09, SD = 1.34, z = −7.26, p < .001), and Digit Span Backward (M = −0.84, SD = 0.96, z = −75, p < .001) 3 months after surgery (Table 3).
Using a cut-off of Z ≤ −1.5, 15 patients (32%) had low scores on the Stroop Test, 14 (30%) on the SAT, and 12 (26%) patients scored low on Digit Span Backward. In total, 31 patients (66%) had Z scores ≤ −1.5 on one or more tests.
Correlation coefficients of neuropsychological tests scores with patient-report and proxy-report are listed in Table 4. Overall, nonsignificant very small correlations were observed, with r ranging from −0.29 to 0.10, except for Digit Span Backward, where significant negative correlations were observed with patient-report of EF (r of −0.33 and −0.37 for Behavioral Regulation and Metacognition, respectively).
Table 4. Performance-based measures of EF: Mean scores compared to normative values and correlations with reported EF
a Sample means compared to normative values (Mean = 0, SD = 1).
b Correlation coefficients between 0.10 and 0.29 were considered as small, 0.30–0.49 as medium, and 0.50–1.0 as large (Cohen, Reference Cohen1988).
* p < .05
Association Between EF Test Performance and Patient–Proxy Differences
Multiple regression analyses indicated that scores on performance-based measures of EF (Stroop Test, SAT, and Digit Span Backward) had no significant influence on the patient–proxy difference scores of Behavioral Regulation [F (3,47) = 0.109, p = .954, R 2 = 0.008] and Metacognition [F (3,47) = 0.924, p = .412, R 2 = 0.065].
DISCUSSION
The present study was performed to evaluate patient–proxy agreement with respect to report of EF in patients with meningioma and low-grade glioma. In addition, the relationship between reported EF with performance-based measures of EF was investigated, as was the effect of EF test performance on the level of agreement. Since patients are increasingly involved in health-care decisions, it is important to gain more insight in their metacognitive abilities and how their experiences relate to experiences of their informal caregivers and to performance-based measures of EF.
Firstly, compared to normative data, lowered patient scores were observed on reported EF (Metacognition Index) and on EF test performance (Stroop Test, SAT and Digit Span Backward). These findings contribute to the large body of evidence showing that patients with primary brain tumors experience complaints and deficits in EF after neurosurgery. Moreover, it stresses the importance of careful assessment of EF. Since problems in EF can have major impact on everyday life of patients and health-care decisions, attention should be paid to EF of patients throughout their disease trajectory, in both clinical care and research settings.
In this study, patients reported significantly more problems than their proxies on the Metacognition Index. A similar mismatch between patients and partners has been reported in patients with dementia. It has been suggested that in the early stages of the disease, patients experience subtle cognitive complaints that are not always noticed by their partners (e.g., Fogarty, Almklov, Borrie, Wells, & Roth, Reference Fogarty, Almklov, Borrie, Wells and Roth2017; Rueda et al., Reference Rueda, Lau, Saito, Harvey, Risacher, Aisen, Petersen, Saykin and Farias2015). However, when the disease progresses, diminished disease awareness often arises (Sunderaraman & Consentino, Reference Sunderaraman and Consentino2017), leading to the reversed pattern: partners often report more problems than patients (Rueda et al., Reference Rueda, Lau, Saito, Harvey, Risacher, Aisen, Petersen, Saykin and Farias2015). Our sample consists of brain tumor patients who have slow-growing tumors and a relatively favorable prognosis. It is possible that self-reported EF problems may decrease as the disease progresses and that studies in patients with high-grade glioma or metastases with less favorable prognosis would render different results. However, more research is needed to further explore this hypothesis. Furthermore, no systematic differences between patients and proxies were observed in group means of the Behavioral Regulation Index. Possibly, this is due to the fact that this index generally contains more items that are aimed at the report of concrete, overt behavior (e.g., “I drum my fingers or wiggle my legs”; “I start things on the last moment”), which makes it easier for proxies to assess patients’ functioning (Rooney et al., Reference Rooney, McNamara, Mackinnon, Fraser, Rampling, Carson and Grant2013).
Regarding the degree of concordance as measured with Lin’s CCCs, the findings indicate moderate to substantial agreement among dyads on patients’ EF, with CCCs of 0.61 and 0.57 for the Behavioral Regulation Index and Metacognition Index. Based on these CCCs, we can conclude that proxy-reports are a reasonable estimate of patient-reported EF. The additional information from the Bland–Altman plots and cross tables suggests less agreement between patients and proxies on the Metacognition Index compared with the Behavioral Regulation Index. This can also be explained by the more concrete and observable symptoms assessed by the Behavioral Regulation Index, as described above.
Although patients showed lower EF test scores, no clear associations were observed between patients’ self-reports and proxy-reports with performance-based measures of EF. Overall, (very) small correlations were observed in this study, which is in line with previous research in brain tumor patients and other neurologic populations (Lanni et al., Reference Lanni, Ross, Higginson, Dressler, Sigvardt, Zhang, Malhado-Chang and Disbrow2014; Pranckeviciene, Deltuva, Tamasauskas, & Bunevicius, Reference Pranckeviciene, Deltuva, Tamasauskas and Bunevicius2017; Rabin, Wang, Katz, Derby, Buschke, & Lipton, Reference Rabin, Wang, Katz, Derby, Buschke and Lipton2012; Schiehser et al., Reference Schiehser, Delis, Filoteo, Delano-Wood, Han, Jak, Drake and Bondi2011). Based solely on the absence of associations of patients’ self-report and performance-based results, one might conclude that patients are not able to estimate the objective level of their EF, possibly due to a lack of insight or reduced cognitive capabilities. However, the current findings render this explanation more unlikely, since we found that there was concordance between patient and proxies, and proxy-report was also not significantly correlated with patients’ test performance. Thus, there is a mismatch between reported EF and test performance, but we do not know which of the measures is the most accurate.
There are many difficulties involved in measuring EF, and an objective gold standard is missing. Based on the finding that reported EF was not correlated with EF test performance in both clinical and nonclinical samples, Toplak, West, and Stanovich (Reference Toplak, West and Stanovich2013) concluded in their review that rating measures of EF and performance-based measures of EF appear to capture different underlying mental constructs and therefore, they cannot be interpreted as equivalent nor be used interchangeably. Tests often focus on a particular domain of EF and are administered in a quiet and controlled environment on a certain moment in time, while EF in everyday life requires complex, integrated, and dynamic skills. Ecological validity of performance-based measures of EF is limited, and thus will not fully capture experienced problems in patients’ day-to-day functioning. On the other hand, several studies demonstrated that self-reported EF is associated with psychological variables, such as personality and mood (Buchanan, Reference Buchanan2016; Meltzer et al., Reference Meltzer, Kapoor, Fogel, Elbulok-Charcape, Roth, Katz and Rabin2016; Pranckeviciene et al., Reference Pranckeviciene, Deltuva, Tamasauskas and Bunevicius2017), and that self-report response biases can affect validity and reliability (Althubaiti, Reference Althubaiti2016).
In addition to neuropsychological tests, Noll, Bradshaw, Weinberg, and Wefel (Reference Noll, Bradshaw, Weinberg and Wefel2018) made use of measures of functional independence (i.e., the Functional Independence Measure and the Karnofsky Performance Status) and found significant correlations between functional independence and performance-based measures of EF (r between 0.28 and 0.42). A potential promising alternative is using an Experience Sampling Method to measure self-reported EF, at multiple times throughout the day, actually integrated in the daily lives of patients (Moore, Schwendsen, & Depp, Reference Moore, Schwendsen and Depp2017; Myin-Germeys et al., Reference Myin-Germeys, Kasanova, Vaessen, Vachon, Kirtley, Viechtbauer and Reininghaus2018). By gathering real-time data, some biases that occur through reporting in retrospect on conventional self-report questionnaires can be avoided. Furthermore, virtual environments are increasingly being used to enhance neuropsychological testing, so that they better reflect situations patients face in the outside world (Parsey & Schmitter-Edgecombe, Reference Parsey and Schmitter-Edgecombe2013; Parsons, Carlew, Magtoto, & Stonecipher, Reference Parsons, Carlew, Magtoto and Stonecipher2015). These relatively new assessment techniques, with potentially improved validity and reliability, could also be explored in future studies in patients with primary brain tumors.
In this study, no effect of executive test performance on dyadic agreement was found, which is in contrast with our expectations based on previous research on health-related quality of life in brain tumor patients (Brown et al., Reference Brown, Decker, Rummans, Clark, Frost, Ballman, Arusell and Buckner2008; Ediebah et al., Reference Ediebah, Reijneveld, Taphoorn, Coens, Zikos, Aaronson, Heimans, Bottomley and Klein2017; Sneeuw et al., Reference Sneeuw, Aaronson, Osoba, Muller, Hsu, Yung, Brada and Newlands1997) and studies in other populations (Hart, Whyte, Kim, & Vaccaro, Reference Hart, Whyte, Kim and Vaccaro2005; Howland, Allan, Carlton, Tatsuoka, Smyth, & Sajatociv, Reference Howland, Allan, Carlton, Tatsuoka, Smyth and Sajatociv2017). It should be noted that the instruments used to assess cognitive performance differed across studies (e.g., the Mini Mental State Examination or extensive neuropsychological test batteries), as well as the statistical procedures used to examine the possible effect of test performance on dyadic agreement. Moreover, it is possible that deficits in EF are more likely to relate to patient–proxy agreement when patients report less problems than proxies (potentially resulting from impaired insight), instead of more problems, as was observed in our sample of meningioma and low-grade glioma patients.
Besides EF test performance, other factors may also have modulated the level of agreement between patients and proxies, including patient- or disease-related factors, proxy-related factors or factors related to the assessment of EF. The small sample size of this study limited our possibilities to explore the potential influence of, for example, tumor location or tumor histology, with valid statistical testing. Also, we had little information available on the informal caregivers or the quality of the relationship with the patient, restricting the possibility to take this into account. Moreover, informal caregivers were involved in the cognitive rehabilitation study, only if patients (and proxies) were willing to. Possibly, this may have led to a selection bias (i.e., sampling bias), namely, the inclusion of dyads who are highly motivated and might agree on the patient’s functioning more than dyads not included in this study. In addition, all patients in this study were participants in a cognitive rehabilitation trial, with strict in- and exclusion criteria, which may also hamper generalizability of the findings to the low-grade glioma and meningioma patient population as a whole.
In conclusion, patients with primary brain tumors experience executive deficits after neurosurgery. Congruence between brain tumor patients and their proxies was observed in the evaluation of patients’ EF, but correlations with performance-based measures of EF were low. Since problems in EF are commonly observed in brain tumor patients and can lead to lowered quality of life, it is important to assess EF carefully throughout the disease trajectory, using both questionnaires and performance-based measures of EF. At the same time, research on the use of innovative methods to assess EF in patients with primary brain tumors should be expanded.
ACKNOWLEDGEMENTS
This work was supported by the Dutch organization for health research and innovation (ZonMw) (grant number 842003009).
CONFLICTS OF INTEREST
The authors declare they have no conflict of interests.
Open access
