The development, and day-to-day variation, of a Military-Specific Auditory N-Back Task and Shoot-/Don’t -Shoot Task

Abstract

During military operations, soldiers are required to successfully complete numerous physical and cognitive tasks concurrently. Understanding the typical variance in research tools that may be used to provide insight into the interrelationship between physical and cognitive performance is therefore highly important. This study assessed the inter-day variability of two military-specific cognitive assessments: a Military-Specific Auditory N-Back Task (MSANT) and a Shoot-/Don’t-Shoot Task (SDST) in 28 participants. Limits of agreement ±95% confidence intervals, standard error of the mean, and smallest detectable change were calculated to quantify the typical variance in task performance. All parameters within the MSANT and SDST demonstrated no mean difference for trial visit in either the seated or walking condition, with equivalency demonstrated for the majority of comparisons. Collectively, these data provided an indication of the typical variance in MSANT and SDST performance, while demonstrating that both assessments can be used during seated and walking conditions.

Introduction

During military operations, personnel are required to maintain performance in both their role-specific physical tasks (e.g., load carriage) and corresponding cognitive tasks (e.g., decision making and communication) (Crawford et al., 2017; Scribner, 2016). Failure to maintain performance, in either domain, can result in reduced combat readiness and decreased operational performance (Crawford et al., 2017; Vrijkotte et al., 2016). Consequently, there is growing interest in the relationship between military-specific physical activity and cognitive performance within military operators (Armstrong et al., 2022; Bhattacharyya et al., 2017; Eddy et al., 2015; Giles et al., 2019; Kobus et al., 2010; Nibbeling et al., 2014; Son et al., 2019, 2022; Vine et al., 2021). Despite this interest, the methodologies and approaches used to investigate this relationship have differed considerably, particularly concerning the assessment of cognitive performance.

Based on the assessment tools used to date, and the visual and auditory requirements of soldiers, two assessment tools were developed: A Military Specific Auditory N-Back Task (MSANT), and a Shoot-/Don’t-Shoot (SDST). The former used phonetically described pairs of letters and represented aspects of military radio communications, while the latter represented aspects of any military scenario where visual search and inhibition are required (e.g., assaulting an enemy position or operations in builtup areas). The current study, therefore, aimed to detail the methodology of the MSANT and SDST, along with quantifying the typical day-to-day variability of both assessment tools under seated and walking condition. The investigation did not seek to investigate the influence of physical fatigue or dual tasking on the performance of these assessment tools.

Methods

The full methods for this project are available in the Supplementary Material, with the raw data available at: https://osf.io/jekv8/. Briefly, the study comprised of two elements. First, the day-to-day variability of the MSANT and SDST was assessed in a seated condition on three separate occasions (Part 1). This was chosen due to the large variability in potential application of these assessment tools in future projects. Second, within a sub-sample of the study population, the day-to-day variability of the MSANT and SDST was assessed during a 10-min walking activity, on three separate occasions (Part 2). While a matched study population for this part of the study would have been optimal, given the time required for this portion of the study (a result of the necessity to reach a physiological steady state before conducting the test, and the recovery period required between each walking bout to prevent the onset of fatigue), a sub-sample approach was instead chosen. Physiological steady state refers to the stabilization in the physiological responses to exercise (e.g., increases in heart rate). Without this stability, variability in cognitive performance could be induced as a consequence of adapting to the exercise stimulus opposed to just reflecting the typical variation in test performance.

All laboratory visits were separated by a minimum of 48 hr, and participants were required to arrive in a fed and hydrated state having avoided caffeine for a minimum of 3 hr. Study visits were completed at approximately the same time of day (±2 hr) to control for the potential effect of circadian rhythm on test performance. All participants were recruited from the university population (all were students or from academic positions), spoke fluent English, and had self-declared normal or corrected to normal vision.

Twenty-eight participants volunteered for Part 1 of the study (14 male, 14 female, age [mean ± SD] 27.3 ± 4.3 year) and 12 participants for Part 2 (6 male, 6 female, age 28.4 ± 3.5 year). Sample size for Part 1 was calculated using an a priori power calculation (G Power; version 3.1.9.4) (Prajapati et al., 2010). For the seated portion of the investigation, 28 participants were required to a moderate effect size (f = 0.25), with a statistical power of 80% and an alpha level of 0.05, based upon a correlation coefficient of r = 0.5 (identified from initial pilot testing). A moderate effect size (Cohen, 1988) was selected based on the combination of effect sizes reported in previous investigations, utilizing similar cognitive assessment tools (Eddy et al., 2015), and the anticipated smallest effect size of interest to military policymakers. The sub-sample size was designed to represent the typical size (and therefore likely variation) of study populations within this research area (e.g., Bhattacharyya et al., 2017; Crowell et al., 1999; Eddy et al., 2015). Ethical approval was provided by the Institution’s Research Ethics Committee, with written consent obtained from all participants.

Cognitive assessments

The MSANT involved identifying a pair of phonetically described letters two previous to an auditory tone (i.e., 2-back). During the seated condition, participants recorded their answers, while during walking trials, participants were required to relay their answers verbally which were recorded on their behalf. The SDST was designed to be a visual search and inhibition task similar to those tasks previously employed within the literature (Armstrong et al., 2022; Eddy et al., 2015; Kobus et al., 2010). The assessment involved responding appropriately to targets and non-targets. Participants were instructed to place equal importance on both response time and accuracy. For the SDST, there was a 2:1 ratio between targets and non-targets.

For Part 1, during the first visit, participants were familiarized (two full trial completions of each assessment) with the MSANT and SDST, in a randomized counterbalanced order. For the second, and third visits, participants completed the MSANT and SDST in the same randomized counterbalanced order. For Part 2, a sub-sample of 12 participants completed 3 additional laboratory visits completing the SDST and MSANT while walking on a treadmill. Again the MSANT and SDST were completed in a randomized order. All tests were completed with 10 min of seated rest between trials to negate the influence of physical fatigue. To enable a physiological steady state to occur, participants completed 5 min of walking before the commencement of the cognitive assessments. For all walking trials, participants walked on a motorized treadmill (6.5 km·h⁻¹, 1% gradient) at a load carriage speed representing a typical “enemy contact” speed (Armstrong et al., 2019).

Statistical analysis

Data were principally analyzed using JASP (JASP, 2020; v0.14.1). For normally distributed data, a one-way ANOVA was employed to identify whether a likely main effect of assessment time point was apparent. Effect sizes are presented as Omega squared (Ѡ ²) (Levine & Hullett, 2002). For non-normally distributed data, a Friedman’s test was employed with effect sizes presented using Kendall’s W. Holm-Bonferroni adjusted pairwise comparisons, and pairwise comparisons using Conover’s test were made post-hoc as appropriate. For key assessment variables, equivalency between trials was calculated using the two one-sided test approach (Lakens et al., 2018). Based upon the a priori sample size calculation, d = 0.5 was employed as the smallest effect size of interest. To describe the typical variation in assessment parameters between trials, Limits of agreement ± 95% confidence intervals, standard error of the mean, and smallest detectable change values were calculated (Hopkins, 2000; Ludbrook, 2010; van Kampen et al., 2013).

Results

Descriptive statistics are presented in Table 1, with day-to-day variation descriptors reported in Table 2. One participant was removed from the analysis, due to being more than two SDs outside the remainder of the data set.

Table 1. Descriptive statistics for cognitive assessments (mean ± SD [range]) during seated (S) and walking (W) conditions

Note. Blank cells denote data that were not collected due to the seated condition acting as the familiarization for the walking condition.

Abbreviations: ASTO, accuracy-speed trade-off; CR, correct response; FAM, familiarization; MSANT, Military-Specific Auditory N-Back Task; RT, response time; S, seated; SDST, Shoot-/Don’t-Shoot Task; W, walking.

Table 2. Descriptive statistics for cognitive assessments (mean ± SD) during seated (S) and walking (W) conditions

Abbreviations: ASTO, accuracy-speed trade-off; CI, confidence intervals; CR, correct response; MSANT, Military-Specific Auditory N-Back Task; RT, response time; S, seated; SDC, smallest detectable change; SDST, Shoot-/Don’t-Shoot Task; SEM, standard error of the mean; W, walking.

Seated performance

Military-Specific Auditory N-Back Task

There was no likely main effect for time for total correct response (χ ²₍₄₎ = 4.531, p = .361, Kendall’s W = 0.492) or combined correct responses (χ ²₍₄₎ = 3.856, p = .426, Kendall’s W = 0.488); however, a likely main effect for time was evident for partial correct responses (χ ²₍₄₎ = 11.846, p = .019, Kendall’s W = 0.426). For the key variable of combined correct responses, the comparison between trial 1 versus trial 2 was both statistically equivalent (W ₍₂₅₎ = 64, p = .002) and not statistically different (W ₍₂₅₎ = 70, p = .938). Similarly, trial 2 versus trial 3 were both statistically equivalent (W ₍₂₅₎ = 20, p = .06) and not statistically different. Likewise, trial 1 versus trial 3 were also both statistically equivalent (W ₍₂₅₎ = 50, p = .032) and not statistically different.

Shoot-/Don’t-Shoot Task

There was no likely main effect for time on either shoot correct (χ ²₍₄₎ = 4.00, p = .406, Kendall’s W = 0.175), don’t-shoot correct (χ ²₍₄₎ = 3.069, p = .546, Kendall’s W = 0.482), total correct (χ ²₍₄₎ = 3.375, p = .497, Kendall’s W = 0.471), and average response time (F _{(2.981,77.515)} = 1.035, p = .382, Ѡ ² = 0.001). There was, however, a main effect for time in the accuracy-speed trade-off (ASTO) parameter (F _(4,104) = 7.037, p < .001, Ѡ ² = 0.089). Importantly, the sole noteworthy difference occurred between familiarization 1 and trial 3 (t ₍₂₆₎ = 4.855, p < .001, d = 0.756) suggesting no discernible difference was likely between performances in the three experimental trials, following two familiarization trials. For the key variable of total correct responses, trial 1 versus trial 2, trial 1 versus trial 3, and trial 2 versus trial 3 were both statistically equivalent (1 vs. 2: W ₍₂₆₎ = 93, p = .011; 1 vs. 3: W ₍₂₆₎ = 61, p = .047; 2 vs. 3: W ₍₂₆₎ = 41, p = .040) and not statistically different. For the other key variable of ASTO, all comparisons were likely neither statistically equivalent (1 vs. 2: t ₍₂₆₎ = −1.701, p = .050; 2 vs. 3: t ₍₂₆₎ = −0.127, p = .45; 1 vs. 3: t ₍₂₆₎ = 0.287, p = .612) nor statistically different.

Walking performance

Military-Specific Auditory N-Back Task

As with seated MSANT performance, there was no likely effect of time on total correct responses (χ ²₍₂₎ = 1.000, p = .607, Kendall’s W = 0.568), partial correct responses (χ ²₍₂₎ = 1.280, p = .527, Kendall’s W = 0.541), and combined correct responses (χ ²₍₂₎ = 1.000, p = .607, Kendall’s W = 0.582). For the key variable of combined correct responses, trials 1 versus 2 and trials 2 versus 3 were statistically equivalent (1 vs. 2: W ₍₁₁₎ = 12, p = .017; 2 vs. 3: W ₍₁₁₎ = 13, p = .020) and not statistically different. Conversely, trial 1 versus 3 was neither statistically equivalent nor statistically different.

Shoot-/Don’t-Shoot Task

Again there were no likely effects of time on shoot correct responses (χ ²₍₂₎ = 4.800, p = .091, Kendall’s W = 0.449), don’t-shoot correct responses (χ ²₍₂₎ = 2.480, p = .289, Kendall’s W = 0.672), total correct responses (χ ²₍₂₎ = 3.161, p = .206, Kendall’s W = 0.741), response times (F _(2,22) = 2.880, p = .077, Ѡ ² = 0.018), and ASTO (F _(2,22) = 2.713, p = .088, Ѡ ² = 0.042). For the key variable of total correct responses, all comparisons were neither statistically equivalent (1 vs. 2: W ₍₁₁₎ = 6, p = .096; 1 vs. 3: W ₍₁₁₎ = 0, p = .093; 2 vs. 3: W ₍₁₁₎ = 14, p = .084) nor statistically different. Similarly, for the other key variable of ASTO all comparisons were likely neither statistically equivalent (1 vs. 2: t ₍₁₁₎ = 0.127, p = .549; 2 vs. 3: t ₍₁₁₎ = −1.205, p = .127; 1 vs. 3: t ₍₁₁₎ = 0.787, p = .776) nor statistically different.

Discussion

This study has described the methods of two military-specific cognitive assessment tools (MSANT and SDST) and quantified their typical day-to-day variability. These data provide typical magnitudes of variance for the key assessment parameters. While no likely performance differences were observed across the experimental measurement points, not all walking comparisons were statistically equivalent, suggesting additional data are required before this assertion is made, for the given equivalency bounds. It should however be noted that borderline statistically significant results may become non-significant where correction for multiple testing is utilized. The current investigation has also demonstrated the suitability of these assessment tools for use during military-specific physical activity within a laboratory setting.

Before this investigation, the day-to-day performance variation in any military-specific cognitive assessments had not been quantified. This is an issue for several reasons, including the translational ability of research findings to the “real world” (Close et al., 2019) and also for methodological decision making (e.g., sample size calculations). Moreover, with military operations rarely conducted in isolation, information on inter-test performance is highly relevant to research investigating sequential or repeated bout performance. The comparison between seated and walking performance was not a research question of interest in the current study, particularly given that deficits in cognitive performance are typically observed after ~30 min of military activity (e.g., Eddy et al., 2015; Giles et al., 2019). However, observationally, the typical variation in performance between trials appears similar between seated and walking conditions.

Familiarizing participants with assessment tools is critical for research, particularly when time limitations may inhibit access to study participants (e.g., military populations). Collectively, the current study’s data demonstrate that beyond two full seated trials, a continued improvement in performance was not likely apparent, suggesting this familiarization length is sufficient to minimize possible learning effects.

Several limitations exist with the current investigation, including the use of a civilian population, and the limited walking sub-sample size. As acknowledged previously, the smaller sub-sample size was chosen for largely practical reasons, although it does match many studies within this area, highlighting issues with underpowered investigations. Future research should attempt to pair reliable and applied cognitive tasks (such as those described herein) with operationally relevant and appropriate physical activity. This in turn will support enhanced applied research as well as enabling a greater focus to be placed on developing mitigation strategies where the greatest mission impact can be obtained.