A shift in strategy or “error”? Strategy classification over multiple stochastic specifications

Abstract

We present a classification methodology that jointly assigns to a decision maker a best-fitting decision strategy for a set of choice data as well as a best-fitting stochastic specification of that decision strategy. Our methodology utilizes normalized maximum likelihood as a model selection criterion to compare multiple, possibly non-nested, stochastic specifications of candidate strategies. In addition to single strategy with “error” stochastic specifications, we consider mixture specifications, i.e., strategies comprised of a probability distribution over multiple strategies. In this way, our approach generalizes the classification framework of Bröder and Schiffer (2003a). We apply our methodology to an existing dataset and find that some decision makers are best fit by a single strategy with varying levels of error, while others are best described as using a mixture specification over multiple strategies.

1 Introduction

Within the decision making literature, there exist many interesting decision theories, both normative and descriptive, that are defined deterministically. These theories, hereafter termed decision strategies, are not defined in terms of random variables and therefore do not explicitly account for fluctuations in choice by a decision maker (DM). Yet, many researchers have observed that DMs are not deterministically consistent in their observed choices across repeated comparisons of the same stimuli set or choice type (e.g., Ballinger & Wilcox, 1997; Busemeyer, Weg, Barkan, Li, & Ma, 2000; Hey, 2001; 2005; Hey & Orme, 1994). This variability of choice does not disappear over repeated experimental sessions for the same subject (e.g., Regenwetter, Dana, & Davis-Stober, 2011) nor can it be attributed entirely to random responses and/or learning and experience effects (Hey, 2001; Loomes, Moffatt, & Sugden, 2002).

Many researchers have suggested different methods of incorporating a stochastic specification to deterministic decision strategies to accommodate variability of choice. Perhaps the simplest stochastic specification is to add a fixed level of “error” to a DM’s responses. This “trembling hand” stochastic specification assumes that a DM follows a single strategy imperfectly with some probability of incorrectly selecting a choice alternative that is not truly preferred (Harless & Camerer, 1994). Other specifications, such as the class of Thurstonian and Fechnerian models, treat preference as deterministic and model choice by adding a random variable representing “noise” (e.g., Böckenholt, 2006). Another class of stochastic specifications are those that incorporate multiple decision strategies with choice modeled as a sampling process over a set of strategies (Loomes et al., 2002; Loomes & Sugden, 1995; Loomes, 2005; Marschak, 1960; Rapoport & Wallsten, 1972; Rieskamp & Otto, 2006). There are, of course, many variations and combinations of the above specifications as well (Loomes et al., 2002). Further complicating matters, the choice of which stochastic specification to use is not an innocuous one. Hey (2005) recently argued that different specifications can lead to different conclusions for the same set of decision strategies and urged that we “should not be silent about noise” when evaluating decision strategies.¹

In this article, we present a new strategy classification framework that evaluates a set of candidate strategies over multiple types of stochastic specification. We consider single strategy with “error” specifications as well as probabilistic mixtures of strategies, and, using modern model selection criteria, directly compare these different models to one another retaining the optimal strategy-specification pair for a set of observed choice data. Thus, our framework assigns not only the best-fitting strategy(ies) to a DM’s choices, but also a best-fitting stochastic specification. We illustrate our classification framework with a simulation study that considers three different decision strategies over four types of stochastic specification. We also apply our framework to an existing empirical data set collected to test the transitivity of preference axiom (Tversky, 1969). Throughout this article we will be concerned with the classification of strategies to DMs using outcome-based choice data, a perspective often referred to as comparative model fitting.²

How can one carry out strategy classification via choice data? At the most basic level, a researcher could classify strategies by simply counting the number of times a strategy was in accordance with the observed data, thereby ignoring the problem of stochastic specification altogether (e.g., Marewski, Gaissmaier, Schooler, Goldstein, & Gigerenzer, 2010). However, this “accordance rate” approach can lead to significant biases in strategy classification (Bröder, 2010; Bröder & Schiffer, 2003a; Hilbig, 2010a). Hilbig (2010a; 2010b) demonstrates how these practices can lead to theoretical confoundings and uninterpretable data. Regenwetter et al. (2009) provides a critique of the related practice of evaluating strategies by modal pair-wise majority (e.g., Brandstätter, Gigerenzer, & Hertwig, 2006).

More recent approaches have leveraged statistical models to carry out strategy classification. Bröder and Schiffer (2003a) developed a methodology built on a binomial (multinomial) statistical framework that models DMs as utilizing a single strategy with some fixed probability of making an “error”. Under this framework, strategy classification is likelihood-based and is carried out by calculating and comparing Bayes factor scores (Bröder & Schiffer, 2003a). This basic approach has been successfully applied to memory-based multiattribute choice (Bröder & Gaissmaier, 2007; Bröder & Schiffer, 2003b) and decision making under risk paradigms (Glöckner & Betsch, 2008). Glöckner (2009; 2010) generalizes this approach by incorporating any number of additional measures, such as decision time and confidence data, into the likelihood function to allow theories with identical predictions at the choice level to be distinguished, see also Bergert and Nosofsky (2007) and Glöckner (2006). These approaches share the similar assumption that all DMs utilize the same stochastic specification. To tackle the problem of DMs following different stochastic specifications, Loomes et al. (2002) adapted a non-nested likelihood ratio test (Vuong, 1989) to evaluate a set of candidate strategies under various stochastic specifications. In a comprehensive set of studies, they found that DMs were best fit by a variety of stochastic specifications with a mixture specification of strategies consistent with expected utility yielding an excellent fit to many DMs. The non-nested likelihood ratio test, however, is an asymptotic test that may lack statistical power when applied to small samples (Genius & Strazzera, 2002). Nor does the non-nested likelihood ratio test provide an explicit measure of model complexity.

In this article, we generalize the strategy classification framework of Bröder and Schiffer (2003a) by considering multiple stochastic specifications of decision strategies, including probabilistic mixtures of multiple strategies. While many specifications are possible under our framework, we will restrict ourselves to the three specifications described in Table 1 for the applications in this article. These specifications were chosen for their theoretical relevance to the Tversky data set (Section 4) as well as to showcase the generality of our approach. We apply the latest operationalization of the minimum description length principle, normalized maximum likelihood, to carry out model selection among all strategy-specification pairs (e.g., Myung, Navarro, & Pitt, 2006). Normalized maximum likelihood is an information-theoretic model selection criteria which evaluates models based upon the ratio of a model’s goodness-of-fit to its complexity, adhering to Occam’s razor by favoring the simplest model that provides a good account of the data. The normalized maximum likelihood criteria accurately accounts for the stochastic complexity of a model and is equally valid for both small and large samples.

Table 1: Stochastic specifications.

2 Preliminary definitions

Similar to many previous approaches (e.g., Davis-Stober, 2009; Iverson & Falmagne, 1985; Myung, Karabatsos, & Iverson, 2005), we model stochastic choice between two choice alternatives a and b as a fixed probability, _ab, i.e., the probability that a is chosen over b. If we model each paired comparison in a two-alternative forced choice task as a Bernoulli process then, assuming independence between trials, a DM’s choice responses follow a binomial distribution. Let n _ab be the number of times the DM chose a over b and let N _ab be the total number of times the pair was presented to the DM. Let K be the set of all distinct choice pairs under consideration.³These assumptions give the following likelihood,

(1)

where 0<θ _ab<1, ∀ (a,b) ∈ K. To further simplify notation, let θ = (_ab)_{∀ (a,b) ∈ K} and n = (n _ab)_{∀ (a,b) ∈ K}. Note that (1) can easily be extended to ternary choice by using a trinomial distribution, and, more generally, to k-ary choice, where k ∈ ℕ, using the appropriate multinomial distribution.

Let m be the number of distinct decision strategies under consideration. To each candidate strategy we assign a vector, denoted z, that encodes that strategy’s prediction for each choice pair. Thus, each vector z is of size |K| × 1 with coefficients of 1 (predicts a is preferred to b for the respective choice pair), 0 (predicts b is preferred to a for the respective choice pair), or .5 (predicts guessing between a and b).

We model the different strategy-specification pairs under the common likelihood framework of (1) by placing a series of systematic constraints upon the θ parameters. Let Θ denote the parameter space of all possible θ. By placing these models under a common likelihood structure, we are, in a sense, placing these different strategy-specification pairs on a “common ground” and can carry out model selection to find the best account of the data.

2.1 Single strategy with error specifications

Similar to the formulation in Bröder and Schiffer (2003a) and Glöckner (2009), we can conceptualize the parameters of the binomial likelihood (1) as representing “error” terms for individual strategies. Perhaps the simplest such stochastic specification is to assume a single error rate, є, that is equally applied to all choice pairs and is estimated separately for each strategy under consideration. We can model this stochastic specification under our framework by requiring that all of the parameters in θ are a simple function of є and are bounded in a systematic way, such that if the strategy in question, z, predicts that a is preferred to b then the probability of a being chosen over b is (1-є), i.e., _ab = (1−є), if b if predicted to be preferred to a then _ab = є, and so on for all choice pairs. Therefore, instead of estimating all |K|-many parameters in θ, we need only estimate a single parameter, є. We handle strategies that predict guessing between pairs by forcing the respective _ab parameter(s) to be equal to .5.

The error term, є, is the only parameter that accounts for the variability in the DM’s responses under this specification. This raises the question, how much error is reasonable? Or said another way, what constraints should we place on є? For the applications in this article, we distinguish between two cases although many are possible. First, consider a DM that applies a single strategy in a nearly “error-free” fashion. For this low-error case, we bound є to be no larger than some small value; for our applications we chose an upper bound of .03, and thus 0 < є ≤ .03. Here, the DM is assumed to apply the single strategy, z, either deterministically, or, if he does make errors they are quite seldom. For the second “error” specification, we model DMs who apply a single strategy with larger values of error. For this high-error case, we allow є to vary from 0 to .20 (the DM is expected to make an error a maximum of about one in five trials). The latter stochastic specification is better able to account for DM’s choices as it is less constrained than the former, however it is also less parsimonious as it can fit a much wider range of possible observations. It is important to point out that for most applications these bounds must be chosen a priori and be theoretically motivated. It is certainly permissible to make є “unrestricted,” i.e., 0 < є ≤ .50, however this will allow extremely large error rates and, depending on the context, may not be a realistic assumption.

Within the general likelihood framework of (1), it is possible to model other types of single strategy stochastic specifications. One could easily assign independent error terms to each choice pair by considering a separate and independently estimated є value for each distinct _ab, denoted є_ab. This is similar to the stochastic specification used by Sopher and Gigliotti (1993) in their analysis of intransitive preference. This “multiple errors” stochastic specification could be easily incorporated and properly evaluated within our framework.

2.2 Mixtures specifications

One of the most general frameworks for modeling multiple strategies is equivalently termed either a random preference model (Loomes & Sugden, 1995; Marschak, 1960) or a distribution-free random utility model (e.g., Davis-Stober, 2009; Falmagne, 1978; Niederée & Heyer, 1997; Regenwetter, 1996; Regenwetter & Marley, 2001). This stochastic specification models both preference and choice as probabilistic, with a DM’s observed choices being governed by a (possibly unknown) probability distribution over a set of strategies. As an example, suppose a DM eats at a restaurant and tends to follow two different strategies. On some days, he wants to spend as little money as possible and orders the cheapest meal on the menu, at other times he chooses by food quality and therefore chooses the more expensive dishes. This DM’s choices could be well-modeled by a probability distribution over these two strategies (choose by price or food quality).

This type of mixture specification has a very nice geometric interpretation within our framework. Let {z ₁, z ₂, …, z _m} be the vector representations of a set of m strategies to be considered as a mixture specification, i.e., a stochastic specification in which the DM’s choice probabilities are modeled as an arbitrary probability distribution over the m strategies. We can compactly represent the set of all probability distributions over a set of m strategies {z ₁, z ₂, …, z _m} as the set of all sums,

(2)

where w _i ≥ 0, ∀ i ∈ {1,2,…, m} and ∑_i=1^m w _i = 1.

The weights, w _i, are the corresponding probability weights for the mixture distribution over the m many decision strategies. The set of all such sums (sets of probability weights) corresponds to the convex hull of the set of vectors representing the strategies under consideration, thereby defining a convex polyhedron in the appropriate geometric space (Ziegler, 1994). If we consider the points {z ₁, z ₂, …, z _m} as embedded within the parameter space Θ we obtain a polyhedron such that its interior corresponds to each and every possible probability distribution over the decision strategies {z ₁, z ₂, …, z _m}. This polyhedron can be uniquely associated with a set of linear inequality constraints on the parameter space Θ. Said another way, given a set of strategies to consider as a mixture specification, we can solve for the linear inequality constraints on the choice probabilities θ that provide necessary and sufficient conditions for these choice probabilities to be represented as a probability distribution over the strategies under consideration.

As an example, assume that K = {(a,b), (c,d)} with _ab and _cd as the choice probabilities corresponding to the two choice pairs in K. Consider three decision strategies with z ₁ = [ 1   1], z ₂ = [ 0    0], and z ₃ = [ 0   1]. The mixture specification over all three of these states can be written as the interior of the triangle formed by taking the convex hull of these points in the parameter space Θ. If we were to consider the mixture model of just z ₁ and z ₂ its convex hull would simply be the line segment joining these two points in Θ. Figure 1 presents these different mixture specifications plotted within the parameter space Θ along with the respective linear inequality constraints as a function of _ab and θ _cd. This perspective holds generally in that any mixture specification over a finite number of decision strategies can be represented as a polyhedron in the appropriate (possibly high-dimensional) probability space (e.g., Davis-Stober, 2009).

Figure 1: This figure plots the constraints placed on the parameter space Θ for the three-strategy mixture specification over z ₁, z ₂, and z ₃; as well as the two-strategy mixture specification over z ₁ and z ₂.

The researcher must specify a priori which strategies will be considered as mixture specifications.⁴It is important to point out that under this mixture stochastic specification all choice variability is modeled as substantive “shifts” between different strategies, i.e., there is no “error” or “noise” component. Other types of mixture specifications are possible under our framework. One could model a mixture-error ‘hybrid’ specification, where the mixture specification is combined with the low and high-error specifications, by solving for the appropriate constraints on the θ parameters.

2.3 Maximum likelihood estimation

To carry out model estimation for the strategy(ies)-specification pairs, we define the maximum likelihood estimator in the standard way,

(3)

where the parameter space Θ is constrained according to the strategy-specification pair being considered.

If the stochastic specification is either the low or high-error case then θ is constrained to be a function of the single parameter, є, oriented according to the decision strategy being considered, z, and bounded accordingly. For the low and high-error specifications, (3) has a simple closed-form solution (e.g., Bröder & Schiffer, 2003a) with the parameter є estimated as follows,

(4)

where z _ab is the coefficient in z that corresponds to the pair (a,b). Note that the sums are only taken over the elements of z that do not predict guessing. For the low-error specification, let _ab = (1 − min{, .03}) for all choice pairs in z such that a is predicted to be preferred to b, let _ab = min{, .03} for all choice pairs such that b is predicted to be preferred to a, and let _ab = .5 if the strategy represented by z predicts guessing between a and b. For the high error specification, we proceed exactly the same way replacing .03 with .20, i.e., if a is predicted to be preferred to b then _ab = (1 − min{, .20}) and so on. It is routine to show that this estimation method yields the solution to (3) for the low and high-error specifications for any choice of z.

For the mixture case, assume that the choice probabilities of Θ are constrained such that they represent the mixture specification (polyhedron) over the strategies under consideration. The solution to (3) for the mixture cases can then be carried out using standard constrained optimization algorithms.⁵For all of the stochastic specifications described above, the constraints placed on the θ parameters form a closed, convex set, thus the maximum likelihood estimator always exists and is unique (Davis-Stober, 2009).

3 Normalized maximum likelihood

How can we place the different strategy-specification pairs on an “equal footing” in order to carry out model selection and classification? Unfortunately, traditional model selection methods, such as the Akaike Information Criteria (AIC) (Akaike, 1973) and the Bayesian Information Criterion (BIC) (Schwarz, 1978), are not appropriate in this situation. Conceptually, the problem lies in the fact that the complexity penalty terms of both the AIC and BIC are simple functions of the number of parameters and thus cannot fully measure the parametric complexity of the stochastic specifications we consider. Within our framework, all pairs comprised of the low-error and high-error specifications would have exactly the same complexity (penalty) term for either the AIC or BIC as both cases have only the single free parameter, є. Yet, the high-error case is able to accommodate a much wider range of data and, in fact, contains the low-error specification as a (greatly constrained) special case! If we consider mixture specifications, the AIC and BIC measures further break down as different mixture specifications may have the same “number” of parameters, but different linear inequality constraints on θ will be able to accommodate data in vastly different ways.

To solve this problem, we will make an appeal to the minimum description length principle (Rissanen, 1978). In contrast to many other methods, such as the Bayes factor, the minimum description length principle does not require the assumption of a “true” data generating distribution or process. As argued by Rissanen (2005), this property is a major strength of the minimum description length perspective, especially in cases where determining or estimating a “true” data generating model is difficult or impossible due to a lack of sample size or fundamental understanding of the phenomenon being observed. Rather, in accordance with Occam’s razor, the minimum description length principle states that the best model of a set of data is the one that leads to the most efficient compression of the data. (Grünwald (2005) provides an in-depth introduction and overview.) Several model selection criteria are derived from the minimum-description-length principle. The most general such derivation is the normalized maximum likelihood (NML) criteria (Barron, Rissanen, & Yu, 1998; Rissanen, 2001) and is defined as follows,

(5)

where L(θ|n) is the maximized likelihood for the observed data n and X is the sample space of the model, i.e., the set of all possible data values. For the case of continuous data, the summation operator in the denominator is replaced by an integral. The NML criteria (5) can be described as a ratio of a model’s goodness-of-fit to its complexity with larger values indicating a superior model. The denominator in (5) is defined as the sum of maximized likelihoods for all possible data points, not just the data that were observed. In this way, the denominator accurately measures the stochastic complexity of a model, incorporating both the size of the parameter space as well as functional complexity (Myung, Navarro, & Pitt, 2006). For example, consider two models that provide an equally good fit to a set of observed data. If one of these models fit well many more data points than the other the denominator term in (5) for that model would be quite large and thus, ceteris paribus, the simpler model would be preferred. See Myung, Navarro and Pitt (2006) for a tutorial on the NML criteria.

To obtain an NML value for a strategy-specification pair within our framework, we first obtain the maximum likelihood estimate for the observed data, n, via (3) and then use this value to calculate L(|n). This forms the numerator of (5). We then carry out this process for all possible data points in the sample space corresponding to (1) for fixed values of N _ab. Taking the sum over all of these maximized likelihood terms gives the denominator in (5). We then carry out this process for each strategy-specification pair. Finally, we compare all NML values, selecting the strategy-specification pair with the largest NML value. To summarize, our methodology proceeds as follows:

1. 1. Given a set of candidate strategies, assign each strategy its corresponding prediction vector, z _i, and decide which stochastic specifications to consider for each z _i (or set of z _i in the case of mixture specifications).

2. 2. Given a set of observed choice data, n, calculate NML values for all strategy-specification pairs under consideration (including mixture specifications).

3. 3. Select the strategy-specification pair with the largest value of NML.

Calculation of the denominator term in (5) can be computationally difficult. All possible data points can be enumerated when both the number of choice pairs in K and the N _ab terms are relatively small, thereby allowing a direct computation of the NML terms. For cases when this is not feasible, estimation of the denominator of (5) can be carried out via Monte Carlo sampling (Preacher, Cai, & MacCallum, 2007). It is important to note that, as the values of N _ab and/or the number of choice pairs in K increase, the larger the sample space becomes and therefore the more Monte Carlo random samples that are needed to estimate (5). We used 10,000 samples per strategy-specification pair for the empirical example in this article (Section 4). When using Monte Carlo sampling, it is often easier to estimate the average maximized likelihood value over the sample space and use this value in lieu of the sum in the denominator of (5) (Preacher et al., 2007). This substitution will leave the model ordering unchanged. Throughout this article, we report NML values by taking the natural logarithm of (5).

4 Simulated data example

In this section, we illustrate our methodology with a set of candidate strategies and choice alternatives adapted from Bröder and Schiffer (2003a). We consider the following three decision strategies: Take The Best (TTB) (Gigerenzer & Goldstein, 1996), Dawes Rule (DR) (Dawes, 1979), and Franklin’s Rule (FR) (Gigerenzer & Goldstein, 1999), see Table 2 for descriptions. We consider four choice alternatives each comprised of four binary cues with corresponding validities (denoted v).⁶These choice alternatives are displayed in Table 3 where “+” or “–” denote the presence or absence, respectively, of a given cue for that choice alternative. We consider two paired comparisons of these four choice alternatives, i.e., |K| = 2 and each choice pair is labeled “Type I” and “Type II” respectively. Table 3 lists the predictions made by these strategies for both choice pairs. We will encode a prediction of alternative a preferred over b as “1” and b preferred over a as “0”. Therefore, z ₁ = [1     1] is the vector representation of TTB, z ₂ = [0     0] is the vector representation of DR, and z ₃ = [0     1] is the vector representation of FR. Again, these decision strategies and cues are chosen for purely illustrative purposes. Our framework can accommodate any decision strategy that makes predictions on binary (or k-ary) choice data.

Table 2: Decision strategies.

Table 3: Choice alternatives and decision strategy predictions

To illustrate how our methodology maps candidate strategy-specification pairs to individual DMs, we simulated data from 25 hypothetical DMs whose “true” choice probabilities, θ, were uniformly distributed over all possible choice probabilities. We carried out our methodology on the three decision strategies considering: both low and high-error specifications for each strategy, all two-strategy mixture specifications, and the single mixture specification over all three strategies. We assumed that each choice type was presented a total of 30 times each. Table 4 lists the NML values for each strategy-specification pair over the 25 simulated data sets. The optimal NML values are in bold. Due to the relatively small sample space, we were able to completely enumerate all possible data points and calculate the NML values exactly. Each data set yielded a unique optimal strategy-specification pair, although several NML values were close in value for some data sets indicating several “good” alternative models. The complexity terms (denominator in (5)) for the single strategy error specifications were approximately .56, both low and high-error specifications differing in the fifth decimal place. The complexity terms for the two-strategy mixture specifications were 2.015 for both {z ₁, z ₂} and {z ₁, z ₃}, and were 2.014 for {z ₃, z ₃}. The three-strategy mixture had a complexity of 3.05. The raw Matlab ^© code we used for this example is available as supplementary online material (http://journal.sjdm.org/vol6.8.html).

Table 4: NML values for simulated Data.

Figure 2 plots the observed choice proportions of the simulated data and labels each point according to the best-fitting strategy-specification pair for those data. The mixture polyhedron(s) are plotted in this space to give the reader a sense of the geometry of the different specifications. It is interesting to note that the amount of choice variability as well as the relative location of the data influence which strategy-specification pair yields the optimal NML value. The structure of the choice variability also plays a large role in determining which stochastic specification best accounts for the data. If the variability is roughly equal across paired comparisons then two-strategy mixture and single state error specifications perform well. If the choice variability is not uniform across paired comparisons then the three-strategy mixture specification tends to perform well.

Figure 2: Classification of the 25 simulated data points under all stochastic specifications for the TTB, DR, and FR strategies.

To illustrate the differences between using NML as compared to traditional model selection criteria, we calculated BIC values for each strategy-specification pair over all 25 simulated data sets.⁷ Table 5 lists these BIC values with optimal values listed in bold, non-unique optimal values are in bold and underlined. The NML and BIC criteria select the same optimal strategy-specification pair for 76% of the simulated data sets. They disagree, however, in two systematic ways. First, several simulated data points are classified as two-strategy mixtures according to BIC, whereas NML classifies them as being best explained via a three-strategy mixture specification. This discrepancy comes from BIC “over-penalizing” the three-strategy mixture by only considering the number of parameters (two in this case) and not taking into account the constraints on that space, as NML does. Said another way, BIC penalizes the three-strategy mixture model as if its parameter space spanned the entire unit square as opposed to spanning only half of it—see Figures 1 and 2. Second, three of the simulated data sets select multiple optimal strategy-specification pairs. In all three cases, a high-error and two-strategy mixture specification have identical optimal BIC values and therefore cannot be distinguished, see Table 5. This problem arises because these two specifications produce identical maximized likelihood values, and, since the BIC criteria considers them equally complex (single parameter models), they are penalized equally. Normalized maximum likelihood, on the other hand, correctly determines that the two-strategy mixture specifications are more complex and thus makes the correct classification.

Table 5: BIC values for simulated Data

4.1 Tversky intransitivity Data

We applied our methodology to the Tversky (1969) Experiment 1 intransitivity of preference data set. The Appendix contains a detailed description of Tversky’s experimental method as well as the results of our statistical analysis. In summary, we analyzed eight subjects while considering three decision strategies, two transitive and one intransitive. We considered seven possible strategy-specification pairs, including both low and high “error” specifications (є ≤ .03 and є ≤ .20) as well as a mixture specification. We found that Subjects 1 and 6 were best fit by an intransitive strategy with a high-error specification, while the remaining subjects were best fit by a mixture specification over transitive strategies. We conclude that evidence for intransitivity of preference is relatively weak. Our conclusions generally support several previous re-analyses and replications of this experimental paradigm (Birnbaum, 2010; Birnbaum & Gutierrez, 2007; Birnbaum & LaCroix, 2008; Iverson & Falmagne, 1985; Regenwetter et al., 2011). Due to the number of choice alternatives under consideration, we estimated these NML values via Monte Carlo simulation.

5 Discussion

In this article, we presented an outcome-based strategy classification methodology that assigns to a DM a best-fitting strategy-specification pair given a set of choice data. Our methodology allows for the evaluation of many stochastic specifications, including probabilistic mixtures of strategies. This methodology is based upon a binomial (multinomial) random variable framework and uses normalized maximum likelihood as a model selection criteria. This methodology generalizes the basic approach of Bröder and Schiffer (2003a) to multiple stochastic specifications using a different model selection criteria. We illustrated our approach using data from a well-known choice experiment and found that some DMs were best fit by a single strategy with “error” specification while others were best fit by a probabilistic mixture over multiple strategies.

While we focused on deterministic decision strategies, this perspective could be applied more generally to stochastic decision theories. Here we refer to theories which explicitly account for choice variability via random variables, stochastic processes, etc. One could simply apply the same NML criteria to any stochastic decision theory that could be estimated via choice data and carry out model selection in a similar fashion. These stochastic theories would not require an additional stochastic specification per se, although multiple stochastic theories could be combined via a mixture process.

For most applications of our framework to real-world data, it is critical to determine the set of strategies and types of stochastic specifications a priori. This leads to two natural questions. First, what decision strategies are reasonable? Fiedler (2010) recently pointed out that, given enough strategies, one could model nearly any decision process. Under our framework, considering a large number of strategies is not necessarily a problem, nor is considering a mixture specification over a large number of strategies as this very complex model would be sufficiently penalized for its ability to accommodate many data sets. However, as with any model selection framework, it is critical that strategies of interest are put up against legitimate competitor candidate strategies and stochastic specifications—not “straw models” to be trivially rejected. Second, what stochastic specifications are reasonable? This will depend upon both the types of decision strategies that the researcher is interested in evaluating as well as the experimental paradigm itself. For example, a researcher may be justified in selecting very large error bounds on single-strategy specifications for an experiment that involves presenting “noisy” unidimensional information to a subject.

There are currently two major limitations of our approach. First, similar to Bröder and Schiffer (2003a), we require all strategies to make distinct predictions, although one could differentiate strategies that made identical predictions based on stochastic specification if theoretically warrented. In this respect, increasing the number of distinct choice pairs can often assist in distinguishing between strategies as this allows more opportunities for different strategies to disagree. Glöckner (2009; 2010) overcomes this limitation within a maximum likelihood classification framework by incorporating additional dependent variables such as decision time and confidence data. Second, calculation of the NML criteria can be very computationally demanding. For large values of |K| and/or N _ab, our method estimates the denominator of the NML value via Monte Carlo estimation, which is well-known to have relatively slow convergence rates and does not easily scale up to very high dimensional data. Future work could explore more efficient methods of estimation.

This methodology is well-suited to evaluate how the nature of variability of choice changes over time. It would be possible to consider choice data collected over multiple time points. Given a set of choice data at each time point, our methodology could be used to determine if the best-fitting stochastic specification type, as well as strategy, changes over time. Earlier approaches have found that random “error” tends to dissipate over time with “substantive” shifts of preference, via mixture specifications, remaining (Loomes et al., 2002). Future work could also investigate how our methodology compares with other methods of strategy classification, including “accordance rate” methods.