This target article is concerned with the implications of the surprisingly different experimental practices in economics and in areas of psychology relevant to both economists and psychologists, such as behavioral decision making. We consider four features of experimentation in economics, namely, script enactment, repeated trials, performance-based monetary payments, and the proscription against deception, and compare them to experimental practices in psychology, primarily in the area of behavioral decision making. Whereas economists bring a precisely defined “script” to experiments for participants to enact, psychologists often do not provide such a script, leaving participants to infer what choices the situation affords. By often using repeated experimental trials, economists allow participants to learn about the task and the environment; psychologists typically do not. Economists generally pay participants on the basis of clearly defined performance criteria; psychologists usually pay a flat fee or grant a fixed amount of course credit. Economists virtually never deceive participants; psychologists, especially in some areas of inquiry, often do. We argue that experimental standards in economics are regulatory in that they allow for little variation between the experimental practices of individual researchers. The experimental standards in psychology, by contrast, are comparatively laissez-faire. We believe that the wider range of experimental practices in psychology reflects a lack of procedural regularity that may contribute to the variability of empirical findings in the research fields under consideration. We conclude with a call for more research on the consequences of methodological preferences, such as the use on monetary payments, and propose a “do-it-both-ways” rule regarding the enactment of scripts, repetition of trials, and performance-based monetary payments. We also argue, on pragmatic grounds, that the default practice should be not to deceive participants.

The methods of experiments in the social sciences should depend on their purposes. To support this claim, I attempt to state some general principles relating method to purpose for three of the issues addressed. (I do not understand what is not a script, so I will omit that issue.) I illustrate my outline with examples from psychological research on judgment and decision making (JDM).

Hertwig and Ortmann suggest paying participants contingent upon performance in order to increase the thoroughness they devote to a decision task. We argue that monetary incentives can yield a number of unintended effects including distortions of the subjective representation of the task and impaired performance. Therefore, we conclude that performance-contingent payment should not be generally employed in judgment and decision research.

While correctly describing the differences in current practices between experimental psychologists and economists, Hertwig and Ortmann do not provide a compelling explanation for these differences. Our explanation focuses on the fact that psychologists view the world as composed of categories and types. This discrete organizational scheme results in merely testing nulls and wider variation in observed practices in experimental psychology.

Hertwig and Ortmann suggest methodological practices from economics (script enactment, repeated measures, performance based payments, and absence of deception) for psychology. Such prescriptive methodologies may be unrepresentative of real world behaviors because people are not: always behaving with complete information, monetarily rewarded for important activities, repeating tasks to perfection, aware of all contributing variables. These proscriptions, while useful in economics, may obfuscate important psychological phenomena.

This commentary extends Hertwig & Ortmann's analysis by asking how stricter model selection conventions can facilitate the accumulation of information from experimental studies. In many cases researchers are currently motivated to summarize their data with ambiguous and/or multi parameter models. A “generality first” convention can help eliminate this problem.

Experimental psychologists in the learning tradition stress the importance of three of the authors' four key variables of experimental design. We review research investigating the roles played by these variables in studies of choice from our laboratory. Supporting the authors' claims, these studies show that the effects of these variables are not fixed and should not be taken for granted.

Most students are trained in using but not in actively choosing a research methodology. I support Hertwig and Ortmann's call for more rationality in the use of methodology. I comment on additional practices that sacrifice experimental control to the experimenter's convenience, and on the strange fact that such laissez-faire attitudes and rigid intolerance actually co-exist in psychological research programs.

I argue that (1) the accusation that psychological methods are too diverse conflates “reliability” with “validity”; (2) one must not choose methods by the results they produce – what matters is whether a method acceptably models the real-world situation one is trying to understand; (3) one must also distinguish methodological failings from differences that arise from the pursuit of different theoretical questions.

Features of experimental design impose auxiliary hypotheses on experimenters. Hertwig & Ortmann rightly argue that the ways some variables are implemented in psychology cloud results, whereas the different implementations in economics provide for more robust results. However, not all design variables support this general conclusion. The repetition of trials may confuse results depending on what theory is being tested. We explore this in the case of simple bargaining games.

Scripts and deception are alternative means, both imperfect, to the goal of simulating an environment that cannot be created readily. Under scripts, participants pretend they are in that environment, while deception convinces participants they are in that environment although they are not. With repeated trials, they ought to be unnecessary. But they are not, which poses challenges to behavioral sciences.

Methodological practices differ between economics and psychology because economists use game theory as the basis for the design and interpretation of experiments, while psychologists do not. This methodological choice explains the “four key variables” stressed by Hert-wig and Ortmann. Game theory is currently the most rigorous basis for modeling strategic choice.

Clear-cut designs have a number of methodological virtues, with respect to internal and external validity, which I illustrate by means of informal causal analysis. In contrast, a more uniform experimental practice across disciplines may not lead to progress if causal relations in the human sciences are highly dependent on the details of the context.

Psychologists can learn from the procedural conventions of experimental economics. But the rationale for those conventions must be examined and understood lest they become constraints. Field referents and the choice of heuristic, matter for behavior. This theme unites the fields of experimental psychology and experimental economics by the simple fact that the object of study in both cases is the same.

This commentary suggests: (1) experimentalists must expand their subject pools beyond university students; (2) the pollution created by deception would not be a problem if experimentalists fully used non-student subjects; (3) one-shot games remain important and repeated games should not ignore social learning; (4) economists need to take better control of context; and (5) using computers in experiments creates potential problems.

I suggest that contemporary economics shares many of the characteristics of methodological behaviourism in psychology, with its emphasis on the influence of motivation, learning, and situational incentives on behaviour, and minimal interest in the details of the cognitive processes that transform input (information) into output (behaviour). The emphasis on these characteristics has the same strengths and weaknesses in economics as in behaviourist psychology.

In conducting experiments, economists take more care than psychologists in specifying characteristics of people and tasks and are clearer about the conditions to which results can be generalized. Psychologists could learn much from this practice. They should also think in terms of programs of research – involving laboratory and field studies – that will allow them to test the generality of their theories.

The use of high hypothetical payoffs has been justified by the realism and relevance of large monetary consequences and by the impracticality of making high cash payments. We argue that subjects may not be able to imagine how they would behave in high payoff situations.

The practices of economists increase experimental reproducibility relative to those of selected psychologists but should not be universally adopted. Procedures criticized by Hertwig and Ortmann as producing variable data are valuable, instead, for generating questions. The procedure of choice should depend on the theoretical goal: measure a known factor or learn what factors are important and need to be measured.

Decision making can be studied using hypothetical payoffs because it is hypothetical to its very core. However, the core process can be influenced by contextual features. As there is no theory for these contextual features, a “do-it-both-ways” rule amounts to a waste of money. If we had such a theory, doing it both ways would be unnecessary.