Hostname: page-component-78c5997874-mlc7c Total loading time: 0 Render date: 2024-11-14T10:09:32.835Z Has data issue: false hasContentIssue false
  • English
  • Français

The value of uncertainty: An active inference perspective

Published online by Cambridge University Press:  19 March 2019

Giovanni Pezzulo  and
Karl J. Friston
Giovanni Pezzulo
Affiliation:
Institute of Cognitive Sciences and Technologies, National Research Council, 00185 Rome, Italy. giovanni.pezzulo@istc.cnr.ithttp://www.istc.cnr.it/people/giovanni-pezzulo
Karl J. Friston
Affiliation:
Wellcome Trust Centre for Neuroimaging, University College London, London WC1N 3BG, UK. k.friston@ucl.ac.ukhttp://www.fil.ion.ucl.ac.uk/Friston/
Get access Rights & Permissions [Opens in a new window]

Abstract

We discuss how uncertainty underwrites exploration and epistemic foraging from the perspective of active inference: a generic scheme that places pragmatic (utility maximization) and epistemic (uncertainty minimization) imperatives on an equal footing – as primary determinants of proximal behavior. This formulation contextualizes the complementary motivational incentives for reward-related stimuli and environmental uncertainty, offering a normative treatment of their trade-off.

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 42 , 2019 , e47
Check for updates
Copyright
Copyright © Cambridge University Press 2019 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Attias, H. (2003) Planning by probabilistic inference. In: Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics (AISTATS), Key West, Florida, January 3–6. https://dblp.uni-trier.de/rec/bibtex/conf/aistats/Attias03.Google Scholar
Baldassarre, G. & Mirolli, M., eds. (2013) Intrinsically motivated learning in natural and artificial systems. Springer.Google Scholar
Behrens, T. E. J., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. S. (2007) Learning the value of information in an uncertain world. Nature Neuroscience 10(9):1214–21. http://doi.org/10.1038/nn1954.Google Scholar
Berlyne, D. E. (1960) Conflict, arousal, and curiosity. McGraw-Hill.Google Scholar
Berridge, K. C. (2004) Motivation concepts in behavioral neuroscience. Physiology & Behavior 81:179–209.Google Scholar
Botvinick, M. & Toussaint, M. (2012) Planning as inference. Trends in Cognitive Sciences 16:485–88. https://doi.org/10.1016/j.tics.2012.08.006.Google Scholar
Charnov, E. L. (1976b) Optimal foraging, the marginal value theorem. Theoretical Population Biology 9:129–36. doi: 10.1016/0040-5809(76)90040-X.Google Scholar
Christiansen, A. D., Mason, M. T. & Mitchell, T. M. (1991) Learning reliable manipulation strategies without initial physical models. Robotics and Autonomous Systems (Special Issue: Toward Learning Robots) 8:7–18. https://doi.org/10.1016/0921-8890(91)90011-9.Google Scholar
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. (2006) Cortical substrates for exploratory decisions in humans. Nature 441(7095):876–79. http://doi.org/10.1038/nature04766.Google Scholar
Dayan, P. & Sejnowski, T. J. (1996) Exploration bonuses and dual control. Machine Learning 25:5–22.Google Scholar
Donnarumma, F., Maisto, D. & Pezzulo, G. (2016) Problem solving as probabilistic inference with subgoaling: Explaining human successes and pitfalls in the tower of Hanoi. PLOS Computational Biology 12:e1004864. https://doi.org/10.1371/journal.pcbi.1004864Google Scholar
Friston, K. (2010) The free-energy principle: A unified brain theory? Nature Reviews Neuroscience 11:127–38. https://doi.org/10.1038/nrn2787Google Scholar
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O'Doherty, J. & Pezzulo, G. (2016a) Active inference and learning. Neuroscience & Biobehavioral Review 68:862–79. https://doi.org/10.1016/j.neubiorev.2016.06.022Google Scholar
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P. & Pezzulo, G. (2016b) Active inference: A process theory. Neural Computation 29:1–49. https://doi.org/10.1162/NECO_a_00912.Google Scholar
Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T. & Pezzulo, G. (2015) Active inference and epistemic value. Cognitive Neuroscience 6(4):187–214. https://doi.org/10.1080/17588928.2015.1020053.Google Scholar
Friston, K., Schwartenbeck, P., FitzGerald, T., Moutoussis, M., Behrens, T. & Dolan, R. J. (2014) The anatomy of choice: Dopamine and decision-making. Philosophical Transactions of the Royal Society: Biological Sciences 369:20130481. https://doi.org/10.1098/rstb.2013.0481.Google Scholar
Friston, K. J., Lin, M., Frith, C. D., Pezzulo, G., Hobson, J. A. & Ondobaka, S. (2017) Active inference, curiosity and insight. Neural Computation 29(10):2633–2683. https://doi.org/10.1162/neco_a_00999.Google Scholar
Gallistel, C. R. & Gibbon, J. (2001) Computational versus associative models of simple conditioning. Current Directions in Psychological Science 10:146–50.Google Scholar
Gottlieb, J., Oudeyer, P.-Y., Lopes, M. & Baranes, A. (2013) Information-seeking, curiosity, and attention: Computational and neural mechanisms. Trends in Cognitive Sciences 17:585–93. https://doi.org/10.1016/j.tics.2013.09.001Google Scholar
Hayden, B. Y., Pearson, J. M. & Platt, M. L. (2011) Neuronal basis of sequential foraging decisions in a patchy environment. Nature Neuroscience 14:933–39. doi: 10.1038/nn.2856.Google Scholar
Inglis, I. R. (2000) The central role of uncertainty reduction in determining behaviour. Behaviour 137:1567–99. https://doi.org/10.1163/156853900502727Google Scholar
Inglis, I. R., Langton, S., Forkman, B. & Lazarus, J. (2001) An information primacy model of exploratory and foraging behaviour. Animal Behavior 62:543–57. https://doi.org/10.1006/anbe.2001.1780Google Scholar
Iodice, P., Ferrante, C., Brunetti, L., Cabib, S., Protasi, F., Walton, M. E. & Pezzulo, G. (2017) Fatigue modulates dopamine availability and promotes flexible choice reversals during decision making. Scientific Reports 7:535. https://doi.org/10.1038/s41598-017-00561-6.Google Scholar
Maisto, D., Donnarumma, F. & Pezzulo, G. (2015) Divide et impera: Subgoaling reduces the complexity of probabilistic inference and problem solving. Journal of the Royal Society Interface 12:20141335. https://doi.org/10.1098/rsif.2014.1335.Google Scholar
Oudeyer, P.-Y., Kaplan, F. & Hafner, V. (2007) Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11:265–86.Google Scholar
Pezzulo, G., Cartoni, E., Rigoli, F., Pio-Lopez, L. & Friston, K. (2016) Active inference, epistemic value, and vicarious trial and error. Learning & Memory 23:322–38. https://doi.org/10.1101/lm.041780.116Google Scholar
Pezzulo, G. & Rigoli, F. (2011) The value of foresight: How prospection affects decision-making. Frontiers in Neuroscience 5:79.Google Scholar
Pezzulo, G., Rigoli, F. & Chersi, F. (2013) The mixed instrumental controller: Using value of information to combine habitual choice and mental simulation. Frontiers in Cognition 4:92. https://doi.org/10.3389/fpsyg.2013.00092Google Scholar
Pezzulo, G., Rigoli, F. & Friston, K. J. (2015) Active inference, homeostatic regulation and adaptive behavioural control. Progress in Neurobiology 134:17–35.Google Scholar
Pezzulo, G., Rigoli, F. & Friston, K. J. (2018) Hierarchical active inference: A theory of motivated control. Trends in Cognitive Sciences 22(4):294–306. https://doi.org/10.1016/j.tics.2018.01.009.Google Scholar
Salamone, J. D., Correa, M., Farrar, A. M., Nunes, E. J. & Pardo, M. (2009) Dopamine, behavioral economics, and effort. Frontiers in Behavioral Neuroscience 3:13. https://doi.org/10.3389/neuro.08.013.2009.Google Scholar
Schmidhuber, J. (1991) Adaptive confidence and adaptive curiosity (No. FKI-149-91). Institut für Informatik, Technische Universitat.Google Scholar
Schwartenbeck, P., FitzGerald, T., Dolan, R. & Friston, K. (2013) Exploration, novelty, surprise, and free energy minimization. Frontiers in Psychology 4:710.Google Scholar
Singh, S., Barto, A.G. & Chentanez, N. (2005) Intrinsically motivated reinforcement learning. In: Advances in neural information processing systems, vol. 17, ed. Saul, L. K., Weiss, Y., & Bottou, L., pp. 1281–88. MIT Press.Google Scholar
Stoianov, I., Genovesio, A. & Pezzulo, G. (2015) Prefrontal goal codes emerge as latent states in probabilistic value learning. Journal of Cognitive Neuroscience 28:140–57.Google Scholar
Stoianov, I., Pennartz, C., Lansink, C., & Pezzulo, G. (2018) Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis. PLoS Computational Biology 14(9):e1006316. https://doi.org/10.1371/journal.pcbi.1006316.Google Scholar
Sutton, R. S. (1990) Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: Proceedings of the Seventh International Conference on Machine Learning, ed. Porter, B. W. & Mooney, R. J., pp. 216–24. Morgan Kaufmann.Google Scholar
Walton, M., Kennerley, S., Bannerman, D., Phillips, P. & Rushworth, M. (2006) Weighing up the benefits of work: Behavioral and neural analyses of effort-related decision making. Neural Networks 19:1302–14. https://doi.org/10.1016/j.neunet.2006.03.005Google Scholar
Woodworth, R. S. (1958) Dynamics of behavior. Henry Holt.Google Scholar