A Two-Layered Diffusion Model Traces the Dynamics of Information Processing in the Valuation-and-Choice Circuit of Decision Making

A circuit of evaluation and selection of the alternatives (see here) is considered a reliable model in neurobiology.

In this published study, valuation and choice of a decisional process during Two-Alternative Forced-Choice (TAFC) task are represented as a two-layered network of computational cells, where information accrual and processing progress in nonlinear diffusion dynamics. 

The evolution of the response-to-stimulus map is thus modeled by two linked diffusive modules (2LDM) representing the neuronal populations involved in the valuation-and-decision circuit of decision making (Figure 1). Diffusion models are naturally appropriate for describing accumulation of evidence over the time [see here]. This allows the computation of the response times (RTs) in valuation and choice, under the hypothesis of ex-Wald distribution. A nonlinear transfer function integrates the activities of the two layers. The input-output map based on the infomax principle makes the 2LDM consistent with the reinforcement learning approach. Results from simulated likelihood time series indicate that 2LDM may account for the activity-dependent modulatory component of effective connectivity between the neuronal populations. Rhythmic fluctuations of the estimate gain functions in the delta-beta bands also support the compatibility of 2LDM with the neurobiology of DM.

Figure 1. The two-layered diffusion model (2LDM) for decision making. 

Both stages (valuation and choice) are affected by noise. In the valuation stage the critical threshold indicates the firing rate of the neuronal populations involved, to which would correspond the expected reward. The outputs of this stage then are the differences between the responses of observed neuronal activity at the stimuli provided by the alternatives and the target. These measurements enter the next stage, where the decision is taken so as to optimize some utility criterion (reward). Hence, the attainment of the threshold in the decision stage indicates the preferred alternative. Feedback information flows from the decision stage in order to elicit the adaptation of the boundary in the valuation layer. In this way, a mechanism of reinforcement determines the competition between the alternatives and the valuation is biased to the most probable rewarded one.




The exploration of the neurobiological bases of the cognitive processes that underlie the decision-making (DM) have been the object of many studies of neurophysiology and computational neuroscience [1-8]. By tracing the neuronal circuits that are involved in the DM it is possible to get biophysically reliable models linking the dynamics of the neuronal activities to decisional behavior. Actually, DM is a process that involves different areas of the brain. These regions include the cortical areas that are supposed to integrate evidence supporting alternative actions, and the basal ganglia (BG,  see here the basal ganglia circuit), that are hypothesized to act as a central switch in gating behavioral requests [9-15]. In natural environments several sensory stimuli produce different alternatives and hence demand the evaluation of different possible responses, i.e. a variety of behaviors. In other terms, it arises also a selection question [12] whereby the (probability) distribution of the correct response has to take control of the individual’s motor plant [16]. The action selection then would resolve a conflict among decisional centers throughout the brain. A central switch that considers the urgency and opportunity of specific response to the stimuli result an optimal solution in computational terms and physiologically reliable by taking the BG as the neural base for that switch. Accordingly, BG gather input from all over the brain and by sending tonic inhibition to midbrain and brain stem targets involved in motor actions, block the cortical control over these actions [9,10,17]. Therefore, the inhibition of the neurons in the output nuclei, caused by BG activity, determines the disinhibition of their targets and the actions would be consequently selected. This model, ultimately explains that in the DM among alternative options, the cortical areas associated with the alternatives integrate their corresponding evidence, whilst the BG by acting as a central switch evaluates the evidence and facilitates the best supported responses (behaviors) [16]. Many studies have also reported a significant increase in the firing rate of the neurons of cortical areas representing the alternative choices during DM in visual tasks. The increase of the firing rates then would provide accumulation of evidence (i.e., information) related to the alternatives [13,14]. Reliable models of DM based on the neurophysiology, consider connections from neurons representing stimuli to the appropriate cortical neurons representing decisions (e.g., motor actions). 


There is a theoretical linkage between 2LDM and the well-recognized integrate-and-fire attractor network model [18–21] since both models rely on nonlinear diffusive dynamics. Major difference rests in the expected dynamics of the basal ganglia involved during the decision making process, which we considered driven by nonlinear patterns rather than linear patterns. Furthermore, the characterization of the input-output map in terms of the infomax principle makes, ultimately, the 2LDM an entropy-thresholding algorithm where the model’s parameters (threshold, diffusion noise, and drift) should be tuned to maximize the mutual information between the representations they engender and the inputs that feed the layers. This is consistent with the Q-learning adaptation, since learning the “best” action on the two thresholds to maximize the cumulative entropy is equivalent to learning the optimal behavior which maximizes the reward [22, 23]. Nonlinearity in the 2LDM is given by static linear-nonlinear functions that express the gain of the input-output map, so overcoming the theoretical weakness inherent in the canonical diffusion models which assume that momentary evidence is accumulated continuously and at constant rate, that is, linearly, until a decision threshold is reached. This way to model nonlinear dynamics is not a novelty in neuroscience because it fits for Volterra series representation which, through the first- and second-order kernels, estimates the driving and modulatory influence that one population exerts on the other. The slope of sigmoidal transfer function yields information about the effective connectivity between the neuronal populations, because it is a proxy of the Volterra kernels [24].

REFERENCES

1. Platt, M.L. & Glimcher, P.W. (1999). Neural correlates of decision variables in parietal cortex. Nature 400, 233–238.
2. Sugrue, L.P., Corrado, G.S. & Newsome, W.T. (2004). Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787.
3. Tom, S.M., Fox, C.R., Trepel, C. & Poldrack, R.A. (2007).The neural basis of loss aversion in decision-making under risk. Science 315, 515–518.
4. Plassmann, H., O’Doherty, J.P. & Rangel, A. (2007). Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J Neurosci 27, 9984–9988.
5. Knutson, B., Taylor, J., Kaufman, M., Peterson, R. & Glover, G. (2005). Distributed neural representation of expected value. J Neurosci 25, 4806–4812.
6. Boorman, E.D., Behrens, T.E.J., Woolrich, M.W. & Rushworth, M.S.F. (2009). How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62, 733–743.
7. Blair, K., Marsh, A., Morton, J., Vythilingam, M., Jones, M. ,Mondillo, K., Pine, D.C., Drevets, W.C., Blair, J.R. (2006). Choosing the lesser of two evils, the better of two goods: specifying the roles of ventromedial prefrontal cortex and dorsal anterior cingulate in object choice. J Neurosci 26, 11379–11386.
8. Kable, J.W. & Glimcher, P.W. (2007). The neural correlates of subjective value during intertemporal choice. Nat Neurosci 10: 1625-1633.
9. Chevalier, G., Vacher, S., Deniau, J. M., Desban, M. (1985). Disinhibition as a basic process in the expression of striatal functions. I. The striato-nigral influence on tectospinal/tecto-diencephalic neurons. Brain Res, 334(2), 215-226.
10. Deniau, J. M., Chevalier, G. (1985). Disinhibition as a basic process in the expression of striatal functions. II. The striato-nigral influence on thalamocortical cells of the ventromedial thalamic nucleus. Brain Res, 334(2), 227-233.
11. Medina, L., Reiner, A. (1995). Neurotransmitter organization and connectivity of the basal ganglia in vertebrates: implications for the evolution of basal ganglia. Brain Behav Evol, 46(4-5), 235-258.
12. Redgrave, P., Prescott, T. J., Gurney, K. (1999). The basal ganglia: a vertebrate solution to the selection problem? Neurosci 89(4), 1009-1023.
13. Schall, J. D. (2001). Neural basis of deciding, choosing and acting. Nat Rev Neurosci, 2(1), 33-42.
14. Shadlen, M. N., Newsome, W. T. (2001). Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. J Neurophysiol, 86(4), 1916-1936.
15. Smith, Y., Bevan, M. D., Shink, E., Bolam, J. P. (1998). Microcircuitry of the direct and indirect pathways of the basal ganglia. Neurosci, 86(2), 353-387.
16. Bogacz, R., Gurney, K. (2007). The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Comput 19(2):442-477.
17. Parent, A., Hazrati, L. N. (1995). Functional anatomy of the basal ganglia. I. The cortico-basal ganglia thalamocortical loop. Brain Res Brain Res Rev, 20(1), 91-127.
18. G. Deco, E. T. Rolls, and R. Romo, (2009). Stochastic dynamics as a principle of brain function. Progress in Neurobiology, 88 (1),1–16.
19. E. T. Rolls, Emotions and Decision-Making Explained, Oxford University Press, 2014.
20. G. Deco, E. T. Rolls, L. Albantakis, and R. Romo (2013). Brain mechanisms for perceptual and reward-related decision-making. Progress in Neurobiology, 103, 194–213. 
21. A. Insabato, M. Pannunzi, E. T. Rolls, and G. Deco (2010). Confidence-related decision making. J Neurophysiol, 104 (1), 539–547.
22. P. Yin (2002). Maximum entropy-based optimal threshold selection using deterministic reinforcement learning with controlled randomization. Signal Processing, 82 (7), 993–1006. 
23. J. N. Kapur, P. K. Sahoo, and A. K. C. Wong (1985). A new method for gray-level picture thresholding using the entropy of the histogram. Computer Vision, Graphics, & Image Processing, 29 (3), 273–285.
24. S. Ostojic and N. Brunel (2011). From spiking neuron models to linear-nonlinear models. PLoS Computational Biology, 7 (1), Article ID e1001056.