Exploration behavior of task-related attributes in uninformed reinforcement learning task

Abstract

participants were not informed of the rule and were to find optimal strategies to achieve their goals. During behavioral tasks, subjects' performance and the fMRI signals were simultaneously recorded. To examine the exploratory behavior of new attributes and the learning processes for newly acquired attributes, we fitted behavior data into two computational models - two separate probabilistic policy searching model using HMM or soft-max function and two distinctly initialized RL learning models. The comparison within each of two models demonstrates that the participants explores under high cognitive ambiguity and learns new values inferring from previous policies. After confirming that information is explored and learning occurs with regard to previous values, we examined the related brain areas from the fMRI data. The state-action value signals, the reward prediction error signals, the cognitive entropy signals, which is cognitive ambiguity that induces exploration, and policy transition time-points were extracted from the model and detected in fMRI results by GLM analysis. As previously reported, the value signals and the error signals were observed in ventromedial prefrontal cortex (vmPFC) and anterior cingulate cortex respectively. The exploration related signals, transition time points and entropy, were activated in the areas including exploration related area frontopolar cortex (FPC). These results indicate that higher cognition areas including vmPFC, ACC, and FPC work as a meta-controller and integrator of value and error signals. Overall, our result explains how our brain explores and learn new attributes for making a decision during reinforcement learning. Our study, therefore, suggests how human effectively processes new information when the dimension is extended.

We often face uncertain decision making situations, in which we don't aware of full components to accomplish the tasks. However, the majority of studies were conducted with full explanation of including factors, only a few studies were performed along insufficient task information owing to experimental difficulty and hardship in making computational model. Beside active investigation on exploration of choice options, no previous study has yet investigated how the brain explores new attributes during learning trials. In this study, we investigated how the brain integrates, so that explores new information and learning processes during inadequately informed multi-dimensional reinforcement learning task using computational models including hidden markov model (HMM), soft-max function, and reinforcement learning (RL) and fMRI

we observed that our brain integrates pieces of information in a sequential manner and that the insula is responsible for the exploration process. 29 subjects participated in the multi-dimensional behavioral task. Subjects were shown pictures with multiple features-shapes, color, and patterns-and asked to collect as many points (reward) as possible