You are here
Cognitive evaluation by tasks in a virtual reality environment in multiple sclerosis
Journal of the Neurological Sciences, Volume 359, Issue 1-2, December 2015, Pages 94 - 99
Background: The assessment of cognitive impairment in multiple sclerosis (MS) requires large neuropsychological batteries that assess numerous domains. The relevance of these assessments to daily cognitive functioning is not well established. Cognitive ecological evaluation has not been frequently studied in MS.
Objectives: The aim of this study was to determine the interest of cognitive evaluation in a virtual reality environment in a sample of persons with MS with cognitive deficits.
Methods: Thirty persons with MS with at least moderate cognitive impairment were assessed with two ecological evaluations, an in-house developed task in a virtual reality environment (Urban DailyCog®) and a divided attention task in a driving simulator. Classical neuropsychological testing was also used.
Results and conclusion: Fifty-two percent of the persons with MS failed the driving simulator task and 80% failed the Urban DailyCog®. Virtual reality assessments are promising in identifying cognitive impairment in MS.
- Cognitive testing in virtual reality environment is feasible in multiple sclerosis.
- Virtual reality tasks are considered pleasant by patients.
- Virtual reality assessments are sensitive in identifying cognitive impairment in MS.
Keywords: Multiple sclerosis, Virtual reality, Cognition, Cognitive assessment, Ecological evaluation, Neuropsychology.
Cognitive impairment (CI) has been increasingly recognized as a major contributor to the disability of persons living with multiple sclerosis (PwMS) . To evaluate the entire spectrum of cognitive deficits, large neuropsychological batteries are required including tests that assess information processing speed (IPS), episodic memory (EM), working memory (WM), executive functions (EF) and attention . However, this comprehensive NP assessment lasts at least 2 h and it is therefore rarely feasible in daily practice. Several shorter batteries have been proposed such as the Brief Repeatable Battery of NP tests and the Minimal Assessment of Cognitive Function in MS . These batteries include not only tests for IPS and EM, which are the most frequently impaired functions, but also tests for WM and some EF. However, these relatively concise batteries take between 40 and 90 min to perform and applying these batteries to every patient is not easy. Recently, the Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS) has been proposed as a short assessment . This assessment tool is composed of only 3 NP tests, the symbol digit modalities tests (SDMT) , the learning trials of the California Verbal Learning Test-Second Edition (CVLT-II)  and the Brief Visuospatial Memory Test-Revised (BVMT-R) . The SDMT is a test of IPS that has been shown to be very sensitive in MS  and has been proposed for use in the MS Functional Composite . The CVLT and the BVMT-R are two tests of EM, the former for verbal EM and the latter for visuospatial EM. This battery takes approximately 15 min and is more feasible than the classical batteries. Validation of the BICAMS is in progress in several countries following published recommendations . Although the use of the BICAMS allows for cognitive assessment in more PwMS, even in centers in which neuropsychologists are not available, it has some limitations, specifically, it detects only patients impaired in the two main domains, IPS and EM. Indeed, other cognitive domains can also be affected such as working memory or EF . Moreover, it is well established that NP tests do not correlate well with patient-reported cognitive complaints and patient-related outcome tools  and . If classical NP tests offer the advantages of standardization, reproducibility, reliability and normalization, their ability to detect cognitive abilities used in daily life is questionable. An alternative to assessment in real situations, which is not easy, may be the use of virtual reality environments (VRE). In VRE, a person is able to perform sensory-motor or cognitive activity in an artificial world that is created digitally, which can be imaginary, symbolic, or a simulation of certain aspects of the real world . Tests using VRE allow for testing in a standardized and reproducible manner. Recently, some studies have shown the ability of VRE to assess behavioral and cognitive skills in specific populations and in some diseases , , and . Well-known tasks include the virtual supermarket  and , virtual classroom , virtual kitchen  or virtual library . Some NP tests have also been adapted in VRE , , and .
To the best of our knowledge, no published study has used VRE in MS. We developed a simple VRE task that was designed to detect attention deficits and working memory impairment. We also developed a task requiring attentional skills during a driving simulator session. We hypothesized that these more ecological skills could be impaired in the majority of PwMS with CI.
This study was performed as part of a study on cognitive rehabilitation (the REACTIV study, which is in progress), and the data presented here are baseline data. REACTIV has been approved by the local ethics committee (CPP Bordeaux).
PwMS were selected from the MS clinic (University Hospital of Bordeaux). We selected PwMS who complained of discomfort in their daily life due to cognitive problems during routine outpatient visits. The inclusion criteria were as follows: MS according to Polman et al.  (relapsing remitting, secondary progressive and primary progressive), age 18–55, disease duration > 6 months and ≤ 15 years, right-handedness, and having a driver's license. To be eligible, PwMS also had to fulfill a cognitive criterion of mild CI (2 scores < 1 standard deviation (SD) on tests from the whole battery).
The exclusion criteria (PwMS) were as follows: previous history of other neurological or psychiatric disorders, visual, oculomotor, auditory and motor impairments precluding the ability to perform computerized tasks, addictive behavior, MS attack in the two months preceding the screening, corticosteroid pulse therapy within two months preceding screening, severe cognitive deficits or dementia (Mini Mental Status Examination (MMS-E) < 27) , moderate to severe visuo-spatial incapacity (score < 28 on the Rey figure) , and moderate to severe depression (Beck Depression Inventory (BDI) > 27) .
2.2. Healthy subjects
Healthy subjects (HS) matched for age, gender and education were enrolled as controls.
3.1. Clinical assessment
A standardized neurological examination was performed by a senior neurologist to establish scores of the Expanded Disability Status Scale (EDSS) . Depressive symptoms were assessed using the Beck Depression Inventory (BDI) , and anxiety was measured by the State-Trait Anxiety Inventory (STAI) . Fatigue was assessed using the Modified-Fatigue Impact Scale (MFIS) .
3.2. Neuropsychological assessment
The entire battery of this study lasted 3 h and assessed IPS, attention EF, WM, and EM. The battery included paper-and-pencil tests and computerized sub-tests of the Test of Attentional Performances (TAP 2.1) , namely all NP tests except the VRE tests. The tests and cognitive domains are listed in Table 1.
|Tests||Subtests||Main domain considered||Variables||% of PwMS with CI (N = 30)|
|MMS-E ||Global cognitive efficiency||AAns||Non-inclusion criteria|
|Symbol digit modalities test||IPS||AA***,a
(in 90 s)
|Alertness of the TAP ||Intrinsic alertness (simple reaction time)||IPS
|Phasic arousal (stimulus preceded by a cue as a warning tone)||IPS
|Visual scanning of the TAP ||With target||IPS
|Divided attention of the TAP ||Simple condition: visual attention||IPS
|Simple condition: auditory attention||IPS
|Double condition: visual attention||IPS
|Double condition: auditory attention||IPS
|N-back of the TAP ||Working memory||RT**
|Stroop test ||Color naming||IPS||Time***, a||40%|
|Interference||EF||Time** (ratio inhibition: interference task time–reading task time)||23.3%|
|Trail-making test ||Parts A and B||EF (mental flexibility: B–A)||Time*||40%|
|Span of Baddeley Double task ||Forward span||WM||AA***, a||20%|
|Baddeley Double task ||Double task||Attention||Rations (MU)||10%|
|Verbal fluency ||Semantic||EF||AA* (in 120 s)||3.3%|
|Phonemic||EF||AA* (in 120 s)||16.7|
|Reverse span ||Backward span||WM||AA**||0%|
|California Verbal Learning Test ||Learning trials||EM||AA**||30%|
|Immediate recall||EM||AA***, a||56.7%|
|Immediate cued recall (indiced scores RICT)||EM||AA***, a||53.3%|
|Delayed recall (recall RLLT)||EM||AA**||26.7%|
|Delayed cued recall (indiced scores RILT)||EM||AA***, a||40%|
|Global CVLT (at least 1 of the 6 EM scores)||56.7%|
|Intrusions||EF||Errorsns (total number of intrusions made)||20%|
|Rey complex figure ||Figure scoring ||Visuo-constructive praxia||AA*||Non-inclusion criteria|
|Naming task (DO 80) ||Access to lexical store||AA*||26.7%|
AA: accurate answer; TR: reaction time; significant differences between PwMS and healthy subjects are indicated without Bonferroni correction as *p < 0.05, **p < 0.01, ***p < 0.001; ns: non-significant and after applying Bonferroni correction as a.
3.3. Urban DailyCog©
An in-house conceived task was developed in collaboration with Komenvoir Inc. (Toulouse, France) for computer graphics. The subject is seated in front of a screen on which the VRE is projected. An urban environment is featured, placing the subject in a car that is stopped at a street intersection (Fig. 1(a)). A traffic light is featured on the right side of the road that could change from red to green and the reverse. In this animated environment, the subject can see other cars moving and some featured people walking, entering and emerging from shops, waiting to cross the street and eventually crossing it. Three tasks were performed, the whole test lasting approximately 20 min. At the beginning of the first task, the traffic light was red. The subject was instructed to left-click with the computer mouse when the light turned green. This task was a simple reaction time (RT) task (alert). The inter-stimulus duration was random.
During the second task, the subject was instructed to right-click when a boy with a blue cap (Fig. 1(b)) came out of a shop. In the environment, other characters could be seen, including distractors (girl with blue cap, boy without a cap, man with blue cap) walking around. The subjects were supposed to do nothing when these distractors appear. This part of the task is designed to test choice RT (selective attention and inhibition). The third part of the test was a mixture of the two previous parts. In this task, the subject was required to monitor the green traffic light and the boy with the blue cap simultaneously. The aim was to study divided attention. The recorded parameters were the RT and the correct answers.
3.4. Driving Simulator Dual Task (DSDT)
The test was conducted on a driving simulator (Premium Simulator delivered by OKTAL Inc.). The simulator commands were similar to usual commands in real cars. The subject was immersed in a VRE that projected onto three screens representing a section of highway with mild traffic (Fig. 2).
Subjects performed one short training session and a testing session. During both sessions subjects were asked to drive in the right lane of the highway, to maintain one direction, to stay strictly within the lane and not cross the white line and to maintain a speed of 130 km/h. The testing session included several parts lasting 5 min each. During the third part, a divided attention task was proposed and the subject was instructed to continue to drive with the same instructions as before and to pay attention to a newscast broadcasted by the car radio and answer questions and count the number of countries mentioned in the radio program. The whole task lasted approximately 20 min. The recorded parameters were the means and SD of the position of the car on the road, the length of the road traveled, and the means and SD of the speed.
3.5. Convenience of the test
After the end of the study, all PwMS were contacted by email and phone and asked to score the pleasantness of the test between 0 (very unpleasant) and 5 (very pleasant).
CI for a specific cognitive domain was characterized when a PwMS exhibited CI based on at least two tests for IPS and one test for the other domains. CI for each of the VRE tests was characterized as when at least one of the scores (reaction times and correct answer scores for each of the 3 parts) for a PwMS was less than 1.5 SD of the HS scores. We analyzed CI at each of the VRE tasks and at the battery, which included all tests except the VRE tests and the tests used for inclusion criteria (Table 1). We also analyzed CI at the SDMT, which is considered to be the most sensitive cognitive test in MS  and . As IPS and EM are the two most affected domains in MS  and  we also analyzed the CI at the first two tests of the BICAMS , the SDMT  and the learning trials of the CVLT (CVLT-LT) . CI for the whole battery was defined as CI based on at least 3 tests. Statistical analyses were performed with Statview version 5.0 software for Windows. Unpaired t-tests were used to compare clinical characteristics, such as mean scores and age between PwMS and HS and Chi2 for sex and education level. Differences were considered significant when the p values were less than 5%. Correlations between cognitive VRE z scores and cognitive tests z scores and between cognitive z scores and BDI, STAI, MFIS, EDSS scores and disease duration were evaluated by Pearson's correlation coefficient. Considering the number of variables, the Bonferroni correction was applied.
5.1. Characteristics of PwMS and HS
Thirty PwMS (27 with relapsing remitting MS, two with primary progressive MS and one with secondary progressive MS) and 22 HS were studied. Table 2 describes the demographics and clinical characteristics of the two groups. There were no statistically significant differences in demographic variables between PwMS and HS. Although patients had significantly higher scores on the BDI, STAI and M-FIS scales than HS, there were no significant correlations between these scores and cognitive scores.
|PwMS (N = 30)||Healthy controls (N = 22)|
|Age (years)ns||41.7 ± 7.2||37.8 ± 9.2|
|Gender (% of women)ns||21/30 70%||72.7|
|Education (% > baccalaureate)ns||15/30 50%||54.5|
|Disease duration (years)||9.6 ± 6.1|
|EDSS median||2.5 (0–8)|
|BDI***||14.5 (0–26)||6 (0–14)|
|STAI Ans||34.5 (3–55)||29 (20–44)|
|STAI B***||46 (27–55)||34 (20–60)|
|MFIS***||Total***||53 (24–77)||12 (1–33)|
|Physical***||24.5 (13–35)||5 (0–14)|
|Cognitive***||25 (4–36)||5 (0–21)|
|Social***||4 (0–8)||0 (0–5)|
***: p < 0.001; ns: non-significant; age and disease duration are expressed as mean ± SD (standard deviation); gender and education are expressed in percentage; EDSS: Expanded Disability Status Scale; STAI: State-Trait Anxiety Inventory; BDI: Beck Depression Inventory; MFIS: Modified Fatigue Impact Scale; EDSS, BDI, STAI, and M-FIS are expressed as median (minimum–maximum).
5.2. Cognitive findings
The proportion of PwMS with CI for each test score is presented in Table 1. Fig. 3 shows the proportion of PwMS with CI for each cognitive domain. PwMS were impaired to varying degrees on all CI scores, excepted for the accurate answers of alertness and visual scanning or for the backward span. Fifty-two percent of the persons with MS failed the driving simulator task and 80% failed the Urban DailyCog®. Fig. 4 presents the proportion of PwMS with CI for the VRE tasks, the SDMT, the complete battery and the combination of the SDMT/CVLT-LT.
5.3. Correlations of cognitive VRE z scores and NP z scores
There was no significant correlation between the z-score of the Urban DailyCog and the z-scores of cognitive tests or cognitive domains derived from NP tests, or between the z score of the DSDT and the z-scores of cognitive tests or cognitive domains.
5.4. Other correlation studies
There were no significant correlations between the z-scores of the VRE tasks and BDI, STAI and MFIS scores. There were no significant correlations between the z-scores of the VRE evaluations and the EDSS score or the disease duration.
5.5. Comparison of the cognitive characteristics of the patients detected by the different batteries
In the 24 PwMS with CI according to the Urban DailyCog, CI in IPS was observed in 75.0%, in EF in 70.3%, in EM in 66.7%, in WM in 62.5%, in attention in 45.8%, and in language (lexical storage) in 25.0%. Among these 24 PwMS, only one did not have CI for the complete battery or for the combination SDMT/CVLT-LT. Six other PwMS, who were identified as having CI based on the Urban DailyCog but not according to the first two tests of the BICAMS, had CI in at least two cognitive domains. The main impaired cognitive domains of these PwMS were IPS (4), WM (3), and EF (3).
All the PwMS impaired in the DSDT were impaired according to the battery.
5.6. Convenience of the test
The mean score for the pleasantness phone survey was 4.09 ± 0.9 (n = 22) with a maximum score of 5. Eight patients could not be reached.
This study demonstrates the feasibility of VRE assessment of cognitive function in PwMS patients. The Urban DailyCog was easily completed by all the patients and was considered friendly by the participants. This test requires a computer with a sound system and could be easily performed by the patient alone by following instructions that were written on the screen at the beginning. After proper validation this test could be implemented online. The test is relatively quick to perform. Moreover, VRE allows for the exploration of the different cognitive domains and can improve the reliability of conventional assessments minimizing variability due to differences between examiners , , and .
The DSDT is not as easy to perform in clinical practice because it requires a specific installation that is usually only accessible through specialized structures such as sleep laboratories. It could also be limited by motion sickness. In the present study, 5 patients were not able to complete the task due to motion sickness.
In this selected sample of PwMS with CI, the evaluation by the Urban DailyCog detected 80% of CI patients, but the DSDT identified only 52% of CI patients. The DSDT appears to be less sensitive than the Urban DailyCog. During this task, PwMS who have a driver's license may implement more automated strategies because the instructions are quite simple and resemble what they do every day.
The Urban DailyCog detected more PwMS with CI than any of the single NP test, including the SDMT, although the statistical significance of these differences was not measured due to the small sample size. In the present sample of PwMS with CI, 63.3% were impaired in the SDMT. Although IPS impairment is the most frequent cognitive disturbance in MS, CI involves different domains. The SDMT has been shown to be the most sensitive test , but it does not capture all PwMS with CI and the use of only one screening test is certainly not sufficient. As IPS and EM are the two most frequently impaired domains in MS, we also analyzed CI with the first two tests of the BICAMS, the SDMT and the CVLT-LT. This combination detected only 70% of PwMS with CI, while the VRE test identified 80% of them.
The Urban DailyCog includes tasks that assess IPS, attention, inhibition and flexibility, and more specifically divided attention, resource allocation, coordination and supervision. PwMS with CI based on Urban DailyCog were impaired in IPS, EF, EM, WM and attention. VRE tests could involve several different cognitive abilities, may allow for the detection of impairment in a broad area of cognitive domains, and could therefore serve to assess the main cognitive functions affected in multiple sclerosis, such as WM, EF, IPS, and attention, while being sufficiently sensitive to detect any isolated and specific deficits of these functions. We did not observe any significant correlation between the VRE tasks and the NP tests' z scores, but this could be due to the limited power of the study that included only 30 patients. A possible interpretation is related to the difference between the two types of tests (classical vs. VRE). Unlike conventional tests that minimize interactions and multimodality, the Urban DailyCog is multimodal and delivered many informations, as in daily life. Moreover, the space is a three-dimensional wide space that includes the whole body. This makes it possible to create a set of many interactions between cognitive processes, as was our objective. This could also mean that these VRE tasks do not estimate the cognitive processes in the same way as the conventional cognitive tests. VRE tests require several different cognitive abilities and may detect impairment resulting from the interactions between cognitive domains..
One goal of an ecological test could be to identify PwMS in whom CI has a strong interaction with daily life. Ecological validity is defined as “the functional and predictive relationship between the patient's performance on a set of NP tests and the patient's behavior in a variety of real world settings” . The ecological validity of cognitive assessment through VRE has been proven , , , and  and can be conceptualized in the following two approaches: verisimilitude, which refers to a task close to daily activities, and veridicality, which refers to links between neuropsychological tests and measures of everyday functioning. Tasks with verisimilitude compared to traditional tasks may be most effective at predicting everyday functioning . What the patient can do in everyday life could be underestimated by classical testing  because the subject could use various compensatory skills, which are different from those used during classical testing that can be reproduced in ecological tasks. Associating ecological tests and conventional tests could then be a good option for the assessment of cognitive functioning dependent on the patient's own resources but also on environmental resources, specifically in predicting potential recovery, setting up axes of rehabilitation, monitoring cognitive functioning and predicting the ability to maintain employment.
This study has some limitations. It is a preliminary study with a limited sample. In the future, studies including PwMS with and without CI should be included to assess the specificity of the Urban DailyCog. Another limitation is that the task does not assess EM. Finally, we did not directly assess the veridicality of the test by assessing measures of daily functioning.
In conclusion, VRE evaluation using a self-performed task in 20 min could be a useful way to briefly evaluate CI in MS patients. The Urban DailyCog task is reliable and easy to administer and can be delivered in a standardized and reproducible manner, regardless of the examiner. VRE assessments are promising in detecting CI while providing friendly assessments for patients and simulating daily activities. Observing the attitudes and strategies of patients in an everyday environment can also guide rehabilitation tracks and identify everyday discomfort that patients encounter in their environment. Performances of this assessment with verisimilitude could be used to monitor patients according to increased functional skills and could be a good way to monitor rehabilitation. Further studies are necessary to assess its specificity and reliability and to provide a formal validation before implementation in clinical practice. Further studies are also needed to establish the accurate ecological validity of VRE by directly assessing the consequences of CI in the daily lives of PwMS.
Conflicts of interest
On behalf of all authors, the corresponding author states that there are no conflicts of interest in relation with the study.
This study was performed as part of the REACTIV program and was funded by Foundation ARSEP (Association pour la Recherche contre la Sclérose en Plaques), TRAIL Cluster of Excellence (Université de Bordeaux, France) (ANR-10-LABX-57) and a grant from Merck-Serono. These funding partners did not participate in any aspect of the design or in the conduct of the study, including data collection, management, analysis or interpretation. They were not involved in the preparation, review or approval of this manuscript.
This work was made possible by the contribution of the AQUISEP network, a clinical network of Aquitaine devoted to MS care. The following neurologists contributed to the study: J-C Ouallet, A. Laporte, P. Louiset, I. Patry, A. Bredin, P. Gaida J-M. Faucheux and F. Radji. We thank Pierre Joly, statistician at ISPED U897, for his advice.
-  B. Brochet. Prevalence, profile and functional impact of cognitive impairment in multiple sclerosis. M.P. Amato (Ed.) Cognitive Impairment in Multiple Sclerosis (Elsevier, Milano, 2011) 1-8
-  D.W. Langdon, M.P. Amato, J. Boringa, et al. Recommendations for a Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS). Mult. Scler.. 2012;18:891-898 Crossref
-  A. Smith. Symbol Digit Modalities Test (SDMT) Manual (Revised). (Western Psychological Services, Los Angeles, 1982)
-  D.C. Delis, J.H. Kramer, E. Kaplan, et al. California Verbal Learning Test. 2nd edn (Psychological Corporation, San Antonio, 2000)
-  R. Benedict. Brief Visuospatial Memory Test, Revised. (Psychological Assessment Resources, Odessa, FL, 1997)
-  M.S. Deloire, M.C. Bonnet, E. Salort, et al. How to detect cognitive dysfunction at early stages of multiple sclerosis?. Mult. Scler.. 2006;12:445-452 Crossref
-  B. Brochet, M.S. Deloire, M. Bonnet, et al. Should SDMT substitute for PASAT in MSFC? A 5-year longitudinal study. Mult. Scler.. 2008;14:1242-1249 Crossref
-  R. Benedict, M.P. Amato, J. Boringa, et al. Brief International Cognitive Assessment for MS (BICAMS): international standards for validation. BMC Neurol.. 2012;12:55
-  R.H. Benedict, F. Munschauer, R. Linn, et al. Screening for multiple sclerosis cognitive impairment using a self-administered 15-item questionnaire. Mult. Scler.. 2003;9:95-101 Crossref
-  B. Arnaldi, P. Fuchs, J. Tisseau. Traité de la réalité virtuelle. in: 4 (Les Presses Mines, Paris, 2003)
-  A.A. Rizzo, T. Bowerly, J.G. Buckwalter, et al. A virtual reality scenario for all seasons: the virtual classroom. CNS Spectr.. 2006;11:35-44 Crossref
-  E. Pedroli, P. Cipresso, S. Serino, et al. Virtual Multiple Errands Test: reliability, usability and possible applications. Stud. Health Technol. Inform.. 2013;191:38-42
-  L. Zhang, B.C. Abreu, G.S. Seale, et al. A virtual reality environment for evaluation of a daily living skill in brain injury rehabilitation: reliability and validity. Arch. Phys. Med. Rehabil.. 2003;84:1118-1124 Crossref
-  P. Grewe, D. Lahr, A. Kohsik, et al. Real-life memory and spatial navigation in patients with focal epilepsy: ecological validity of a virtual reality supermarket task. Epilepsy Behav.. 2014;31:57-66 Crossref
-  P. Werner, S. Rabinowitz, E. Klinger, et al. Use of the virtual action planning supermarket for the diagnosis of mild cognitive impairment: a preliminary study. Dement. Geriatr. Cogn. Disord.. 2009;27:301-309 Crossref
-  B. Renison, J. Ponsford, R. Testa, et al. The ecological and construct validity of a newly developed measure of executive function: the Virtual Library Task. J. Int. Neuropsychol. Soc.. 2012;18:440-450 Crossref
-  T.D. Parsons, C.G. Courtney, M.E. Dawson. Virtual reality Stroop task for assessment of supervisory attentional processing. J. Clin. Exp. Neuropsychol.. 2013;35:812-826 Crossref
-  T.D. Parsons, C.G. Courtney. An initial validation of the Virtual Reality Paced Auditory Serial Addition Test in a college sample. J. Neurosci. Methods. 2014;30(222):15-23 Crossref
-  C.H. Polman, S.C. Reingold, G. Edan, et al. Diagnostic criteria for multiple sclerosis: 2005 revisions to the ‘McDonald Criteria’. Ann. Neurol.. 2005;58:840-846 Crossref
-  M.F. Folstein, S.E. Folstein, P.R. McHugh. “Mini-mental state”. A practical method for grading the cognitive state of patients for the clinician. J. Psychiatr. Res.. 1975;12:189-198 Crossref
-  A. Rey. Manuel du Test de copie et de reproduction de mémoire de figures géométriques complexes. (Éditions Centre de psychologie appliquée, Paris, 1959)
-  A.T. Beck, G. Brown, R.A. Steer. Beck Depression Inventory II Manual. (The Psychological Corporation, San Antonio, TX, 1996)
-  J.F. Kurtzke. Rating neurologic impairment in multiple sclerosis: an Expanded Disability Status Scale (EDSS). Neurology. 1983;33:1444-1452
-  C.D. Spielberger, R.L. Gorssuch, P.R. Lushene, et al. Manual for the State-Trait Anxiety Inventory. (Consulting Psychologists Press, Palo Alto, 1983)
-  J.D. Fisk, P.G. Ritvo, L. Ross, et al. Measuring the functional impact of fatigue: initial validation of the fatigue impact scale. Clin. Infect. Dis.. 1994;18(S1):S79-S83 Crossref
-  P. Zimmermann, B. Fimm. TAP 2.1 Tests d'evaluation de l'attention. (Psytest, Wurzelen, 2009)
-  O. Godefroy, GREFEX. Fonctions Exécutives et Pathologies Neurologiques et Psychiatriques. (Evaluation en pratique clinique, Solal, Marseille, 2008)
-  M.D. Lezak, D.B. Howieson, D.W. Loring. Neuropsychological Assessment. fourth edition (Oxford university press, Oxford, 2004)
-  J. Poitrenaud, B. Deweer, M. Kalafat, M. Van der Linden. CVLT (California Verbal Learning Test): Test d’apprentissage et de mémoire verbale. (ECPA, Paris, 2007)
-  G. Deloche, D. Hannequin. Test de dénomination oral d'images, DO 80. (Edition du Centre de psychologie appliquée, Paris, 1997)
-  R.J. Sbordone, A.D. Purisch. Hazards of blind analysis of neuropsychological test data in assessing cognitive disability: the role of confounding factors. NeuroRehabilitation. 1996;7:15-26 Crossref
-  F. La Paglia, C. La Cascia, R. Rizzo, et al. Cognitive assessment of OCD patients: NeuroVR vs neuropsychological test. Stud. Health Technol. Inform.. 2014;199:40-44
-  D.M. Spooner, N.A. Pachana. Ecological validity in neuropsychological assessment: a case for greater consideration in research with neurologically intact populations. Arch. Clin. Neuropsychol.. 2006;21:327-337 Crossref
-  C.J. Long, M.Y. Kibby. Ecological validity of neuropsychological tests: a look at neuropsychology's past and the impact that ecological issues may have on its future. Adv. Med. Psychother.. 1995;8:59-78
a Université de Bordeaux, France
b INSERM U.862, Neurocentre Magendie, Bordeaux, France
c CHU de Bordeaux, INSERM-CHU CIC-P 0005, Service de Neurologie, F-33076 Bordeaux, France
d CNRS UMR 5227, GENPPHAASS laboratory, Bordeaux, France
⁎ Corresponding author at: Service de Neurologie, CHU Bordeaux, F-33076 Bordeaux Cedex, France.
© 2015 Elsevier B.V., All rights reserved.