Telehealth-based assessment of cognition, social cognition, mood, and functional independence in older adults
Michelle Kelly
A * , Simon Mierendorff A , Kylie Wales B , Johanna Voeste A , Joanne Allen A and Skye McDonald C
A
B
C
Abstract
Mild cognitive impairment affects over 15% of adults aged 50+ years and is a primary risk indicator for dementia. Although access to assessment is crucial, many older adults face barriers to in-person evaluation.
This study used a randomised cross-over design to assess the practicality, acceptability, and adaptation of a telehealth-based screening battery tailored for older adults. Forty-three volunteers aged 50+ years (m = 70.3, s.d. = 10.8) completed in-person or videoconference assessments, including the Addenbrooke’s Cognitive Examination-III (ACE-III), Brief Assessment of Social Skills (BASS), Hospital Anxiety and Depression Scales (HADS), Modified Barthel Index (MBI), and Assessment of Living Skills And Resources-2 (ALSAR-R2). The alternate format was administered after 3 weeks. Practicality was assessed with reference to task modifications, completion, and administration time. Acceptability was evaluated via questionnaire. Reliability was assessed using intraclass correlation coefficients (ICCs).
Minimal modifications were needed for the videoconference format, and it was highly acceptable to respondents. Reliability across formats was excellent for BASS Empathy, HADS Depression, MBI and ALSAR-R2 (ICC = 1.00–0.92) and good for ACE-III, HADS Anxiety, and BASS Face Emotion Perception, Face Identification, and Social Disinhibition scales (ICC = 0.77–0.89).
Findings support the feasibility of telehealth-based administration of the screening battery; however, biases in emotion perception performance between modalities require further research.
Keywords: accessibility, cognitive assessment, dementia, functional assessment, screening battery, screening tool, social neuroscience, telehealth.
Introduction
Mild cognitive impairment (MCI) is a condition typified by reduced cognitive abilities, on standardised measures of cognition, that is greater than expected for age. MCI is estimated to affect over 15% of community-dwelling adults over the age of 50 years and 21% of community-dwelling people over the age of 79 years worldwide (Bai et al. 2022) and is considered a primary indicator of a person’s risk for later dementia (Petersen et al. 2018). Early identification of symptoms and risk factors for MCI and dementia are considered crucial for monitoring disease progression and slowing the onset of symptoms (Livingston et al. 2017; Woodward et al. 2022). However, personal health, mobility, and transport issues exacerbate service-level barriers to assessment among older adults (Van Gaans and Dent 2018; Lavingia et al. 2020; Krasniuk and Crizzle 2023). In countries with large, dispersed populations, such as Australia, sparsity and distance from services make reliance on in-person services for older adults untenable (Blackberry and Morris 2023), with similar barriers to health care identified in other regions of the world, including Canada (Krasniuk and Crizzle 2023), the United States (Wolfe et al. 2020), and South Korea (Jang et al. 2020).
Telehealth has long been recognised for its capacity to enable assessment and early intervention in key health areas (Harrell et al. 2014; Kann et al. 2023). These formats provide an opportunity to increase assessment rates, with interest in using telehealth-based cognitive assessment (Carotenuto et al. 2021; Zeghari et al. 2022; Turner and Brearly 2023) compounded by the COVID-19 pandemic (Marra et al. 2020; Parks et al. 2021). However, there are recognised risks to the psychometric properties of assessment tools when administered via telehealth (Luxton et al. 2014), highlighting a need for careful selection and evaluation of tests suitable for telehealth administration (Turner and Brearly 2023). Researchers have responded to this clinical demand, with several studies demonstrating high levels of acceptability and reliability of neuropsychological tests across telehealth and in-person delivery modes (Turner and Brearly 2023). However, some studies have observed differences associated with assessment modes (Hernandez et al. 2022). Due to the potential for such effects to bias conclusions made concerning existing clinical criteria, careful assessment of the feasibility of telehealth delivery of established measures is needed to ensure appropriate screening and equality in service.
Current study
The current study adopts questionnaire and randomised cross-over methods to evaluate the feasibility of a videoconference-based screening battery compiled to identify areas of concern among an older adult population. Test practicality and score reliability are compared between videoconference and in-person administrations of the screening battery in the same sample. Screening tools were selected based on established sensitivity to clinical outcomes, appropriateness for the target population, accessibility, and ease of use to assess performance across four key domains associated with MCI and dementia, including (1) cognition (Campbell et al. 2013; Mueller et al. 2018), (2) social cognition (Kumfor et al. 2013, 2015; Kelly and McDonald 2020), (3) mood (Fiske et al. 2009; Leyhe et al. 2017), and (4) activities of daily living (ADL) (Inouye et al. 1998; Tabert et al. 2002).
The current study assesses three domains of feasibility (Bowen et al. 2009) of the videoconference administration of this screening battery among a sample of older adults with no known diagnosis of MCI or dementia. Outcome variables examine the evidence for practicality, acceptability, and adaptation of videoconference administration. Practicality was investigated by determining whether the screening battery could be delivered within the same testing restrictions for both videoconference and in-person modalities, with reference to task completion and administration times. Acceptability was examined using responses to a participant survey regarding the videoconference format, including frequency and impact of technical issues and whether these varied with participant demographic or health-related factors. Adaptation was examined by determining whether the screening tools could be adapted for videoconference administration with limited disruption to reliability compared to in-person format. In line with existing evaluations of in-person vs telehealth administration of neuropsychological assessment batteries (Munro Cullum et al. 2014; DeYoung and Shenal 2019; Mahon et al. 2022), it was hypothesised that (1) the tools chosen would allow for practical adaptation for use via videoconference, (2) there would be a high rate of acceptability of the videoconference modality that would not vary with respondent demographics or indicators of adverse health, and (3) assessment scores would display high reliability across videoconference and in-person assessment modes.
Materials and methods
Participants
Participants were recruited through a local medical research volunteer registry (n = 31), local aged facilities (n = 12), and word of mouth (n = 9) from the Newcastle region, New South Wales, Australia. Eligible participants were over 50 years of age, had conversational English language skills, were medically fit to complete the study, and had no prior diagnosis of dementia or current neurological or mental health condition. Of the 52 participants recruited, six were excluded due to suspected cognitive impairment as indicated by the ACE-III at in-person assessments (total scores <82: Mioshi et al. 2006), and one was excluded due to evidence they were receiving assistance during the videoconference format session. Two further participants were excluded, as they did not complete their second (videoconference) assessment session. None of the measures administered for the purposes of the current research form part of the research volunteer registry intake protocols, and as such, participants are unlikely to have had prior exposure to these measures.
Measures
Before the first assessment session, a brief clinical interview was conducted to obtain relevant medical and demographic information.
The Addenbrooke’s Cognitive Examination-III (ACE-III: Hsieh et al. 2013) is a brief assessment of cognition (15–20 min), commonly used to screen for dementia. Five domains of cognition are assessed (i.e. memory, attention, verbal fluency, visuospatial ability, and language), and a total score is calculated (range 0–100). The ACE-III total score has excellent inter-rater reliability (ICC ≥ 0.90) and test–retest reliability (ICC ≥ 0.90: Bentvelzen et al. 2017).
The Brief Assessment of Social Skills (BASS: Kelly and McDonald 2020) is a screen for social cognition impairments in people with dementia. It is suitable for use with older people with or without MCI, can discriminate between people with dementia and healthy adults, and is sensitive to advancing dementia (Kelly and McDonald 2020). The BASS comprises seven tasks assessing six key domains of social cognition: Face Emotion Perception, Empathy and Theory of Mind, Social Disinhibition, Social Reasoning, Memory for New Faces, and Facial Identification (Adolphs 2003). Higher task scores indicate higher social cognitive performance, except for the Social Disinhibition tasks in which higher scores indicate more disinhibited performance.
The Hospital Anxiety and Depression Scale (HADS: Zigmond and Snaith 1983) is a brief assessment of mood, consisting of 14 questions suitable for all ages, with subscale scores indicating greater symptoms of depression (range 0–21) or anxiety (range 0–21).
The Modified Barthel Index (MBI: Shah et al. 1989) is a brief assessment of ADL. The MBI consists of 11 domains of functioning that are discussed with the respondent and rated by the clinician. A total score is calculated, with higher scores indicating higher functional ability (range 0–100). It is suitable for use with all presentations of dementia and is a significant predictor of change in functionality, future admissions, and required services (Bentvelzen et al. 2017).
The Assessment of Living Skills and Resources-Revised, 2nd Edition (ALSAR-R2) is a brief assessment of instrumental activities of daily living (IADL) that has been validated for use with older adults with MCI (Clemson et al. 2009). It assesses the accomplishment of 11 IADLs, including using the telephone, participating in leisure activities, and completing household tasks. Higher scores indicate greater difficulty accomplishing the task, lower availability of resources to complete the task, or both (range 0–44).
This 10-item questionnaire on the acceptability of assessment via telehealth was adapted from similar studies examining participant attitudes to telehealth (Grubaugh et al. 2008). One item assessed the presence of technical difficulties (open response), and the remainder assessed acceptability of the telehealth modality on a scale of 1–10, with higher scores indicating greater acceptability.
Procedure
Study protocols were approved by The University of Newcastle and Hunter New England Local Health District ethics committees (HNEHREC – 14/05/21/4.02; HREC – H-2015-0255). Participants were randomly allocated to either the videoconference-first or in-person-first assessment format. Two postgraduate psychology students were trained to deliver the assessments consistently and competently (observed to competence by a clinical psychologist), utilising the same equipment. Participants were re-assessed in the alternate format after a minimum 3-week interval. The order of tool administration was kept the same across assessment sessions. Sessions occurred at the same time of day to control for the effects of fatigue. In-person testing was conducted at participant’s homes, a local aged care facility, or the University. Videoconference testing was performed remotely using Skype™, with the researcher and participant in quiet locations free from distraction. The Telehealth Acceptability Questionnaire was administered immediately following the videoconference assessment. In cases where one or more screening tools indicated cognitive impairment or adverse mental health, this was shared with the participant, and consent was obtained to provide a referral to their general practitioner. All completed test forms were double-scored (SMi and JV).
Analysis
Descriptive statistics were examined for all outcome variables. Mann–Whitney U tests were used to examine differences in Telehealth Acceptability Questionnaire item ratings between categories of participants (e.g. age <65 vs 65+ years, sensory impairment vs not, family history of dementia vs not, prior history of mental health concerns vs not, medication vs not, drinker vs not). Where two or more groups were present (e.g. medications, alcohol), data were recoded so that none = 0, and 1+ = 1. Total scores were calculated where all items were answered/observed, and listwise deletion was used to address missing data. Normal Q–Q plots were used to assess the univariate normality of difference scores between modalities and the presence of outliers. Paired samples t-tests were conducted to assess whether the mean difference between in-person and videoconference assessment scores significantly differed from zero. Agreement between assessment modalities for each assessment tool was assessed using intraclass correlation coefficients (ICCs) and associated 95% confidence intervals (CIs) using a mean-rating, absolute agreement, two-way mixed effects model (Bartko 1966). Values less than 0.50 indicated poor reliability, 0.50–0.74 moderate reliability, 0.75–0.90 good reliability, and values greater than 0.90 excellent reliability (Koo and Li 2016). As ICC cannot be interpreted where scores lack variability between subjects (Lee et al. 1989; Bland and Altman 1990), score ranges and measures of dispersion for each measure were inspected. Where measures displayed low score variability, ICC was not calculated, and the proportion of cases displaying agreement in values across modalities was described.
Results
Sample
Of the 43 participants who completed assessments at both time points, n = 24 completed the face-to-face, in-person-first condition, and n = 19 completed telehealth-first. The median lag between assessments was 4.0 weeks. For most participants (n = 36), the time between assessments ranged from 3.0 to 6.0 weeks (m = 4.35, s.d. = 0.91), with an additional two completing follow-up at 6.1 and 7.6 weeks. Follow-up of five participants was delayed due to their inability to attend the University campus during the initial COVID-19 pandemic response for in-person testing (range = 43.0–62.7 weeks, m = 52.81, s.d. = 8.26). Sensitivity analyses, excluding the n = 7 cases who did not complete follow-up within 6 weeks, were conducted to assess the impact of these cases on reduced estimates of reliability analyses based on the whole sample (Supplementary File S1).
Participants were aged between 51 and 93 years (m = 70.33, s.d. = 10.81) and reported a range of exposure to formal education (range = 9–20 years, m = 14.26, s.d. = 2.99), and 74.4% were retired. A summary of participant health and demographic characteristics is presented in Table 1. Around 60.5% were on one or more prescribed medications, and around half did not drink alcohol. Participants were located 3–56 km from the University (m = 14.73, s.d. = 14.74), with 88.4% residing in areas classified as a major city (ABS 2018).
| Characteristics | n | % | |
|---|---|---|---|
| Age | |||
| <65 years | 13 | 30.2 | |
| 65+ years | 30 | 69.8 | |
| Hearing difficulties | 10 | 23.3 | |
| Vision difficulties (corrected) | 31 | 72.1 | |
| Family history of dementia | 13 | 30.2 | |
| Previous history of mental health issues | 9 | 20.9 | |
| Medications | |||
| 0 | 17 | 39.5 | |
| 1–3 | 18 | 42.0 | |
| 4–6 | 5 | 11.7 | |
| 7+ | 3 | 6.9 | |
| Alcohol use (standard drinks per week) | |||
| 0 | 22 | 51.2 | |
| 1–7 | 16 | 37.2 | |
| 8–14 | 4 | 9.3 | |
| 15+ | 1 | 2.3 | |
| Regular tobacco/nicotine use | 1 | 2.3 | |
| Residence A | |||
| Major city | 38 | 88.4 | |
| Inner regional area | 4 | 9.3 | |
| missing | 1 | 2.3 | |
Practicality
Minor modifications were needed to adapt the assessment battery for videoconference administration. For the ACE-III, test items were displayed on-screen, as opposed to the conventional use of paper to present materials, with participants holding completed written and visuospatial task responses to the camera for the examiner to review for completion and take a screen shot for scoring. To avoid altering the cognitive skills involved in the task (Luxton et al. 2014) and to stay consistent with guidelines for remote administration of the ACE-III (FRONTIER 2020), where participants would typically be asked to view and point to the correct image (i.e. language task), stimuli were viewed online, with participants’ responses indicated using their computer mouse. No modifications were required for BASS administration, as stimuli were presented on-screen in both formats and verbal instructions and answers were provided. The paper-and-pencil HADS task was presented on-screen so participants could select their responses using their mouse. No adaptations were required for the MBI and ALSAR-R2.
One participant could not complete the visuospatial aspects of ACE-III in either format due to motor issues, one did not complete the BASS Social Disinhibition Part II assessment in either format, and one did not complete this task in the in-person format. All other participants completed tasks in each format. The average test administration time was ~90 min in each format.
Acceptability
Table 2 presents a summary of participant responses to the Telehealth Acceptability Questionnaire. One participant reported experiencing minor audio issues when completing the videoconference format; however, these were resolved during the initial clinical interview phase of the assessment. No other technical difficulties were reported. Except for Q5 regarding confidence in using telehealth for the assessment (median = 9), half or more respondents gave Telehealth Acceptability Questionnaire indicators the highest available rating (median = 10), and no participant rated any indicator below five.
| Questionnaire item | n | Median | Mean | s.d. | Range | |
|---|---|---|---|---|---|---|
| Q2. How would you rate the audio quality of this assessment session? | 43 | 10 | 9.49 | 0.91 | 6–10 | |
| Q3. How would you rate the video quality of this assessment session? | 43 | 10 | 9.42 | 0.93 | 7–10 | |
| Q4. Being available online was a good way to complete this assessment | 42 | 10 | 9.12 | 1.37 | 5–10 | |
| Q5. I felt confident using telehealth for this assessment | 43 | 9 | 8.88 | 1.22 | 6–10 | |
| Q6. I could easily talk and express myself effectively | 43 | 10 | 9.56 | 0.73 | 8–10 | |
| Q7. Online is an acceptable way to receive this assessment | 43 | 10 | 9.42 | 1.11 | 5–10 | |
| Q8. The assessment provided online was as good as face-to-face | 27 | 10 | 9.07 | 1.18 | 5–10 | |
| Q9. I would recommend this type of assessment to others | 27 | 10 | 9.52 | 1.09 | 5–10 | |
| Q10. Overall, I was satisfied with this online experience | 43 | 10 | 9.60 | 0.73 | 7–10 |
Note: Item 1 is not listed above as it was an open response item. Items Q2–Q3 were rated on a scale of 0 (poor) to 10 (excellent); items Q4-Q10 were rated on a scale of 0 (strongly disagree) to 10 (strongly agree); items Q8 and Q9 were added to the questionnaire after data collection began and so were not collected for all participants.
Examining the association of item ratings with participant characteristics, participants who drank alcohol more highly endorsed the telehealth assessment as being as good as the in-person assessment (U = 138.5, P = 0.011) than those who did not. The association of item ratings with smoking status was not assessed due to the small number of smokers (n = 1). No other associations between item ratings and participant characteristics listed in Table 1 were observed. The order of assessment format displayed no association with the Telehealth Acceptability Questionnaire item ratings.
Adaptation
Descriptive statistics by task and assessment format are presented in Table 3. Score ranges were highly comparable across assessment modes. BASS criterion reference tests (Social Reasoning and Social Disinhibition Part II (errors)) displayed a restricted range of scores, reflecting the anticipated performance on these measures among a community volunteer sample. Social Reasoning scores in both modalities were integers 4–5, with 90.7% of participants displaying the same score at in-person and videoconference assessments. Social Disinhibition Part II (errors) scores represented the difference in the number of errors made between inhibition and non-inhibition conditions of a social Stroop task, with 75.6% of respondents having the same score ±1 at in-person and videoconference assessments. Normal Q–Q plots of differences in scores between modalities were normally distributed, with the exception of BASS Social Reasoning (discussed above), HADS Anxiety and Depression scores, and MBI scores. A log transformation was used to address asymmetry in the distribution of HADS and MBI scores and assess the impact of non-normality on parametric test statistics. BASS Empathy displayed one outlier, and analyses including and excluding this data were point conducted to assess its impact on test statistics.
| In-person | Videoconference | |||||||
|---|---|---|---|---|---|---|---|---|
| Potential score range | n | Mean (s.d.) | Range | n | Mean (s.d.) | Range | ||
| Cognitive screen (ACE-III) | 0–100 | 42 | 94.88 (3.58) | 84–100 | 42 | 94.57 (3.07) | 89–100 | |
| Social functioning (BASS) | ||||||||
| Face Emotion Perception | 0–54 | 34 | 39.71 (4.93) | 27–48 | 34 | 41.47 (5.54) | 28–50 | |
| Face Identification | 0–48 | 43 | 32.58 (4.94) | 18–42 | 43 | 32.49 (5.03) | 10–40 | |
| Empathy/theory of mind | 0–133 | 41 | 75.24 (14.82) | 41–110 | 42 | 75.12 (13.63) | 43–103 | |
| Social Disinhibition (part I) | 0–5 | 43 | 1.56 (1.79) | 0–5 | 43 | 1.70 (1.83) | 0–5 | |
| Social Disinhibition (part II errors) | . | 42 | 0.27 (0.92) | −2–3 | 42 | 0.55 (1.37) | −1 to 4 | |
| Social Disinhibition (part II time (s)) | . | 41 | 9.92 (7.93) | −13.5–31.0 | 42 | 9.42 (5.19) | −0.62 to 25.7 | |
| Social Reasoning | 0–5 | 43 | 4.95 (0.21) | 4–5 | 43 | 4.91 (0.29) | 4–5 | |
| Memory for New Faces | 0–24 | 43 | 20.26 (2.79) | 12–24 | 43 | 20.95 (2.48) | 14–24 | |
| Mood (HADS) | ||||||||
| Anxiety | 0–21 | 43 | 4.95 (4.05) | 0–20 | 43 | 4.33 (3.85) | 0–21 | |
| Depression | 0–21 | 43 | 3.02 (2.74) | 0–10 | 43 | 2.77 (2.44) | 0–12 | |
| Daily activities (MBI, ALSAR-R2) | ||||||||
| Activities of daily living | 0–100 | 43 | 96.14 (11.92) | 30–100 | 43 | 95.88 (12.31) | 29–100 | |
| Instrumental activities of daily living | 0–44 | 43 | 2.86 (5.40) | 0–19 | 43 | 2.91 (5.27) | 0–18 | |
Notes: ACE-III, Addenbrooke’s Cognitive Examination III; BASS, Brief Assessment of Social Skills; HADS, Hospital Anxiety and Depression Scale; MBI, Modified Barthel Index; ALSAR-R2, The Assessment of Living Skills and Resources-Revised, 2nd Edition.
When assessing score reliability across assessment modes (Table 4), paired samples t-tests indicated that mean differences in scores across formats did not significantly differ from zero, except for BASS Face Emotion Perception. Analysis of log transformations of HADS Depression and Anxiety and MBI data did not impact conclusions of statistical tests of mean differences or reliability of scores across assessment modes, and results based on untransformed data are reported. Exclusion of an outlier in the comparison of BASS Empathy scores also did not impact conclusions, and results including this data point are reported. On average, Face Emotion Perception scores (range 0–54) were 1.9 (s.d. = 4.1) points higher in the videoconference administration mode. ICCs indicated excellent reliability for the BASS Empathy, HADS Depression, MBI, and ALSAR-R2 scores and good reliability for the ACE-III, BASS Face Identification, Face Emotion Perception, Social Disinhibition Part I, and HADS Anxiety scores. BASS Social Disinhibition Part II (time in seconds) and Memory for New Faces task scores displayed moderate reliability across assessment modes. Due to the restricted range and low variability of scores displayed in the current sample, ICC was not calculated for Social Reasoning or Social Disinhibition Part II (errors) tasks. Supplementary analyses excluding cases who completed re-assessment after the planned 6-week follow-up period indicate that their inclusion did not reduce score reliability.
| n | Mean difference (s.d.) | Paired samples t-test | Intraclass correlation coefficient (95% CI) | ||
|---|---|---|---|---|---|
| Cognitive screen (ACE-III) | 42 | −0.31 (2.17) | t(41) = −0.93, P = 0.361 | 0.88 (0.78, 0.94) | |
| Social functioning (BASS) | |||||
| Face Emotion Perception | 30 | 1.87 (4.07) | t(29) = 2.51, P = 0.018 | 0.81 (0.58, 0.91) | |
| Face Identification | 43 | −0.09 (3.26) | t(42) = −0.19, P = 0.852 | 0.88 (0.78, 0.94) | |
| Empathy/theory of mind | 40 | −0.40 (7.79) | t(39) = −0.33, P = 0.747 | 0.92 (0.85, 0.96) | |
| Social Disinhibition (Part I) | 43 | 0.14 (1.30) | t(42) = 0.70, P = 0.486 | 0.85 (0.73, 0.92) | |
| Social Disinhibition (Part II) | 41 | 0.29 (1.52) | t(40) = 1.23, P = 0.225 | . | |
| Social Disinhibition (part II time (s)) | 41 | −0.48 (6.54) | t(40) = −0.48, P = 0.636 | 0.77 (0.57, 0.87) | |
| Social reasoning | 43 | −0.05 (0.31) | t(42) = −1.00, P = 0.323 | . | |
| Memory for New Faces | 43 | 0.70 (2.47) | t(42) = 1.85, P = 0.071 | 0.71 (0.46, 0.84) | |
| Mood (HADS) | |||||
| Anxiety | 43 | −0.63 (2.45) | t(42) = −1.68, P = 0.100 | 0.89 (0.80, 0.94) | |
| Depression | 43 | −0.26 (1.36) | t(42) = −1.23, P = 0.226 | 0.93 (0.86, 0.96) | |
| Daily activities (MBI, ALSAR-R2) | |||||
| Activities of daily living | 43 | −0.26 (1.36) | t(42) = −1.23, P = 0.226 | 1.00 (0.99, 1.00) | |
| Instrumental activities of daily living | 43 | 0.05 (0.75) | t(42) = 0.40, P = 0.688 | 1.00 (0.99, 1.00) | |
Notes: ACE-III, Addenbrooke’s Cognitive Examination III; BASS, Brief Assessment of Social Skills; HADS, Hospital Anxiety and Depression Scale; MBI, Modified Barthel Index; ALSAR-R2, The Assessment of Living Skills and Resources-Revised, 2nd Edition; Intraclass correlation coefficients for BASS Social Reasoning and Social Disinhibition Part II (errors) were not calculated; one participant was not included in analysis of ACE-III total scores due to inability to complete the visuospatial task – the remaining ACE-III domain scores for this participant were highly consistent across assessment modes (difference scores of 1 for attention and 0 for memory, verbal fluency, and language).
Discussion
Results support the practicality and acceptability of a telehealth-based multi-domain assessment for MCI among healthy older adult volunteers. Indicators provide general support for adaptation to the telehealth format. Assessments could be administered in-person and in videoconference formats within a similar timeframe. Where tasks were completed using the in-person format, they were also completed in the telehealth format. Participants reported high audio and video quality, which are core factors in user acceptance (Luxton et al. 2014). Although one participant experienced technical difficulties, these were resolved, suggesting the practicality of the telehealth format may be supported by allowing time to address technical issues.
Consistent with previous research (Tousignant et al. 2011), the current sample highly endorsed statements relating to the acceptability of the videoconference assessment format. Demographic characteristics, hearing difficulties, corrected vision impairment, family history of dementia, history of mental illness, or tobacco or alcohol use did not negatively influence acceptability. In contrast to research conducted with primary care patients in the mid-2000s (Grubaugh et al. 2008), age did not display a significant relationship with telehealth acceptability, which may in part reflect the effects of current technology, as well as exposure and confidence of older healthcare consumers engaging with telehealth modalities in this later period.
Examining the evidence for the telehealth adaptation of the assessment battery, results support the reliability of the cognitive screen (ACE-III), ADL/IADL (MBI and ALSAR-R2), and mood (HADS) elements of the assessment battery, as well as several aspects of the BASS social cognition assessment. Evaluating evidence for systematic bias between assessment modes, only telehealth administration of the BASS Face Emotion Perception task overestimated scores compared to in-person administration by an average of 1.9 points (±4.1), indicating format should be considered when interpreting scores for this task. Although the cause of this bias was not clear, it is possible that the image quality was better on screen, enabling better evaluation. Overall, evidence for the test–retest reliability of these measures match and exceed similar studies of neurocognitive (Prestia et al. 2006), mood (Kobak 2004), and functional (Hwang et al. 2017) assessments among older adult samples. Guidelines for health assessment standards indicate minimum reliability coefficients of 0.70 for group comparisons and 0.90 for individual comparisons (Aaronson et al. 2002), indicating that BASS Empathy, HADS Depression, MBI and ALSAR-R2 may be suitable where multiple modes of assessment are considered in monitoring case progression over time.
Coefficients for the remaining BASS domains indicated greater variability between assessment modes. The lowest reliability indices were observed for Social Disinhibition Part II (time) and Memory for New Faces tasks. Social Disinhibition Part II is a social Stroop task in which respondents are tasked with reading and then inhibiting salient social information (their own name) within stimuli lists, with scores representing the difference in the errors/time to read each column between reading and inhibition conditions. Memory for New Faces tasks respondents with recognition and recall of associated socially relevant information for a series of faces presented at the start of the interview. Although other tools selected for the assessment battery are relatively established, the BASS is a newer assessment, and temporal reliability data are currently limited for this measure. Estimates observed in the current study are similar to test–retest indices for other clinical assessments of social cognition when administered in the same assessment mode (McDonald et al. 2006), suggesting that observations of moderate and poor reliability may be attributable to normal variation in performance rather than limitations of the videoconference adaptation. Similarly, ceiling effects observed on some domains of the BASS have also been seen in other social cognition tests, such as The Awareness of Social Interference Test for emotion recognition (McDonald et al. 2003). Although this does not limit the clinical utility of the tool, it does limit our ability to evaluate score reliability within non-impaired samples.
The study contributes to contemporary drives to confirm the feasibility of telehealth assessment for clinical practice. Indeed, guidelines for remote administration of the ACE-III were published during the current study (FRONTIER 2020), and current results provide initial empirical evidence supporting the feasibility of videoconference administration in line with these guidelines. However, despite the current research exploring several areas of feasibility, future work must examine other relevant elements, including demand, implementation, and integration with existing technologies in the Australian healthcare system.
In line with recommendations for evaluating tools adapted for telehealth (Agboola et al. 2014), a study of healthy volunteers in an experimental setting was undertaken to reduce the possible sources of variance in test results. There are two fundamental limitations to the conclusions drawn from the current sample. First, the acceptance and ability to complete testing via telehealth demonstrated in the current sample may not be the case for clinical samples (Greenhalgh et al. 2013). Second, healthy volunteers generally performed well on assessment battery tools, resulting in limited ability to examine the reliability of scores at lower ends of performance. It can be expected that additional support will be required for clinical populations to boost engagement with telehealth and assist with any technical issues (Banbury et al. 2014). A model whereby a caregiver provides some support during the assessment, as has been done in other studies (DeYoung and Shenal 2019), would be a suitable next step in examining the feasibility of this screening battery among older people with cognitive or physical impairments. A final limitation is the relatively modest sample size, which could introduce unwanted variance in reliability. While not apparent in sensitivity analyses, longer lags associated with test–retest may result in lower reliability across modalities than what is observed here. Although this is acceptable for a feasibility study, it does limit conclusions that can be drawn from some analyses.
Conclusion
This study demonstrates the feasibility and acceptability of a telehealth-based, multi-domain assessment for MCI among older adults. The findings indicate that telehealth can be a practical and reliable alternative to in-person assessments, with minimal modifications required for the assessment of cognitive, social cognition, mood, and functional independence. Participants reported high levels of acceptability and satisfaction with the telehealth format, highlighting its potential to overcome barriers to assessment faced by older adults, such as mobility and transportation issues, and findings provide an initial step towards increasing access to assessment services among older adult populations. The reliability of the assessment scores across both formats further supports the validity of telehealth as a viable method for early detection and monitoring of MCI and dementia symptoms. However, the study also identifies areas for further research, particularly concerning the biases observed in emotion perception tasks, and supports needed for clinical populations engaging with assessment via telehealth. Overall, the adoption of telehealth technologies in cognitive, social, mood, and functional assessments can significantly enhance access to healthcare services for older adults, promoting early intervention and improving health outcomes. The availability and adoption of assessments via telehealth will support early detection of MCI and symptoms of dementia, indicating whether more comprehensive evaluation is necessary and improving outcomes associated with early treatment and monitoring.
Data availability
The data that support this study cannot be publicly shared due to ethical or privacy reasons and may be shared upon reasonable request to the corresponding author if appropriate.
Conflicts of interest
The authors declare no potential conflicts of interest regarding the research, authorship, or publication of this article.
Ethics
Ethics approval was obtained from the University of Newcastle Human Research Ethics and Hunter New England Health Research Ethics Committees (HREC – H-2015-0255; HNEHREC – 14/05/21/4.02). The authors affirm having followed professional ethical guidelines in preparing this work, including obtaining informed consent from participants, maintaining ethical treatment and respect for their rights, and ensuring the privacy of participants and their data, ensuring that individual participants cannot be identified in reported results.
Acknowledgements
Data collection was completed with support from the Hunter Medical Research Institute and Maroba Aged Care.
References
Aaronson N, Alonso J, Burnam A, Lohr KN, Patrick DL, Perrin E, Stein RE (2002) Assessing health status and quality-of-life instruments: Attributes and review criteria. Quality of Life Research: 11(3), 193-205.
| Crossref | Google Scholar | PubMed |
ABS (2018) ‘Australian Statistical Geography Standard (ASGS): Volume 5—Remoteness Structure, July 2016 Correspondence, 2017 Postcode to 2016 Remoteness Area (1270.0.55.005).’ (Australian Bureau of Statistics) Available at https://www.abs.gov.au/
Adolphs R (2003) Cognitive neuroscience of human social behaviour. Nature Reviews Neuroscience 4(3), 165-178.
| Crossref | Google Scholar | PubMed |
Agboola S, Hale TM, Masters C, Kvedar J, Jethwani K (2014) Real-World” Practical Evaluation Strategies: A Review of Telehealth Evaluation. JMIR Research Protocols 3(4), e75.
| Crossref | Google Scholar | PubMed |
Bai W, Chen P, Cai H, Zhang Q, Su Z, Cheung T, Jackson T, Sha S, Xiang Y-T (2022) Worldwide prevalence of mild cognitive impairment among community dwellers aged 50 years and older: A meta-analysis and systematic review of epidemiology studies. Age and Ageing 51(8), afac173.
| Crossref | Google Scholar | PubMed |
Banbury A, Parkinson L, Nancarrow S, Dart J, Gray L, Buckley J (2014) Multi-site videoconferencing for home-based education of older people with chronic conditions: The Telehealth Literacy Project. Journal of Telemedicine and Telecare 20(7), 353-359.
| Crossref | Google Scholar | PubMed |
Bartko JJ (1966) The Intraclass Correlation Coefficient as a Measure of Reliability. Psychological Reports 19(1), 3-11.
| Crossref | Google Scholar | PubMed |
Bentvelzen A, Aerts L, Seeher K, Wesson J, Brodaty H (2017) A Comprehensive Review of the Quality and Feasibility of Dementia Assessment Measures: The Dementia Outcomes Measurement Suite. Journal of the American Medical Directors Association 18(10), 826-837.
| Crossref | Google Scholar | PubMed |
Blackberry I, Morris N (2023) The Impact of Population Ageing on Rural Aged Care Needs in Australia: Identifying Projected Gaps in Service Provision by 2032. Geriatrics (Basel, Switzerland) 8(3), 47.
| Crossref | Google Scholar | PubMed |
Bland JM, Altman DG (1990) A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement. Computers in Biology and Medicine 20(5), 307-310.
| Crossref | Google Scholar |
Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, Bakken S, Kaplan CP, Squiers L, Fabrizio C, Fernandez M (2009) How We Design Feasibility Studies. American Journal of Preventive Medicine 36(5), 452-457.
| Crossref | Google Scholar | PubMed |
Campbell NL, Unverzagt F, LaMantia MA, Khan BA, Boustani MA (2013) Risk factors for the progression of mild cognitive impairment to dementia. Clinics in Geriatric Medicine 29(4), 873-893.
| Crossref | Google Scholar | PubMed |
Carotenuto A, Traini E, Fasanaro AM, Battineni G, Amenta F (2021) Tele-Neuropsychological Assessment of Alzheimer’s Disease. Journal of Personalized Medicine 11(8), 688.
| Crossref | Google Scholar | PubMed |
Clemson L, Bundy A, Unsworth C, Singh MF (2009) Validation of the modified assessment of living skills and resources, an IADL measure for older people. Disabil Rehabil 31(5), 359-69.
| Crossref | Google Scholar | PubMed |
DeYoung N, Shenal BV (2019) The reliability of the Montreal Cognitive Assessment using telehealth in a rural setting with veterans. Journal of Telemedicine and Telecare 25(4), 197-203.
| Crossref | Google Scholar | PubMed |
Fiske A, Wetherell JL, Gatz M (2009) Depression in older adults. Annual Review of Clinical Psychology 5, 363-389.
| Crossref | Google Scholar | PubMed |
FRONTIER (2020) ‘ACE-III Remote Administration Guidelines and Forms (Addenbrooke’s Cognitive Examination-III (ACE-III)).’ (Frontotemporal Dementia Research Group, University of Sydney) Available at https://frontierftd.org/
Greenhalgh T, Wherton J, Sugarhood P, Hinder S, Procter R, Stones R (2013) What matters to older people with assisted living needs? A phenomenological analysis of the use and non-use of telehealth and telecare. Social Science & Medicine 93, 86-94.
| Crossref | Google Scholar | PubMed |
Grubaugh AL, Cain GD, Elhai JD, Patrick SL, Frueh BC (2008) Attitudes Toward Medical and Mental Health Care Delivered Via Telehealth Applications Among Rural and Urban Primary Care Patients. Journal of Nervous & Mental Disease 196(2), 166-170.
| Crossref | Google Scholar | PubMed |
Harrell KM, Wilkins SS, Connor MK, Chodosh J (2014) Telemedicine and the Evaluation of Cognitive Impairment: The Additive Value of Neuropsychological Assessment. Journal of the American Medical Directors Association 15(8), 600-606.
| Crossref | Google Scholar | PubMed |
Hernandez HHC, Ong PL, Anthony P, Ang SL, Salim N, Yew PYS, Ali NB, Lim JP, Lim WS, Chew J (2022) Cognitive Assessment by Telemedicine: Reliability and Agreement between Face-to-Face and Remote Videoconference-Based Cognitive Tests in Older Adults Attending a Memory Clinic. Annals of Geriatric Medicine and Research 26(1), 42-48.
| Crossref | Google Scholar | PubMed |
Hsieh S, Schubert S, Hoon C, Mioshi E, Hodges JR (2013) Validation of the Addenbrooke’s Cognitive Examination III in Frontotemporal Dementia and Alzheimer’s Disease. Dementia and Geriatric Cognitive Disorders 36(3–4), 242-250.
| Crossref | Google Scholar | PubMed |
Hwang R, Mandrusiak A, Morris NR, Peters R, Korczyk D, Russell T (2017) Assessing functional exercise capacity using telehealth: Is it valid and reliable in patients with chronic heart failure? Journal of Telemedicine and Telecare 23(2), 225-232.
| Crossref | Google Scholar | PubMed |
Inouye SK, Peduzzi PN, Robison JT, Hughes JS, Horwitz RI, Concato J (1998) Importance of Functional Measures in Predicting Mortality Among Older Hospitalized Patients. JAMA 279(15), 1187.
| Crossref | Google Scholar | PubMed |
Jang SY, Seon JY, Oh IH (2020) Influencing Factors of Transportation Costs Regarding Healthcare Service Utilization in Korea. Journal of Korean Medical Science 35(35), e290.
| Crossref | Google Scholar | PubMed |
Kann MR, Zeiger PJ, Rizer SJ, Cosentino S, Azar M (2023) Virtual Assessment as a Way to Reduce Help-seeking Barriers in Older Adults With Subjective Cognitive Decline. Alzheimer Disease & Associated Disorders 37(4), 373-378.
| Crossref | Google Scholar | PubMed |
Kelly M, McDonald S (2020) Assessing social cognition in people with a diagnosis of dementia: Development of a novel screening test, the Brief Assessment of Social Skills (BASS-D). Journal of Clinical and Experimental Neuropsychology 42(2), 185-198.
| Crossref | Google Scholar | PubMed |
Kobak KA (2004) A comparison of face-to-face and videoconference administration of the Hamilton Depression Rating Scale. Journal of Telemedicine and Telecare 10(4), 231-235.
| Crossref | Google Scholar | PubMed |
Koo TK, Li MY (2016) A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine 15(2), 155-163.
| Crossref | Google Scholar | PubMed |
Krasniuk S, Crizzle AM (2023) Impact of health and transportation on accessing healthcare in older adults living in rural regions. Transportation Research Interdisciplinary Perspectives 21, 100882.
| Crossref | Google Scholar |
Kumfor F, Irish M, Hodges JR, Piguet O (2013) Discrete Neural Correlates for the Recognition of Negative Emotions: Insights from Frontotemporal Dementia. PLoS One 8(6), e67457.
| Crossref | Google Scholar | PubMed |
Kumfor F, Hutchings R, Irish M, Hodges JR, Rhodes G, Palermo R, Piguet O (2015) Do I know you? Examining face and object memory in frontotemporal dementia. Neuropsychologia 71, 101-111.
| Crossref | Google Scholar | PubMed |
Lavingia R, Jones K, Asghar-Ali AA (2020) A Systematic Review of Barriers Faced by Older Adults in Seeking and Accessing Mental Health Care. Journal of Psychiatric Practice 26(5), 367-382.
| Crossref | Google Scholar | PubMed |
Lee J, Koh D, Ong CN (1989) Statistical evaluation of agreement between two methods for measuring a quantitative variable. Computers in Biology and Medicine 19(1), 61-70.
| Crossref | Google Scholar | PubMed |
Leyhe T, Reynolds CF, Melcher T, Linnemann C, Klöppel S, Blennow K, Zetterberg H, Dubois B, Lista S, Hampel H (2017) A common challenge in older adults: Classification, overlap, and therapy of depression and dementia. Alzheimer’s & Dementia 13(1), 59-71.
| Crossref | Google Scholar | PubMed |
Livingston G, Sommerlad A, Orgeta V, Costafreda SG, Huntley J, Ames D, Ballard C, Banerjee S, Burns A, Cohen-Mansfield J, Cooper C, Fox N, Gitlin LN, Howard R, Kales HC, Larson EB, Ritchie K, Rockwood K, Sampson EL, et al. (2017) Dementia prevention, intervention, and care. The Lancet 390(10113), 2673-2734.
| Crossref | Google Scholar | PubMed |
Luxton DD, Pruitt LD, Osenbach JE (2014) Best practices for remote psychological assessment via telehealth technologies. Professional Psychology: Research and Practice 45(1), 27-35.
| Crossref | Google Scholar |
Mahon S, Webb J, Snell D, Theadom A (2022) Feasibility of administering the WAIS-IV using a home-based telehealth videoconferencing model. The Clinical Neuropsychologist 36(3), 558-570.
| Crossref | Google Scholar | PubMed |
Marra DE, Hamlet KM, Bauer RM, Bowers D (2020) Validity of teleneuropsychology for older adults in response to COVID-19: A systematic and critical review. The Clinical Neuropsychologist 34(7–8), 1411-1452.
| Crossref | Google Scholar | PubMed |
McDonald S, Flanagan S, Rollins J, Kinch J (2003) TASIT: A New Clinical Tool for Assessing Social Perception After Traumatic Brain Injury. The Journal of Head Trauma Rehabilitation 18(3), 219-238.
| Crossref | Google Scholar | PubMed |
McDonald S, Bornhofen C, Shum D, Long E, Saunders C, Neulinger K (2006) Reliability and validity of The Awareness of Social Inference Test (TASIT): A clinical test of social perception. Disability and Rehabilitation 28(24), 1529-1542.
| Crossref | Google Scholar | PubMed |
Mioshi E, Dawson K, Mitchell J, Arnold R, Hodges JR (2006) The Addenbrooke’s Cognitive Examination Revised (ACE‐R): A brief cognitive test battery for dementia screening. International Journal of Geriatric Psychiatry 21(11), 1078-1085.
| Crossref | Google Scholar | PubMed |
Mueller KD, Koscik RL, Hermann BP, Johnson SC, Turkstra LS (2018) Declines in Connected Language Are Associated with Very Early Mild Cognitive Impairment: Results from the Wisconsin Registry for Alzheimer’s Prevention. Frontiers in Aging Neuroscience 9, 437.
| Crossref | Google Scholar | PubMed |
Munro Cullum C, Hynan LS, Grosch M, Parikh M, Weiner MF (2014) Teleneuropsychology: Evidence for Video Teleconference-Based Neuropsychological Assessment. Journal of the International Neuropsychological Society 20(10), 1028-1033.
| Crossref | Google Scholar | PubMed |
Parks AC, Davis J, Spresser CD, Stroescu I, Ecklund-Johnson E (2021) Validity of In-Home Teleneuropsychological Testing in the Wake of COVID-19. Archives of Clinical Neuropsychology 36(6), 887-896.
| Crossref | Google Scholar | PubMed |
Petersen RC, Lopez O, Armstrong MJ, Getchius T, Ganguli M, Gloss D, Gronseth GS, Marson D, Pringsheim T, Day GS, Sager M, Stevens J, Rae-Grant A (2018) Practice guideline update summary: Mild cognitive impairment: Report of the Guideline Development, Dissemination, and Implementation Subcommittee of the American Academy of Neurology. Neurology 90(3), 126-135.
| Crossref | Google Scholar | PubMed |
Prestia A, Rossi R, Geroldi C, Galluzzi S, Ettori M, Alaimo G, Frisoni GB (2006) Validation Study of the Three-Objects-Three-Places Test: A Screening Test for Alzheimer’s Disease. Experimental Aging Research 32(4), 395-410.
| Crossref | Google Scholar | PubMed |
Shah S, Vanclay F, Cooper B (1989) Improving the sensitivity of the Barthel Index for stroke rehabilitation. Journal of Clinical Epidemiology 42(8), 703-709.
| Crossref | Google Scholar | PubMed |
Tabert MH, Albert SM, Borukhova-Milov L, Camacho Y, Pelton G, Liu X, Stern Y, Devanand DP (2002) Functional deficits in patients with mild cognitive impairment: Prediction of AD. Neurology 58(5), 758-764.
| Crossref | Google Scholar | PubMed |
Tousignant M, Boissy P, Moffet H, Corriveau H, Cabana F, Marquis F, Simard J (2011) Patients’ Satisfaction of Healthcare Services and Perception with In-Home Telerehabilitation and Physiotherapists’ Satisfaction Toward Technology for Post-Knee Arthroplasty: An Embedded Study in a Randomized Trial. Telemedicine and E-Health 17(5), 376-382.
| Crossref | Google Scholar | PubMed |
Turner TH, Brearly TW (2023) Videoconference teleneuropsychology. In ‘APA Handbook of Neuropsychology, Volume 2: Neuroscience and Neuromethods. Vol. 2’. (Eds GG Brown, B Crosson, KY Haaland, TZ King) pp. 313–332. (American Psychological Association) 10.1037/0000308-015
Van Gaans D, Dent E (2018) Issues of accessibility to health services by older Australians: A review. Public Health Reviews 39(1), 20.
| Crossref | Google Scholar | PubMed |
Wolfe MK, McDonald NC, Holmes GM (2020) Transportation Barriers to Health Care in the United States: Findings From the National Health Interview Survey, 1997–2017. American Journal of Public Health 110(6), 815-822.
| Crossref | Google Scholar | PubMed |
Woodward M, Brodaty H, McCabe M, Masters CL, Naismith SL, Morris P, Rowe CC, Walker P, Yates M (2022) Nationally Informed Recommendations on Approaching the Detection, Assessment, and Management of Mild Cognitive Impairment. Journal of Alzheimer’s Disease 89(3), 803-809.
| Crossref | Google Scholar | PubMed |
Zeghari R, Guerchouche R, Tran-Duc M, Bremond F, Langel K, Ramakers I, Amiel N, Lemoine MP, Bultingaire V, Manera V, Robert P, König A (2022) Feasibility Study of an Internet-Based Platform for Tele-Neuropsychological Assessment of Elderly in Remote Areas. Diagnostics 12(4), 925.
| Crossref | Google Scholar | PubMed |
Zigmond AS, Snaith RP (1983) The Hospital Anxiety and Depression Scale. Acta Psychiatrica Scandinavica 67(6), 361-370.
| Crossref | Google Scholar | PubMed |
