TALK-Dem: Enhancing Task Planning for Dementia Communication Challenges

Paper: TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns

Authors: Guangxin Zhao, Yiran Hu, Yuan Cao, Chenxi Jiang, Jianfei Yang, Yegang Du, Yasuyuki Taki, Yoshifumi Kitamura, Lin Gu, Zhi Zheng

Executive Summary

Recent research highlights significant limitations in how large language model (LLM)-driven task planners perform when interacting with real users, particularly those with cognitive impairments such as dementia. Traditional task planners operate under the assumption that user instructions will be clear and focused, leading to challenges when faced with the more nuanced and variable communication styles of individuals experiencing these impairments. This disconnect can result in errors that pose safety risks, especially in applications like assistive robotics.

To address these challenges, researchers developed TALK-Dem, a comprehensive benchmark designed to evaluate LLM-driven task planning in the context of dementia-related verbal communication. This benchmark includes 4,800 distinct user instructions, encompassing five typical communication patterns associated with dementia: Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion. Each of these patterns is tested at three intensity levels, allowing for a nuanced understanding of how well these models can respond to varied user inputs.

In their experiments, the researchers evaluated six different open-weight LLMs and found a notable robustness gap in their performance when faced with real-world communication challenges. Specifically, the models experienced performance drops of up to 22.3 percentage points compared to their effectiveness in interpreting ideal instructions. This finding underscores the potential dangers of deploying these models in real-world applications without adaptations that consider the unique communication needs of users with dementia.

To improve task success rates, the research introduced the Context-Aware Retrieval from Experience (CARE) method. This innovative approach retrieves contextually relevant past tasks to enhance the model's understanding and planning capabilities during interactions. The results demonstrated that CARE improved task success by an average of 18.1 percentage points over standard prompting techniques across the tested models.

The implications of this research are significant for the development of assistive robotics. As these robots are often designed to operate in environments with privacy concerns and limited connectivity, it’s crucial that they can effectively interpret and respond to the diverse communication styles of users. The TALK-Dem dataset, which is publicly available, facilitates further research and development in this area, encouraging advancements in the robustness and safety of assistive technologies.

In summary, this research not only addresses the shortcomings of existing LLM-driven task planners but also provides a framework for improving their performance in real-world contexts, particularly for vulnerable populations. By focusing on the unique communication patterns exhibited by individuals with dementia, it opens the door for more effective and safer applications of AI in assistive settings.

Academic Abstract

Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focused. However, when interacting with real-world users, especially those experiencing cognitive impairments, such as people living with dementia (PLWD), the planners often make mistakes and even pose physical safety risks. We proposed TALK-Dem (Talking Attributes and Linguistic Knowledge in Dementia), the first benchmark for evaluating LLM-driven robot task planning under dementia-associated verbal communication. TALK-Dem contains 4,800 instructions and covers five typical communication patterns, including Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion, at three intensity levels. Experiments across six open-weight LLMs reveal a substantial robustness gap. Across communication patterns, open-weight models exhibited performance drops of up to 22.3 percentage points compared to ideal instructions. This revealed a critical gap and even danger for real-world applications, especially in assistive robotics, where locally deployable models are necessary due to privacy concerns and connectivity constraints. To mitigate this issue, we proposed the Context-Aware Retrieval from Experience (CARE) method, which retrieves relevant previously resolved tasks to provide task-specific interpretation and planning context. CARE generally outperformed standard prompting baselines across the six open-weight models, improving average task success by 18.1 percentage points over the vanilla prompt. These results highlighted the importance of both evaluating communication robustness and developing effective adaptation strategies for locally deployable assistive robots. The TALK-Dem dataset is publicly available at https://anonymous.4open.science/r/TALK-Dem-A6B3/.