PANDALens: Towards AI-Assisted In-Context Writing on OHMD During Travels

Abstract: While effective for recording and sharing experiences, traditional in-context writing tools are relatively passive and unintelligent, serving more like instruments rather than companions. This reduces primary task (e.g., travel) enjoyment and hinders high-quality writing. Through formative study and iterative development, we introduce PANDALens, a Proactive AI Narrative Documentation Assistant built on an Optical See-Through Head-Mounted Display that transforms the in-context writing tool into an intelligent companion. PANDALens observes multimodal contextual information from user behaviors and environment to confirm interests and elicit contemplation, and employs Large Language Models to transform such multimodal information into coherent narratives with significantly reduced user effort. A real-world travel scenario comparing PANDALens with a smartphone alternative confirmed its effectiveness in improving writing quality and travel enjoyment while minimizing user effort. Accordingly, we propose design guidelines for AI-assisted in-context writing, highlighting the potential of transforming them from tools to intelligent companions.

For more details, visit https://github.com/Synteraction-Lab/PANDALens 

Team

• Runze Cai
• Nuwan Janaka
• Yang Chen
• Lucia Wang
• Shengdong Zhao
• Can Liu