A Multimodally Controlled Assistive Robotic System for Physically Impaired Users

Sheltered Workshops provide essential structures for vocational rehabilitation and social participation, while simultaneously facing increasing technological demands. These technological demands include competences in AI-assisted work and collaborative robotics. Despite their widespread use in the general population and their capacity to make various applications more accessible, assistive AI tools like speech recognition rarely work for people with disabilities. This article proposes a concept for an accessible AI-supported multimodal control system for collaborative robots to enable more inclusive work environments. With this concept, we aim to combine speech recognition, Sound-to-Text models, intent detection, Large Language Models, and RAG-based control input generation to translate individualized user input into safe and admissible robot actions. The proposed approach therefore addresses both technological innovation and the broader objective of self-determined participation in value-creating work processes.