InkluBot: Inclusive voice control for robots in the age of large language models (LLMs)

The InkluBot project aims to further develop AI-based assistance and robot systems for use in workshops for people with disabilities (WfbMs) and to make them barrier-free. The focus is on adapting and integrating large language models (LLMs) for intuitive robot control in order to enable understandable, situation-appropriate interaction between humans and machines. Particular attention is paid to the processing of non-standardized language and the generation of easily understandable feedback for people with disabilities (PwD).

A central research focus is the improvement of automatic speech recognition (ASR) through optimized models, sound recognition, and processing. To this end, a three-stage adaptive interaction model is being developed: (1) standardized voice control for people without disabilities, (2) ASR optimization for mild to moderate speech impairments, including sentence structure processing, and (3) alternative control approaches such as sound-to-instruction systems based on sounds and nonverbal signals.

Die Innovation des Projekts liegt insbesondere in der domänenspezifischen Anpassung von LLMs sowie der Entwicklung eines neuartigen Sound-to-Instruction-Systems. Durch den Einsatz speziell erhobener Trainingsdatensätze aus WfbMs lernt das System, mit variierenden und abweichenden Sprachmustern zuverlässig umzugehen. InkluBot demonstriert damit das Potenzial von KI und Robotik zur Förderung selbstbestimmter Inklusion, zur Unterstützung rehabilitativer Prozesse und zur Schaffung gesamtunternehmerischer Mehrwerte. Die erzielten Ergebnisse sind auf weitere Anwendungsfelder, insbesondere im Gesundheitswesen und in der Industrie, übertragbar.

The innovation of the project lies in particular in the domain-specific adaptation of LLMs and the development of a novel sound-to-instruction system. By using specially collected training data sets from WfbMs, the system learns to reliably handle varying and deviating speech patterns. InkluBot thus demonstrates the potential of AI and robotics to promote self-determined inclusion, support rehabilitative processes, and create overall business value. The results achieved are transferable to other fields of application, particularly in healthcare and industry.