Introduction

A prototypical Mixed Reality concept of swarm © DALL-E

Swarm robotics involves multiple entities coordinating toward a common goal, often referred to as multi-robot systems in real-world applications [1], which are also described as multi-agent systems [2]. Advancements in Artificial Intelligence (AI), edge computing, and novel sensors have further enabled decentralized coordination and complex behaviors. However, challenges remain around task allocation, communication, and the large state spaces of human interaction. 

As noted in [3], control and interaction vary with autonomy levels, highlighting the importance of human–computer interaction (HCI) in swarm robotics. Motivated by these findings, we intend to extend prior work through a systematic literature review that examines emerging AI and multimodal interfaces in Human–Swarm Interaction (HSI). 

For example, recent reviews focus on HSI modalities, including gesture control, voice commands, and brain–computer interfaces [4, 5]. Further inquiries highlight the importance of user experience design and operator workload management for UAV swarms [6].

 

Opportunities

Example of the use extended reality technology

In the current literature, there is a growing need to explore how Extended Reality (XR) can be integrated with Artificial Intelligence (AI), often referred to as AI/XR to enhance user experiences when interacting with swarm systems.

Human-centered AI integration can enable novel interaction methods such as communication tools, state estimation, visualization, and advanced control mechanisms. These capabilities have the potential to make swarm systems more intuitive and accessible.

Challenges

  1. Lack of Natural Interaction
    Current interfaces often fail to provide a natural way for end users to interact with swarms.

  2. Inconsistent Terminology
    The field uses varied and sometimes conflicting terms (e.g., “multi-agent systems”, “multi-robot systems”, “collective robotics” “swarms”), leading to confusion

  3. Gap Between Simulation and Reality
    Many existing studies rely on unrealistic simulations that do not fully reflect the challenges of implementing swarm systems in real-world settings.

Research Questions

RQ1: Which factors most significantly influence Human-Swarm Interaction (HSI) in current studies?

RQ2: How do swarm architectures enable or constrain HSI?

RQ3: What are the primary patterns and challenges emerging in HSI research?

Objective

  • Develop Humas Swarm Interaction taxonomy;

  • Develop XR simulation infrastructure for human in the loop simulation leveraging mixed reality capabilities;

  • Evaluate novel interaction to monitor, control and decision making;

References

  1. A. Hocraffer and C. S. Nam, “A meta-analysis of human-system interfaces in unmanned aerial vehicle (UAV) swarm management,” Applied Ergonomics, vol. 58, pp. 66–80, 2017, doi: 10.1016/j.apergo.2016.05.011.
  2. A. Dahiya, A. M. Aroyo, K. Dautenhahn, and S. L. Smith, “A survey of multi-agent Human–Robot Interaction systems,” Robotics and Autonomous Systems, vol. 161, 104335, 2023, doi: 10.1016/j.robot.2022.104335.
  3. A. Kolling, P. Walker, N. Chakraborty, K. Sycara, and M. Lewis, “Human interaction with robot swarms: A survey,” IEEE Transactions on Human-Machine Systems, vol. 46, no. 1, pp. 9–26, 2016, doi: 10.1109/THMS.2015.2480801.
  4. V. Steane, J. Oakes, S. Palmer, and M. Chattington, “Human-swarm partnerships: A systematic review of human factors literature,” in D. De Waard, K. A. Brookhuis, C. Weikert, C. Yamamoto, and F. R. Drews (eds.), Human Factors in Robots, Drones and Unmanned Systems, 2023, pp. 121–131, AHFE Open Access, doi: 10.54941/ahfe1003754.
  5. A.-I. Şiean, B.-C. Grădinaru, O.-I. Gherman, M. Danubianu, and L.-D. Milici, “Opportunities and challenges in human-swarm interaction: Systematic review and research implications,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 4, pp. 896–904, 2023, doi: 10.14569/IJACSA.2023.0140498.
  6. S. Aldhaheri, F. Renda, and G. De Masi, “Underwater human-robot and human-swarm interaction: A review and perspective,” arXiv preprint arXiv:2406.12473, 2024, doi: 10.48550/arXiv.2406.12473.

THI contact person

Prof. Dr.-Ing. Simon Schwerd
Phone: +49 841 9348-6417
Room: K210
E-Mail:
Research Assistant
Diah Ayu Irawati
Phone: +49 841 9348-5209
E-Mail: