As robotic systems become increasingly integrated into real-world environments, ranging from autonomous vehicles to household assistants, they inevitably encounter diverse and unstructured scenarios that lead to failures. While such failures pose safety and reliability challenges, they also provide rich perceptual data for improving system robustness. However, manually analyzing large-scale failure datasets is impractical and does not scale. In this work, we introduce the problem of unsupervised discovery of failure taxonomies from large volumes of raw failure logs, aiming to obtain semantically coherent and actionable failure modes directly from perceptual trajectories. Our approach first infers structured failure explanations from multimodal inputs using vision-language reasoning, and then performs clustering in the resulting semantic reasoning space, enabling the discovery of recurring failure modes rather than isolated episode-level descriptions. We evaluate our method across robotic manipulation, indoor navigation, and autonomous driving domains, and demonstrate that the discovered taxonomies are consistent, interpretable, and practically useful. In particular, we show that structured failure taxonomies guide targeted data collection for offline policy refinement and enhance runtime failure monitoring systems.
We cluster robotic failure data into semantically meaningful failure modes using Multimodal Large Language Models, in two steps:
MLLMs struggle at runtime monitoring without context about how a system fails. We supply the discovered clusters as that context, listing the system's most probable failures. A Chain-of-Thought prompt has the model first predict the robot's future trajectory, then check whether that path could lead into any known failure mode.
We compare against LLM-based anomaly detection (LLM-AD), CNN-based classifiers (VideoMAE-BC, ENet-BC), and an ablation with no cluster context (NoContext).
Our method achieves higher F-1 scores and earlier detection times than all baselines — including NoContext, confirming that the failure clusters themselves are what make the difference. (See the paper for baseline details and prompts.)
Below, runtime monitoring in action with an expert fallback policy that overrides the controller on detection.
The discovered clusters guide expert data collection in targeted regions of the environment. We fine-tune the policy on the original training data plus 40K additional samples collected in the identified failure zones.
Failure rate: 46% → 18% versus 34% when fine-tuning on randomly collected data
This closes the loop — failure discovery, targeted intervention, policy refinement — for continuously improving system safety.
