End-to-End Autonomous Systems: Bridging Vision, Language, Action, and Simulation for Next-Generation Autonomous Agents Yakov Miron Bosch Research, Germany

Abstract The promise of truly intelligent autonomous agents hinges on our ability to move beyond both brittle, modular pipelines and opaque “black-box” models. This lecture articulates a comprehensive vision for the next generation of autonomy by charting its evolution toward verifiable, White-Box End-to-End Systems. We begin by tracing the journey from early two-stage architectures to modern one-stage and high-capacity Vision-Language-Action (VLA) frameworks. While these systems show remarkable performance on paper, they suffer from fundamental vulnerabilities: a lack of interpretability, a dramatic performance collapse in closed-loop settings, and a deep-seated tendency for shortcut learning. Far from resolving this, high-capacity VLAs often exacerbate the problem by exploiting spurious correlations rather than learning true physical causality. To address these challenges, we outline a future roadmap toward “White-Box” VLA architectures. We propose structurally constraining the model’s reasoning by forcing it to generate explicit, interpretable intermediate states—such as spatially-grounded BEV maps and trajectory predictions—thereby restricting its capacity to exploit shortcuts. Finally, we close the loop with our completed work in verification. We introduce Pseudo-Simulation, a framework that uses highly realistic virtual worlds built directly from real driving data to test our systems safely. By dynamically changing these virtual environments, we can systematically expose compounding drift and hidden shortcut behaviors before the agent ever drives a real vehicle. Together, this presentation outlines a complete, practical lifecycle for building autonomous agents that are both highly capable and provably safe. Short Biography Yakov Miron received the B.Sc. degree in electrical engineering from Ben-Gurion University, Israel, in 2009, and the M.Sc. degree in electrical engineering from Tel-Aviv University, Israel, in 2016. He received the Ph.D. degree in AI for Robotics from the University of Haifa, Israel, where his research in multi-agent hybrid learning integrated perception, localization, and path planning for mobile robots. He is currently a Senior Manager and Senior Research Scientist with Bosch Re-search, Germany, where he leads a lab of 25 researchers developing end-to-end learning solutions for autonomous systems. He is also a Research Fellow with the University of Haifa, a guest lecturer with the University of Stuttgart, and an Associate Editor for the IEEE Open Journal of Vehic-ular Technology. His research interests include end-to-end AI, perception, motion planning, and decision-making for mobile robots and self-driving cars, with a strong focus on leveraging Large Language Models (LLMs) and Vision-Language-Action (VLA) models for embodied intelligence.