Navigating the data universe: multi-agent AI sytem for proactive database monitoring and DBA decision support in large astronomy science archives

Thursday, October 01 · 13:30–14:20
Letterpress Intermediate

Modern scientific space missions generate data volumes and processing workloads that rival those of the largest commercial platforms. Projects such as ESA's Euclid, PLATO, and Gaia rely on complex database infrastructures to support large scale data ingestion, processing, catalogue generation, and scientific analysis. In these environments, the role of the Database Administrator (DBA) extends far beyond traditional database management. DBAs must simultaneously address high risk operational challenges, ensuring data integrity, availability, performance, and scalability—while also spending significant time on repetitive monitoring and maintenance activities.

Despite advances in observability platforms, a substantial portion of database operations remains focused on routine tasks such as monitoring CPU, memory, and I/O utilization, tracking alert conditions, identifying table bloat, validating optimizer statistics, and investigating performance degradation in append only workloads. These activities are essential but often consume valuable expert time that could otherwise be dedicated to architectural improvements and complex problem resolution.

In this context we propose a multi-agent AI system designed to augment, rather than replace, the DBA. The architecture consists of specialized agents responsible for infrastructure observability, workload analysis, anomaly detection, alert correlation, and remediation planning. By continuously analyzing database and operating-system metrics, the agents can identify emerging issues, prioritize alerts, and generate intervention recommendations. Importantly, the system does not execute corrective actions autonomously. Instead, it presents proposed SQL or operational scripts to the DBA for review and approval, preserving human oversight and operational safety in critical environments.

The proposed approach is applicable to both PostgreSQL and PostgreSQL based MPP clusters (Greenplum, WarehousePG) supporting scientific data processing pipelines like those implemented in ESA's Euclid, PLATO and Gaia space missions. By reducing the operational burden associated with repetitive monitoring and diagnostics, multi-agent AI systems enable DBAs to focus on strategic optimization, performance engineering, and problem-solving activities that require domain expertise and human judgment.

As scientific data platforms continue to expand toward galaxy-scale workloads, AI-assisted database operations offer a practical path toward increased efficiency, improved responsiveness, and more sustainable management of the databases that underpin modern space science.