OpsAgent: An Evolving Multi-agent System for Incident Management in Microservices

Yu Luo; Jiamin Jiang; Jingfei Feng; Lei Tao; Qingliang Zhang; Xidao Wen; Yongqian Sun; Shenglin Zhang; Tong Liu; Wenjie Zhang; Dan Pei

01 THE PAPER

Abstract

Incident management (IM) is central to the reliability of large-scale microservice systems. Yet manual IM, where on-call engineers examine metrics, logs, and traces, is labor-intensive and error-prone in the face of massive and heterogeneous observability data. Existing automated IM approaches often struggle to generalize across systems, provide limited interpretability, and incur high deployment costs, which hinders adoption in practice. In this paper, we present OpsAgent, a lightweight, self-evolving multi-agent system for IM that employs a training-free data processor to convert heterogeneous observability data into structured textual descriptions, along with a multi-agent collaboration framework that makes diagnostic inference transparent and auditable. To support continual capability growth, OpsAgent also introduces a dual self-evolution mechanism that integrates internal model updates with external experience accumulation, thereby closing the deployment loop. Comprehensive experiments on the OPENRCA benchmark demonstrate state-of-the-art performance and show that OpsAgent is generalizable, interpretable, cost-efficient, and self-evolving, making it a practically deployable and sustainable solution for long-term operation in real-world microservice systems. Notably, its deployment in Lenovo's production environment further validates its effectiveness in real-world industrial settings.

Project Overview

Training-Free Data Processor

OpsAgent first converts metrics, logs, and traces into structured textual descriptions without system-specific training, preserving useful anomaly, log, and latency evidence while keeping deployment cost low.

Auditable Multi-Agent Diagnosis

An intent interpreter, orchestrator, and specialized agents cooperate through agent profiles and cross-review, turning root-cause diagnosis into a transparent workflow that OCEs can inspect and act on.

Dual Self-Evolution Loop

The system improves through internal PPO-based optimization and external reflection-based experience accumulation, closing the loop between online incident handling and offline capability growth.

02 PROJECT FILM

One Video, and You Know What We Built.

See how OpsAgent transforms heterogeneous observability data into transparent diagnosis and sustainable self-evolution.

03 THE FRAMEWORK

Method

Training-Free Data Processor: Metrics, logs, and traces are processed separately into compact diagnostic evidence without retraining a system-specific model.

Illustrative example of textual data descriptions

Textual Descriptions: Observability signals are converted into structured descriptions that preserve anomaly rankings, failure-indicative logs, high-latency spans, and call-path statistics.

Multi-Agent Collaboration: An intent interpreter, orchestrator, and task-specific agents cooperate through profiles, cross-review, and root-cause reports.

Self-Evolution Mechanism: OpsAgent combines PPO-based internal optimization with reflection-based reusable knowledge construction and retrieval.

04 FOLLOW THE EVIDENCE

Case Walkthrough

From Observability Signals to an Actionable Diagnosis

Follow an order-service failure through evidence preparation, specialist diagnosis, cross-review, and a report engineers can act on. Then see how the framework learns from completed incidents.

Turn the ticket into a time window and diagnostic tasks

Incident ticket

The order-processing application has repeated request failures between 09:10 and 09:20 on January 18, 2026.
The ticket asks for the root cause and the affected component.

Diagnostic task

The intent interpreter extracts the time window and requested tasks, including failure triage and root-cause localization.
The orchestrator coordinates task-specific agents using evidence from that window.

One request, coordinated diagnostic roles

OpsAgent separates understanding the request from executing the diagnosis. Agent profiles define each role's task and inputs, keeping the final answers tied to the incident the user asked about.

Keep each modality's useful details in a shared text format

Structured descriptions

Metric: disk utilization on db-03 rises sharply at 09:12, with a deviation of about 7.5 sigma.
Log: db-03 reports 'No space left on device' at 09:13:08.
Trace: calls from order-service to db-03 show a maximum latency of 6,421 ms.

How the processor builds this evidence

Metrics retain abnormal times, deviation scores, and pattern descriptions; logs are filtered and ranked for diagnostic relevance.
Traces retain high-latency spans and call-path information, linking slow requests to the affected components.

Prepare evidence before asking the LLM to reason

The training-free processor turns heterogeneous telemetry into compact descriptions without training a new processor for each system. In this case, storage errors explain the disk anomaly, while trace evidence connects it to the order service.

Ask specialists different questions about the same incident

Task-specific findings

Anomaly Sentinel identifies the disk spike at 09:12.
Failure Diagnoser identifies disk space exhaustion.
Root Detective identifies db-03 as the root-cause instance.

Answers with supporting evidence

Each agent returns its task result and the reasoning evidence behind it, rather than only a label.
The orchestrator brings together the time, failure type, and instance for cross-review.

Divide responsibility, not the incident context

The agents specialize in AD, FT, and RCL but reason over the same structured incident evidence. Their separate answers can therefore be checked against one another before being accepted.

Check whether the specialist answers agree with the evidence

Findings to check

The time finding points to a disk spike on db-03 at 09:12.
The type and component findings identify disk exhaustion on the same instance.

Review and refinement

Cross-review lets agents inspect the other findings and provide feedback before the final answers are assembled.
In the illustrated case, the disk anomaly, storage errors, and slow calls support one consistent diagnosis.

Review the reasoning, not just the final labels

This stage creates a chance to challenge unsupported conclusions and refine answers using another agent's evidence. The paper's case does not show an initial disagreement, so no invented correction is added here.

Give engineers a diagnosis they can inspect and act on

Evidence supporting the report

Abnormal disk utilization is concentrated on db-03.
Storage errors and high call latency point to the same instance.

Root-cause report

Cause: disk space exhaustion. Component: db-03. The report includes the supporting observations.
Engineers remove obsolete files and expand disk space; the figure shows the incident being mitigated.

Keep diagnosis separate from production changes

The report connects each conclusion to evidence so on-call engineers can check the diagnosis and choose a response. The repair in this case is carried out by engineers, not autonomously by the agents.

Improve the model and retain reusable experience

Internal learning with PPO

Offline training updates the agents using a reward that combines answer accuracy with reasoning quality.
Reasoning is assessed for consistency, clarity, relevance, and rationality.

External learning with reflection

Reflection turns incident experience into reusable knowledge stored in task-specific knowledge bases.
Similar symptoms can retrieve relevant knowledge for a later diagnosis; related experience is aggregated rather than only appended as raw reports.

Two ways to carry learning into the next incident

PPO changes model parameters, while reflection and retrieval supply external experience at inference time. These are the paper's learning mechanisms; the case figure does not establish a particular reward or improvement from this single incident.

Step 1 of 6

Figure 6 supplies the incident example. Sections 3.2-3.4 explain the processing, review, and learning mechanisms; the learning step is not a measured training run for this individual case. View original figure

05 EVALUATION

Experimental Results

Results at a Glance

OPENRCA results with Qwen2.5-14B-Instruct-1M. Correct and Partial are the paper's diagnosis scores, averaged across the evaluated systems.

Diagnosis quality

16.54% / 41.35%

Average Correct / Partial

The full OpsAgent configuration combines data processing, cross-review, and self-evolution.

Table 1

Diagnosis time

71.18 s

Average time per incident

Measured for the same seed model and configuration. Runtime depends on the model and deployment setup.

Table 1

Self-evolution

8.27% to 16.54%

Average Correct

Correct doubles as the share of cases used for self-evolution increases from 0% to 60%.

Table 3

Table 1: Comparison with baselines across seed LLMs

Overall Performance: OpsAgent achieves state-of-the-art diagnosis quality across seed LLMs on OPENRCA while keeping inference cost practical.

Table 2: Ablation study on key components

Ablation Study: Removing the data processor, cross-review, reflection, or PPO weakens diagnosis quality, showing that each component contributes to the full system.

Table 3: Self-evolution capability under different training budgets

Self-Evolution Budget: Increasing the proportion of self-evolution training cases improves average Correct and Partial scores.

Deployment Case: The case study illustrates how OpsAgent turns incident tickets and observability evidence into actionable root-cause reports for OCEs.

Figure 7: Interpretability assessment in deployment

Interpretability Assessment: Frontline OCEs rate OpsAgent highly on consistency, clarity, relevance, and rationality.

06 START FROM THE CODE

Run the Code

Set up one OPENRCA system first. The research script uses local model and data paths that must be configured for your machine.

Before running

Use Python 3.10 or newer, install requirements.txt, and choose a PyTorch build that matches your hardware.
Download the telemetry data from the README and configure the model, dataset, and query paths. Also set the intent interpreter's model path.
Prepare the reflection files, replace the query path in main.py, create the output directories, and review the case-selection loop.

Run after setup

# From the configured repository directory
python main.py

Output

The script saves diagnostic answers and supporting reasoning to CSV files for the selected incident queries.

Repository entry point; local data and model setup are required.

Setup & data Entry script

07 FREQUENTLY ASKED

Frequently Asked Questions

What is OpsAgent?

OpsAgent is a lightweight, self-evolving multi-agent system for incident management in microservice systems, published in the ASE 2026 Industry Showcase track. It turns heterogeneous observability data into structured textual descriptions through a training-free data processor, and coordinates specialized agents that produce transparent, auditable root-cause reports for on-call engineers.

What problem does it solve, and how?

Manual incident management is labor-intensive and error-prone because metrics, logs, and traces are massive and heterogeneous. OpsAgent first prepares evidence with a training-free processor — no system-specific model training — then splits diagnosis among task-specific agents for anomaly detection, failure triage, and root-cause localization, whose answers are cross-reviewed before reporting. Separately from this fixed pipeline, dual self-evolution improves the system over time: internal PPO fine-tuning of the agents and external reflection that accumulates reusable incident knowledge.

What was evaluated, and what were the results?

On the OPENRCA benchmark with Qwen2.5-14B-Instruct-1M as the seed model, the full system averages 16.54% Correct and 41.35% Partial diagnoses (Table 1), with 71.18 seconds per incident measured for that configuration. Increasing the share of incidents used for self-evolution training from 0% to 60% raises average Correct from 8.27% to 16.54% (Table 3). Runtime depends on the model and deployment setup and is not a cross-hardware guarantee.

How can I use it, and what are its limitations?

The research code is at github.com/Johnn13/OpsAgent; the repository has no license file yet, so check with the maintainers before reuse. This OpsAgent is an academic incident-diagnosis system and is unrelated to Google Cloud's Ops Agent telemetry collector. In the paper's deployment case, OpsAgent produces the diagnosis and engineers decide on and perform the remediation.

08 CITE THIS WORK

BibTeX

@article{luo2025opsagent,
  title={OpsAgent: An Evolving Multi-agent System for Incident Management in Microservices},
  author={Luo, Yu and Jiang, Jiamin and Feng, Jingfei and Tao, Lei and Zhang, Qingliang and Wen, Xidao and Sun, Yongqian and Zhang, Shenglin and Pei, Dan},
  journal={arXiv preprint arXiv:2510.24145},
  year={2025}
}