TrioXpert: An Automated Incident Management Framework for Microservice System

Yongqian Sun; Yu Luo; Xidao Wen; Yuan Yuan; Xiaohui Nie; Shenglin Zhang; Tong Liu; Xi Luo

doi:10.1109/ASE63991.2025.00267

01 THE PAPER

Abstract

Automated incident management plays a pivotal role in large-scale microservice systems. However, many existing methods rely solely on single-modal data (e.g., metrics, logs, and traces) and struggle to simultaneously address multiple downstream tasks, including anomaly detection (AD), failure triage (FT), and root cause localization (RCL). Moreover, the lack of clear reasoning evidence in current techniques often leads to insufficient interpretability. To address these limitations, we propose TrioXpert, an end-to-end incident management framework capable of fully leveraging multimodal data. TrioXpert designs three independent data processing pipelines based on the inherent characteristics of different modalities, comprehensively characterizing the operational status of microservice systems from both numerical and textual dimensions. It employs a collaborative reasoning mechanism using large language models (LLMs) to simultaneously handle multiple tasks while providing clear reasoning evidence to ensure strong interpretability. We conducted extensive evaluations on two microservice system datasets, and the experimental results demonstrate that TrioXpert achieves outstanding performance in AD (improving by 4.7% to 57.7%), FT (improving by 2.1% to 40.6%), and RCL (improving by 1.6% to 163.1%) tasks. TrioXpert has also been deployed in Lenovo's production environment, demonstrating substantial gains in diagnostic efficiency and accuracy.

Project Overview

Multimodal Evidence

TrioXpert treats metrics, logs, and traces according to their own data characteristics, using numerical and textual views to describe the system state more completely.

Multi-Task Management

The framework covers anomaly detection, failure triage, and root cause localization in one end-to-end incident management workflow.

Interpretable Reasoning

LLM-based experts cooperate over structured evidence, producing reasoning traces that help OCEs inspect why a diagnosis was made.

02 DESIGN LESSONS

Key Takeaways

Takeaway 1

Every modality matters, but not every record helps.

Metrics, logs, and traces reflect distinct aspects of system behavior, and all of them carry valuable diagnostic signals. The key difficulty is that logs and traces are often dominated by redundant entries, so useful textual evidence must be filtered before reasoning.

Takeaway 2

A single LLM is not enough for reliable incident management.

Directly feeding complex multimodal incident data into one monolithic LLM can produce fluent but unreliable reports, with hallucinated steps, opaque reasoning, and context-window loss. TrioXpert therefore uses collaborative experts and structured prompts to make reasoning more stable and interpretable.

03 THE FRAMEWORK

Method

04 FOLLOW THE EVIDENCE

Case Walkthrough

Two Views of the System, One Evidence-Backed Diagnosis

The key is not simply to ask three task questions. TrioXpert preserves numerical patterns and text meaning in separate pipelines, then brings their findings together through specialized experts.

Represent different data types in the form they need

Numerical inputs

Metrics form a matrix indexed by time, service instance, and feature channel.
Trace call relationships form a service topology graph aligned with the metric sampling times.

Textual inputs

Logs are filtered first by incident-related keywords and then by LLM-based relevance checks.
Traces retain spans above the P95 latency threshold for their call type, along with ancestors needed to preserve complete call paths.

Preserve both the signal and its context

A time series carries statistical patterns; an error log carries meaning; a call path shows dependencies. TrioXpert keeps these properties instead of flattening all three modalities into one undifferentiated input.

Build a statistical starting point for diagnosis

Time series and topology

A Transformer, GRU, and GraphSAGE feature processor models channel, time, and service-call relationships.
Prediction errors measure how observed behavior differs from the learned normal patterns.

Preliminary task answers

Thresholding supports AD, clustering supports FT, and deviation-based ranking supports RCL.
The Numerical Expert checks these preliminary answers and adds an explanation for the selected tasks.

Ground language-based reasoning in numerical analysis

The feature processor builds on ART rather than being presented as a new encoder. TrioXpert uses its compact diagnostic results as one evidence source, then combines them with the information preserved by the textual pipelines.

Recover diagnostic meaning that a numerical score cannot express

Filtered logs and traces

The Log Summarizer describes incident-related behavior and possible causes using the selected logs.
The Trace Summarizer describes abnormal call behavior while retaining its service context.

Textual Expert findings

The Textual Expert reasons over both summaries and returns task answers with supporting evidence.
Log semantics can explain the kind of failure, while traces help connect that failure to service interactions.

Summarize to preserve meaning, not just shorten the input

The prompts ask summarizers to stay grounded in the supplied records and retain important details. Separating log and trace summarization reduces the amount of raw telemetry passed to the final reasoning stage.

Pass findings and evidence to an Incident Expert

Two independent views

Numerical Expert: task answers plus numerical reasoning evidence.
Textual Expert: task answers plus evidence from log and trace summaries.

Joint diagnosis

The Incident Expert receives both pairs of answers and evidence, not just two candidate labels.
It produces final results and explanations for the requested AD, FT, and RCL tasks.

The three experts specialize by evidence role

Numerical Expert, Textual Expert, and Incident Expert are not three independent agents assigned one downstream task each. The first two analyze complementary views; the third reconciles their findings for the selected tasks.

Use an explicit policy when the evidence views disagree

Possible disagreement

Different modalities can point to different failure types or instances.
The paper specifies how the experts should weigh conflicting sources instead of relying on a majority vote.

Evidence-weighting policy

Within the Textual Expert, logs receive greater weight than traces when the sources conflict.
At final aggregation, the Incident Expert gives greater weight to the numerical perspective, following the paper's policy.

Resolve conflicts through instructed reasoning

These priorities are expressed in prompts, not a fixed numerical averaging formula. Role, goal, constraints, instructions, and examples guide the answer; the prompts prohibit unsupported facts and ask for confidence when uncertain. This step explains the policy, not a claimed disagreement in the database example.

Connect the method to the incident-management result

Outputs shown in the database example

AD: February 1, 2025, from 19:00:00 to 19:05:30.
FT: Database Connection Timeout. RCL: Dbservice-3.

From diagnosis to human action

The example routes the issue to the database team, which increases the connection pool limit to 100 and restarts the instance.
The method returns diagnostic answers and their evidence; the repair shown in the figure is a human action.

One system representation supports several tasks

The same numerical and textual views can serve AD, FT, and RCL without rebuilding separate reasoning systems for each task. The final example illustrates these outputs; it does not provide raw records from which to reconstruct exact intermediate expert answers.

Step 1 of 6

Section III and Figures 2-4 supply the method flow; the incident-management figure supplies the final database example. Intermediate scores, log entries, and expert disagreements are not fabricated for that example. View original figure

05 EVALUATION

Experimental Results

Results at a Glance

Results from Table II. Each pair shows D1 first and D2 second; F1 and Avg@5 are reported on a 0-1 scale.

Anomaly detection

0.924 / 0.909

F1 on D1 / D2

Identify when the system is behaving abnormally.

Table II

Failure triage

0.807 / 0.767

F1 on D1 / D2

Identify the failure type to guide the investigation.

Table II

Root-cause localization

0.773 / 0.750

Avg@5 on D1 / D2

Rank the instances that may be responsible for the incident.

Table II

Table 1: Detailed information of datasets

Datasets: TrioXpert is evaluated on two microservice-system datasets containing multimodal traces, logs, and metrics across multiple failure types.

Table 2: Performance comparison on AD, FT, RCL, and time

Overall Performance: TrioXpert consistently improves AD, FT, and RCL performance over multimodal and task-specific baselines on both datasets.

Ablation Study: Removing modality coverage, collaborative reasoning, coordination, or hallucination mitigation weakens performance, confirming the need for the full design.

06 WHAT WE LEARNED

Key Insights

TrioXpert shows that multimodal fusion in AIOps cannot treat all observability data as the same kind of signal. Metrics should be understood through their temporal and statistical behavior, while logs and traces should be used for their rich semantic evidence. The important move is not simply to put more data into a model, but to preserve the diagnostic value that each modality naturally carries.

The paper also provides an early demonstration that multi-agent collaboration can be effective for AIOps root cause analysis. By separating numerical and textual expertise, coordinating expert outputs, and forcing evidence-backed reasoning, TrioXpert improves interpretability as well as task performance. This became a useful starting point for later work on more structured and reliable agentic diagnosis.

07 START FROM THE CODE

Run the Code

Start with the public D1 dataset and the root-cause localization example. D2 is not publicly released.

Before running

Follow the README's Python 3.10.16 environment and listed PyTorch, Transformers, Accelerate, and DGL versions.
Download and unzip D1 using the link in the README.
Set data_path and model_path for your local data and language model before running the RCL example.
Review the GPU selection, training settings, task mode, and service-dictionary path in run_RCL.py. The script includes model training, not just inference.

Run after setup

# From the configured repository directory
python run_RCL.py

Output

The RCL example evaluates root-cause localization. The repository also provides separate AD, FT, and textual-processing scripts.

Repository entry point; local data and model setup are required.

Setup & data Entry script

08 FREQUENTLY ASKED

Frequently Asked Questions

What is TrioXpert?

TrioXpert is an end-to-end automated incident management framework for microservice systems, published at ASE 2025 (DOI 10.1109/ASE63991.2025.00267). It combines multimodal data processing with collaborative LLM reasoning to handle three downstream tasks in one system — anomaly detection (AD), failure triage (FT), and root cause localization (RCL) — with interpretable reasoning evidence.

What problem does it solve, and how?

Many existing methods rely on single-modal data and handle the three tasks separately, while feeding everything into one monolithic LLM loses diagnostic value. TrioXpert instead matches processing to each modality: metrics and traces are modeled numerically by a feature processor, while logs and traces are also summarized textually. A Numerical Expert and a Textual Expert analyze these complementary views, and an Incident Expert reconciles them under an explicit evidence policy — logs outweigh traces when the two conflict, and the numerical view carries greater weight at final aggregation.

What was evaluated, and what were the results?

TrioXpert was evaluated on two microservice-system datasets, D1 (public) and D2 (not released). On Table II it reaches F1 0.924 / 0.909 for AD, F1 0.807 / 0.767 for FT, and Avg@5 0.773 / 0.750 for RCL on D1 / D2 respectively. These are separate task metrics and cannot be merged into a single accuracy figure.

How can I use it, and what are its limitations?

The code is open at github.com/luoyu100/TrioXpert with a root-cause localization example for the public D1 dataset (Python 3.10 environment). The repository currently has no license file, so clarify reuse terms before building on it, and D2 is not publicly available.

09 CITE THIS WORK

BibTeX

@inproceedings{sun2025trioxpert,
  title={TrioXpert: An Automated Incident Management Framework for Microservice System},
  author={Sun, Yongqian and Luo, Yu and Wen, Xidao and Yuan, Yuan and Nie, Xiaohui and Zhang, Shenglin and Liu, Tong and Luo, Xi},
  booktitle={2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)},
  pages={3239--3250},
  year={2025},
  organization={IEEE}
}