A CONTROLLED EMPIRICAL STUDY

ToolRCA

Which tools help an RCA agent,
and when?

KEY FINDINGS & DESIGN RULES

More tools can lower accuracy.

Qwen3.7-Plus: 82.8% → 78.0% localization, with 4.17× the tokens.

Diagnostic tools only → full toolset.

Design rule: Choose the diagnostic target before adding tools.
The same help can hurt another model.

Qwen3-8B: -13.9 pp. Qwen3.7-Plus: +14.0 pp.

Add diagnostic tools to metric access + evidence analysis.

Design rule: Match tools to the agent's ability to use them.
A useful tool may not transfer.

Online Boutique: +6.9 pp. Train Ticket: -11.0 pp.

Qwen3.7-Plus · add evidence analysis to diagnostic tools.

Design rule: Revalidate the configuration in your target system.

Localization = first-choice root cause accuracy. pp = percentage points. These design rules are study-derived, not separately validated.

Rethinking Tool Design for Agentic RCA: A Controlled Empirical Study

RCA tool design lacks controlled evidence. We compare what tools analyze, which combinations help, and how agents use the results across models and systems.

Paper PDF Why reject the right cause?

375 RE1 failure cases · 3 microservice systems. Findings depend on the tested models and configurations. Study coverage

Yu Luo¹, Rongchen Gao¹, Zhenhui Zhou¹, Changchang Liu¹, Yuliang You¹, Yongqian Sun^1,*, Shenglin Zhang¹, Qiuai Fu², Shijie Wang², Dan Pei³

¹ Nankai University
² Huawei Technologies Co., Ltd.
³ Tsinghua University

* Corresponding author: Yongqian Sun

01 / A REPORTED DECISION FAILURE

The tool found the cause.
The agent changed its mind.

The mistake is not seeing an anomaly. It is treating that anomaly as a cause.

Follow the agent's available observations, its interpretation, and a separate reasoning check. The benchmark label is revealed only after the decision. Checks explain the inference; they are not verbatim agent thoughts.

Two different kinds of help: evidence-analysis tools (L2) identify unusual metrics; diagnostic tools (L3) rank possible root causes. An anomaly ranking is not a cause ranking.

01 / Incident

Two services. An unresolved diagnosis.

AVAILABLE OBSERVATIONS

A Train Ticket investigation

The reported investigation examines ts-auth-service and the payment service. The true faulty service and fault type are not shown to the agent here.

Agent interpretation · summary: The diagnosis is open. No candidate has yet been ranked in this summary.
Reasoning check · reader analysis: Keep the agent's observations separate from the benchmark label, which is revealed after the decision.

CANDIDATE & DECISION

ts-auth-service

Not yet ranked

Payment service

Not yet ranked

Investigation open

A summary of the reported case, not a raw tool transcript. The reasoning checks are explanatory commentary on this page.

Frame 1 / 6

THE INFERENCE TO CHECK

No CPU or memory anomaly does not rule out a delay fault.

A prominent anomaly elsewhere is not enough to replace a candidate. A sound revision needs evidence that supports the new cause and distinguishes it from the old explanation. This does not mean always trusting L3.

Complete case summary and source

Two services. An unresolved diagnosis.
Available observations
The reported investigation examines ts-auth-service and the payment service. The true faulty service and fault type are not shown to the agent here.
Agent interpretation · summary
The diagnosis is open. No candidate has yet been ranked in this summary.
Reasoning check · reader analysis
Keep the agent's observations separate from the benchmark label, which is revealed after the decision.
L3 ranks auth first.
Available observations
The diagnostic tool ranks ts-auth-service as the top root cause candidate.
Agent interpretation · summary
Auth is the current leading recommendation; payment is not the top L3 candidate.
Reasoning check · reader analysis
A candidate ranking is a hypothesis to assess, not ground truth. Any revision still needs relevant evidence.
L2 highlights payment memory.
Available observations
Candidate filtering ranks a memory metric of the payment service highest.
Agent interpretation · summary
The agent now has a prominent payment anomaly alongside the auth recommendation.
Reasoning check · reader analysis
An anomaly ranking answers where a signal is unusual. It does not establish which service caused the incident.
Missing symptoms become a reason to reject.
Available observations
The agent finds no CPU or memory anomaly in auth, while observing its latency.
Agent interpretation · summary
The agent treats auth latency as a propagated symptom and shifts suspicion toward payment.
Reasoning check · reader analysis
No CPU or memory anomaly does not rule out a delay fault. Nor does it show that auth latency propagated from payment.
The agent switches to payment.
Available observations
Payment has the highest-ranked memory anomaly; auth lacks a CPU or memory anomaly. No new L3 ranking is reported here.
Agent interpretation · summary
The agent blames payment instead of ts-auth-service.
Reasoning check · reader analysis
The revision combines two observations, but neither establishes payment as the cause or rules out auth. A worse-looking metric is not enough.
The root cause was auth, not payment.
Benchmark label · hindsight
Root cause entity: ts-auth-service. Injected failure type: delay. This label is available to the reader for the audit, not to the diagnosing agent.
Decision versus ground truth · hindsight
The agent's payment diagnosis is incorrect. In this case, the original L3 recommendation matched the true faulty service.
Reasoning check · reader analysis
Do not conclude that L3 should always be trusted. Require fault-relevant evidence that supports a new candidate and distinguishes it from the old one.

Section 5.1 reports this Train Ticket case as a summarized decision failure. The page does not reconstruct unreported API calls, timestamps, numerical scores, or a separate baseline outcome. The reasoning checks explain the inference rather than quote an unavailable raw transcript.

Where does a diagnosis fail?

01 / PROCESS

No valid finish.

The agent exhausts the budget or calls an interface without meeting its input requirements.

02 / COVERAGE

The cause never enters.

The investigation does not consider the true root cause.

03 / DECISION

Considered, not chosen.

The agent rejects the correct candidate or confirms an incorrect one.

04 / NAMING

Right entity, wrong type.

A prominent symptom is mistaken for the injected fault.

Section 5.1. For Qwen3.7-Plus with L2 or L3, the true cause was considered in 84–95% of trajectories that nevertheless ended with an incorrect localization. That percentage is conditional on localization failures.

Does adding more tools improve the final diagnosis?

02 / COMPOSITION · RQ1

Does more help?

A larger toolset can improve fault typing while hurting localization.

L1 Metric accessL2 Evidence analysisL3 Diagnostic methodsFull tool spectrum

REPRESENTATIVE COMPARISON / QWEN3.7-PLUS · L3 → L1+L2+L3

First-choice root cause accuracy falls from 82.8% to 78.0%; fault type accuracy rises from 45.6% to 49.0%; total tokens rise to 4.17×.

Design implication: decide which diagnostic target matters before adding tools. The same addition can help one target, hurt another, and cost more.

Measured across the three RE1 systems with Qwen3.7-Plus, not a universal ranking of toolsets.

Table 3. Accuracy uses a fixed 0–100% scale. pp = percentage points; token ratios compare total input + output tokens. The full toolset improves TA over L3 but lowers top-1 localization in this setting.

WHAT CHANGES AFTER THE TOOL RETURNS?

The recommendation is stable. The decision is not.

The first L3 candidate is unchanged in 99.9% of paired trajectories that both obtain a ranking. Additional interfaces change whether the agent keeps that candidate. With L2+L3, correct-to-wrong overrides rise to 9.3% of trajectories, while wrong-to-correct revisions are 3.8%.

Inspect a reported decision failure

Complete results · Table 3

Qwen3.7-Plus · mean efficiency per case
Configuration	AC@1 (%)	AC@3 (%)	Avg@5 (%)	TA (%)	Tokens (K)	Latency (s)	Cost (USD)
L0	85.4	93.6	92.7	41.4	50.4	88.6	0.019
L1	52.7	70.0	69.1	24.3	53.7	51.4	0.018
L2	68.8	92.3	87.5	44.4	142.9	52.8	0.044
L3	82.8	96.2	93.7	45.6	22.8	35.8	0.008
L1+L2	64.0	88.4	84.4	47.3	111.6	57.1	0.035
L1+L3	83.6	94.3	92.5	47.6	45.0	39.9	0.015
L2+L3	78.2	95.6	92.3	48.6	79.0	40.0	0.025
L1+L2+L3	78.0	94.8	92.1	49.0	95.1	48.8	0.03

Original composition, efficiency, and candidate-decision figures

Original configuration comparisons for localization and type identification; the type accuracy axis is broken. — Figure 3 · Tool composition. Full resolution

Original localization accuracy versus token and latency plots. — Figure 4 · Effectiveness and efficiency. Full resolution

Original correctness of L3 candidates and final diagnoses, including trajectories without a ranking. — Figure 5 · Candidate decisions. Full resolution

What changes when another model receives the same tools?

03 / MODEL CAPABILITY · RQ2

Available is not
the same as useful.

A tool must be selected, called successfully, and assessed correctly.

REPRESENTATIVE COMPARISON / ADD L3 TO L1+L2

Qwen3-8B: -13.9 pp
Qwen3-32B: +4.1 pp
Qwen3-235B-A22B: +5.3 pp
Qwen3.7-Plus: +14.0 pp

The difference is not just whether L3 is present. Qwen3-8B invokes it in only 0.8% of trajectories. Other models lose useful recommendations during calls or final assessment.

Changes in first-choice root cause accuracy across the tested configurations. These four models are not a calibrated capability scale; parameter count alone does not explain the differences.

Root cause localization

AC@1 (%) · four shared configurations

Failure type identification

TA (%) · four shared configurations

L3 invocation rate0.8%

At least one L3 call under L1+L2+L3; not the share receiving a ranking. Figure 7.

L3 is available, but almost never called. The larger action space accompanies longer trajectories and more invalid actions; this is not evidence that incorrect L3 outputs mislead the model.

Qwen3-8B · complete results

Table 4 · Qwen3-8B
Configuration	AC@1 (%)	AC@3 (%)	Avg@5 (%)	TA (%)	Completion (%)	Tokens (K)	Latency (s)	Cost (USD)
L0	14.8	32.7	31.4	17.5	77.7	80.9	83.4	0.007
L1	29.3	48.4	47.1	24.4	96.4	63.0	41.8	0.005
L1+L2	59.6	77.2	74.2	28.7	98.8	90.7	46.5	0.007
L1+L2+L3	45.7	64.2	63.2	25.7	97.6	122.8	53.7	0.009

The model calls L3 but overrides roughly one third of its correct recommendations. Tool availability, correct use, and sound assessment are separate requirements.

Qwen3-32B · complete results

Table 4 · Qwen3-32B
Configuration	AC@1 (%)	AC@3 (%)	Avg@5 (%)	TA (%)	Completion (%)	Tokens (K)	Latency (s)	Cost (USD)
L0	39.6	57.6	56.1	24.4	94.0	53.5	75.3	0.018
L1	45.2	68.3	64.4	23.4	93.1	68.6	43.2	0.022
L1+L2	49.9	67.3	64.5	28.3	86.8	160.5	62.0	0.049
L1+L2+L3	54.0	72.4	69.9	31.1	87.7	193.8	66.2	0.059

Most trajectories invoke L3, but only about half obtain a ranking. Narrowing the incident window can discard useful evidence; invoking a tool does not guarantee a useful result.

Qwen3-235B-A22B · complete results

Table 4 · Qwen3-235B-A22B
Configuration	AC@1 (%)	AC@3 (%)	Avg@5 (%)	TA (%)	Completion (%)	Tokens (K)	Latency (s)	Cost (USD)
L0	35.6	58.6	56.0	23.5	98.8	48.9	63.4	0.017
L1	45.0	66.8	63.7	24.3	97.5	47.2	40.5	0.015
L1+L2	63.9	83.8	80.7	39.3	98.8	78.2	38.8	0.024
L1+L2+L3	69.2	86.0	83.1	41.2	99.4	87.3	37.6	0.027

This model performs its strongest top-1 localization with its own Python analysis. Additional tools can improve failure typing, while candidate revisions remain a source of localization errors.

Qwen3.7-Plus · complete results

Table 4 · Qwen3.7-Plus
Configuration	AC@1 (%)	AC@3 (%)	Avg@5 (%)	TA (%)	Completion (%)	Tokens (K)	Latency (s)	Cost (USD)
L0	85.4	93.6	92.7	41.4	100.0	50.4	88.6	0.019
L1	52.7	70.0	69.1	24.3	100.0	53.7	51.4	0.018
L1+L2	64.0	88.4	84.4	47.3	100.0	111.6	57.1	0.035
L1+L2+L3	78.0	94.8	92.1	49.0	100.0	95.1	48.8	0.03

01 / SELECT

Delegate the right analysis.

Does the agent choose the interface that can help?

02 / CALL

Supply valid inputs.

Does the call return relevant evidence or a ranking?

03 / ASSESS

Use the result responsibly.

Does the agent keep or revise a candidate for a sound reason?

Findings 5–7. The model categories are not a calibrated capability scale. Only Qwen3.7-Plus was tested under all eight configurations; the plots here show the four common configurations.

Original model effects and L3 usage

Original changes after adding L2 and L3, and L3 usage rates across four models. — Figure 7 · Model effects and L3 use. Full resolution

Will the same configuration help in another system?

04 / SYSTEM ENVIRONMENT · RQ3

Same addition.
Different consequences.

A tool addition can help one system and hurt another.

REPRESENTATIVE COMPARISON / QWEN3.7-PLUS · L3 → L2+L3

Online Boutique: +6.9 pp
Sock Shop: -9.6 pp
Train Ticket: -11.0 pp

Observed: first-choice root cause accuracy improves in Online Boutique but falls in the other two systems. Trace analysis points to different outcomes when agents switch to an anomalous entity.

Not established: that service count causes the reversal. Topology, communication, and metric composition also differ. Revalidate the configuration in the target environment.

Add one tool levelQwen3.7-Plus · Table 5

Online Boutique

Localization +6.9 pp

Failure typing +4.8 pp

Sock Shop

Localization −9.6 pp

Failure typing +1.6 pp

Train Ticket

Localization −11.0 pp

Failure typing +2.7 pp

Changes are percentage points, not relative percentages. All nine possible one-level additions among the seven structured configurations are available. Environment properties vary jointly; the results do not establish a causal effect of service count alone.

What do candidate switches reveal?

Online Boutique

When an agent switches from an L3 candidate to a different entity ranked highest by L2, the switch is correct in 79% of those cases.

Sock Shop

In the same conditional analysis, switches to a different L2-ranked entity are correct in only 5% of cases.

Train Ticket

Switches to a different L2-ranked entity are correct in only 5% of cases. Service count alone does not explain this contrast.

The percentages above are conditional on an agent switching from an L3 candidate to a different entity ranked most anomalous by L2. They are not overall tool accuracy. The manuscript does not state the corresponding sample counts; these proportions alone do not establish their uncertainty or a causal explanation.

Complete cross-system results · Table 5

Qwen3.7-Plus · all values in percent
Configuration	Online Boutique				Sock Shop				Train Ticket
Configuration	AC@1	AC@3	Avg@5	TA	AC@1	AC@3	Avg@5	TA	AC@1	AC@3	Avg@5	TA
L0	86.4	95.2	94.0	45.9	90.9	96.3	95.9	41.6	78.9	89.3	88.2	36.8
L1	56.3	76.0	73.7	23.2	73.6	87.5	86.5	28.8	28.3	46.7	47.1	20.8
L2	84.5	95.2	93.2	50.1	67.5	98.4	92.0	40.8	54.4	83.2	77.2	42.1
L3	81.1	96.0	93.5	49.6	90.1	100.0	97.8	39.5	77.1	92.5	89.8	47.7
L1+L2	78.7	91.5	89.2	48.8	61.9	96.8	90.1	44.0	51.5	76.8	73.8	49.1
L1+L3	82.7	93.6	92.0	52.3	89.1	98.1	96.0	40.0	79.2	91.2	89.4	50.4
L2+L3	88.0	97.1	95.6	54.4	80.5	99.2	95.0	41.1	66.1	90.4	86.2	50.4
L1+L2+L3	88.0	96.3	95.2	52.8	78.7	98.9	94.8	42.7	67.5	89.3	86.1	51.5

Original cross-system figure

Original per-system localization comparisons and marginal changes after adding tool levels. — Figure 9 · Cross-system effects. Full resolution

What should these findings change about tool design?

05 / FROM EVIDENCE TO DESIGN

Principles & guidelines.

Choose tools for the decision they support, not the size of the toolset.

Practical checks derived from this study, not separately validated improvements. Use the linked evidence to decide what to test next.

01
Separate the two diagnostic targets.
When
A tool addition helps one accuracy metric but hurts another.
Check
Compare first-choice root cause accuracy, fault type accuracy, and cost separately.
Try
Choose the target you need to improve before selecting a configuration; report the tradeoff rather than a single winner.
Accuracy and cost tradeoff
02
Match tools to demonstrated capability.
When
Tools are available, but the agent gains little from them.
Check
Measure selection, valid calls, useful returned results, and whether correct recommendations survive the final decision.
Try
Simplify the failing interface or its instructions; evaluate result assessment separately from tool invocation.
Selection, calls, assessment
03
Make candidate revisions accountable.
When
New observations cause the agent to replace a diagnostic candidate.
Check
Record the old candidate, new candidate, and evidence supporting the change or contradicting the old explanation.
Try
Ask for fault-relevant evidence before a switch; do not equate the most anomalous metric with the root cause.
Reported decision failure
04
Distinguish the fault from its symptoms.
When
The entity is right, but the predicted fault type is wrong.
Check
Look for evidence that separates competing fault types, rather than only the most prominent CPU or latency symptom.
Try
Expose fault-specific observations and ask the agent to compare alternative explanations.
Right entity, wrong type
05
Revalidate when the system changes.
When
A configuration is moved to a different microservice environment.
Check
Re-evaluate accuracy and whether anomaly rankings align with labeled root causes in that environment.
Try
Validate the configuration locally before transferring its ranking from another system.
Cross-system reversal
06
Repair the stage that actually fails.
When
Diagnosis fails, despite adding more tools or evidence.
Check
Classify the failure: invalid execution, missing cause, wrong candidate decision, or wrong fault type.
Try
Improve interface reliability, evidence coverage, candidate comparison, or fault-specific evidence according to that failure.
Four failure locations

These are design implications derived from the empirical findings and discussion, not separately evaluated interventions or universal rankings. Their applicability beyond the tested models and metric-based benchmark requires validation.

What responsibilities do these tools take on?

06 / BACKGROUND · ANALYTICAL RESPONSIBILITY

The tool spectrum.

The levels differ in what tools return, and what the agent must still decide.

SAME INCIDENT DATA

Metrics
Incident context
Service topology

Agent-written analysis

Python interface

Tool work

Run agent-written Python

Returned to the agent

Results of the agent's own programs

Agent decision

Choose, implement, and interpret the analysis.

Metric access

6 structured interfaces

Tool work

Retrieve and summarize metrics

Returned to the agent

Samples, summaries, and local topology

Agent decision

Find unusual behavior and reason about its cause.

Evidence analysis

12 structured interfaces

Tool work

Detect and connect unusual signals

Returned to the agent

Anomalies, change points, and evidence links

Agent decision

Connect the evidence. Distinguish symptoms from causes.

Diagnostic methods

6 structured interfaces

Tool work

Rank and explain candidates

Returned to the agent

Candidate rankings and supporting explanations

Agent decision

Assess the recommendation, then keep or revise it.

The agent makes the final diagnosis at every level.Independent interfaces, not a required pipeline. Higher abstraction does not guarantee a better diagnosis.

Agent-written analysis

Python interface

THE TOOL

Execute Python over the same case metrics, incident context, and service topology.

THE AGENT

Choose, implement, and interpret the entire analysis. There are no predefined RCA procedures.

Returned to the agentResults of the agent's own programs

SCHEMATIC EXAMPLE / L0

Agent-written Python · one analysis step

A ticketing request becomes slow. What does each interface give the agent?

latency_ms = [20, 22, 140, 150]
print(sum(latency_ms) / len(latency_ms))
output: 83.0 ms · computed by the agent's code

Even this summary requires agent-written code. The agent implements any further analysis itself; no ready-made RCA procedure is provided.

Schematic outputs, not a recorded benchmark trace. Sample values are illustrative; no measured anomaly score or diagnostic confidence is implied.

Capabilities covered

Select and process incident data
Compute statistics in agent-written code
Implement further analysis
Interpret the resulting observations

Metric access

6 structured interfaces

THE TOOL

Retrieve metric samples, summary statistics, metric ownership, and local service dependencies.

THE AGENT

Choose what to inspect, identify unusual behavior, and reason about its cause. These tools do not score anomalies or rank root causes.

Returned to the agentObservations and basic summaries

SCHEMATIC EXAMPLE / L1

Metric query and summary

A ticketing request becomes slow. What does each interface give the agent?

service: auth · metric: request latency
samples (ms): [20, 22, 140, 150]
mean (ms): 83 · maximum (ms): 150

These are measurements and summaries. The interface has not labeled them anomalous or named a root cause.

Schematic outputs, not a recorded benchmark trace. Sample values are illustrative; no measured anomaly score or diagnostic confidence is implied.

Capabilities covered

Metric discovery and retrieval
Service, metric, and time-window queries
Batch metric queries
Mean, maximum, minimum, sum, and percentiles
Metric ownership resolution
Local topology inspection

Evidence analysis

12 structured interfaces

THE TOOL

Analyze time series and return anomaly scores, change points, candidate metrics, temporal features, and evidence graphs.

THE AGENT

Relate evidence across services and decide what it means for diagnosis. The most anomalous metric is not necessarily the root cause.

Returned to the agentAnomaly scores, features, and relationships

SCHEMATIC EXAMPLE / L2

Anomaly and relationship analysis

A ticketing request becomes slow. What does each interface give the agent?

auth latency: change detected near the incident
payment memory: highest-ranked anomalous metric
relationship analysis: related metric evidence

This identifies unusual signals and their relationships. Highest anomaly rank does not mean most likely root cause.

Schematic outputs, not a recorded benchmark trace. Sample values are illustrative; no measured anomaly score or diagnostic confidence is implied.

Capabilities covered

Time-series processing and filtering
Anomaly detection: 3-sigma, IQR, MAD, ESD, SPOT
Change points: CUSUM, PELT, binary segmentation, Page–Hinkley
Temporal feature extraction
Anomalous metric filtering
Relationship analysis and evidence graphs

Diagnostic methods

6 structured interfaces

THE TOOL

Generate candidate rankings, combine diagnostic evidence, and assemble explanations for specified candidates.

THE AGENT

Assess whether to keep or revise the recommendations, then submit the final candidate list and failure type.

Returned to the agentRanked candidates and diagnostic explanations

SCHEMATIC EXAMPLE / L3

Diagnostic ranking and explanation

A ticketing request becomes slow. What does each interface give the agent?

root cause candidates: 1. auth · 2. payment
support: latency evidence and service relationships
agent still submits the final entity and fault type

This is a diagnostic recommendation, not the final answer. The agent must assess the support before keeping or revising it.

Schematic outputs, not a recorded benchmark trace. Sample values are illustrative; no measured anomaly score or diagnostic confidence is implied.

Capabilities covered

Propagation-path analysis
Graph ranking: PageRank, personalized PageRank, random walk with restart
Classic RCA: MonitorRank, CloudRanger, MicroRCA, Microscope, MicroCause, MicroHECL
Fusion of anomaly, ranking, topology, and timing evidence
Supporting evidence chains
Diagnostic explanations

Section 3.2. Counts refer to callable interfaces, not the number of algorithms. L2 and L3 access case data directly and perform internal preparation; no prior L1 call or fixed invocation order is required. Final diagnosis remains the agent's responsibility.

How were the comparisons kept controlled?

07 / BACKGROUND · A CONTROLLED COMPARISON

Change the tools.
Keep the task grounded.

Eight configurations do not mean an eight-by-four model sweep.

Qwen3.7-Plus: all 8 configurations. The other three models: L0, L1, L1+L2, and L1+L2+L3. The 7 structured configurations expose different subsets of L1–L3; L0 is evaluated independently with agent-written Python analysis.

Structured interfacesTools: 24
ConfigurationsConfigs: 8
Models: 4
Systems: 3
Failure casesCases: 375

Held in common

Case metrics, incident context, and topology
ReAct agent harness and shared task prompt
Interaction budget and per-response output limits
Ranked root causes and a predicted failure type

Varied deliberately

Tool abstraction and composition
Four models from the Qwen lineage
Three microservice environments
Interface instructions matched to available tools

What was actually evaluated
Model	L0	L1	L2	L3	L1+L2	L1+L3	L2+L3	L1+L2+L3
Qwen3-8B	●	●	—	—	●	—	—	●
Qwen3-32B	●	●	—	—	●	—	—	●
Qwen3-235B-A22B	●	●	—	—	●	—	—	●
Qwen3.7-Plus	●	●	●	●	●	●	●	●

OB / 125 CASES

Online Boutique

12 services · 49–59 metrics per case

gRPC service communication.

SS / 125 CASES

Sock Shop

15 services · 57–63 metrics per case

HTTP service communication.

TT / 125 CASES

Train Ticket

64 services · 198–238 metrics per case

Synchronous and asynchronous communication with complex call chains.

Metrics and study boundaries

AC@1 / AC@3: Whether the true root cause entity is in the top one / three candidates.
Avg@5: The mean of AC@1 through AC@5.
Type Accuracy (TA): Whether the predicted failure type matches the injected fault.
Completion: A valid diagnosis submitted within the interaction budget.
L3 usage: A trajectory invokes at least one L3 tool; this does not imply it receives a candidate ranking.
Efficiency: Mean total input + output tokens, elapsed seconds, and API cost per case.

All accuracy denominators include cases without valid diagnoses. The study uses all 375 RCAEval RE1 cases and five injected fault types: CPU, memory, disk, delay, and packet loss. It studies metrics and four models from one lineage, not production incidents, logs, traces, or other model families.

Comparison conditions and statistical reporting

Coverage Reported: Seven structured configurations plus an independent L0 reference. Qwen3.7-Plus is tested under all eight; the other three models under L0, L1, L1+L2, and L1+L2+L3.
Data and execution conditions Reported: The configurations share case data, the agent harness, the task prompt, and execution limits. Available structured interfaces and their instructions change. Shared data does not imply identical available actions.
Python access and environment Not specified in this manuscript: L0 explicitly exposes a general Python execution interface. The manuscript does not explicitly say whether structured configurations also expose it, or list the sandbox restrictions and installed dependencies.
Accuracy and efficiency aggregation Reported: Accuracy covers all evaluated cases, including invalid or missing diagnoses. Token usage includes input and output across all LLM calls; token, latency, and API cost are averaged over all evaluated cases.
Repeated runs and uncertainty Not specified in this manuscript: Repeat counts, random seeds, and uncertainty intervals are not stated. Small differences should not be read as statistically established improvements.
Conditional switching analysis Partially reported: The 79% / 5% / 5% comparison concerns switches from an L3 candidate to a different entity ranked most anomalous by L2. The corresponding sample counts are not stated; these percentages are not overall tool accuracy.

These notes describe what the public paper reports. Missing details are not inferred from rounded percentages.

Original study design

Original study workflow, tool configurations, model comparison, and system profiles. — Figure 2 · Study design. Full resolution

The research question and full abstract

08 / THE RESEARCH QUESTION

Abstract.

Rethinking Tool Design for Agentic RCA: A Controlled Empirical Study

A study of tool choices, not another RCA framework.

Existing RCA agents often use fixed preprocessing or predefined tools, leaving little experimental guidance on how much analysis tools should do. We define a tool spectrum, compare configurations under shared task and execution conditions, and inspect the decisions that follow tool calls. The contribution is evidence for tool design: what improves, what gets worse, and where useful tool results are lost.

Read the full abstract

Large language model (LLM) agents are increasingly explored for root cause analysis (RCA) in microservice systems, yet empirical guidance on how to design and combine their tools remains limited. We conduct a controlled empirical study of tool abstraction and composition across models and microservice environments. We implement 24 structured tools for metric access (L1), evidence analysis (L2), and diagnosis (L3), alongside a Python-based reference setting (L0). Using 375 failure cases from three microservice systems, we evaluate eight configurations with Qwen3.7-Plus and compare four Qwen models on a shared subset of four configurations. Our results show that tool configurations affect root cause localization and failure type identification differently. With Qwen3.7-Plus, L3 achieves 82.8% top-1 localization accuracy compared with 85.4% for L0, while requiring less than half the time per case. Adding tool levels can improve type identification while reducing localization accuracy. Trajectory analysis reveals cases in which agents override correct diagnostic recommendations after consulting additional evidence. Model choice also changes tool benefits: adding L3 to L1+L2 improves diagnosis for three models but hurts Qwen3-8B, which rarely invokes L3. Tool configuration rankings further change across microservice systems. These findings provide an empirical foundation for designing and using RCA tools, guiding tool selection and composition according to the model, diagnostic objective, and target system.

Paper and citation

09 / CITE THIS WORK

BibTeX.

arXiv 2610.05009 Read the PDF

@misc{luo2026rethinkingtooldesignagentic,
  title={Rethinking Tool Design for Agentic RCA: A Controlled Empirical Study},
  author={Yu Luo and Rongchen Gao and Zhenhui Zhou and Changchang Liu and Yuliang You and Yongqian Sun and Shenglin Zhang and Qiuai Fu and Shijie Wang and Dan Pei},
  year={2026},
  eprint={2610.05009},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2610.05009}
}

The tool found the cause.The agent changed its mind.

Two services. An unresolved diagnosis.

A Train Ticket investigation

ts-auth-service

Payment service

No CPU or memory anomaly does not rule out a delay fault.

Two services. An unresolved diagnosis.

L3 ranks auth first.

L2 highlights payment memory.

Missing symptoms become a reason to reject.

The agent switches to payment.

The root cause was auth, not payment.

Where does a diagnosis fail?

No valid finish.

The cause never enters.

Considered, not chosen.

Right entity, wrong type.

Does more help?

Root cause localization

Failure type identification

The resource tradeoff

The recommendation is stable. The decision is not.

Available is notthe same as useful.

Root cause localization

Failure type identification

Delegate the right analysis.

Supply valid inputs.

Use the result responsibly.

Same addition.Different consequences.

Online Boutique

Sock Shop

Train Ticket

Online Boutique

Sock Shop

Train Ticket

Principles & guidelines.

Separate the two diagnostic targets.

Match tools to demonstrated capability.

Make candidate revisions accountable.

Distinguish the fault from its symptoms.

Revalidate when the system changes.

Repair the stage that actually fails.

The tool spectrum.

Agent-written analysis

Metric access

Evidence analysis

Diagnostic methods

Agent-written analysis

Agent-written Python · one analysis step

Metric access

Metric query and summary

Evidence analysis

Anomaly and relationship analysis

Diagnostic methods

Diagnostic ranking and explanation

Change the tools.Keep the task grounded.

Held in common

Varied deliberately

Online Boutique

Sock Shop

Train Ticket

Abstract.

Rethinking Tool Design for Agentic RCA: A Controlled Empirical Study

BibTeX.

The tool found the cause.
The agent changed its mind.

Available is not
the same as useful.

Same addition.
Different consequences.

Change the tools.
Keep the task grounded.