The Tier 3 mindset: escalation engineering and RCA

From ticket-closer to root-cause engineer

At Tier 1 and Tier 2 your job was speed: restore the user, close the ticket, move on. At Tier 3 the unit of work changes. You are no longer paid to make a symptom disappear — you are paid to find why it happened and make sure it never happens to the rest of the fleet. That shift demands a different discipline: evidence over instinct, hypotheses over hunches, and elimination over guessing.

The core trap to avoid is treating a mitigation as a fix. Restarting a hung service, clearing a cache, or reimaging a machine restores service — but if you cannot say why it broke, you have only bought time. The fault is still latent in every other machine.

Symptom, proximate cause, root cause

Every incident has three layers. The symptom is what the user reports (“VPN drops”). The proximate cause is the immediate technical trigger (“the VPN service faults when the adapter renegotiates”). The root cause is the underlying condition that allowed the trigger to exist and recur (“a driver pushed without version pinning regresses the adapter”).

Tier 2 fixes proximate causes. Tier 3 eliminates root causes. If your fix only changes the proximate layer, expect the ticket to reopen across the fleet.

Hypotheses you can disprove

Good investigation is an elimination game, not a confirmation hunt. State each theory as a falsifiable hypothesis — a claim a specific test could prove wrong. Then run the cheapest test that kills the most hypotheses first. Resist the urge to run the test that confirms your favourite theory; confirmation bias is the single biggest time-sink in RCA.

Start by establishing a baseline with Reliability Monitor and event timelines:

# Pull the recent reliability/stability events into a timeline
Get-WinEvent -LogName System -MaxEvents 200 |
  Where-Object { $_.LevelDisplayName -in 'Error','Critical' } |
  Select-Object TimeCreated, Id, ProviderName, Message |
  Sort-Object TimeCreated

Evidence that survives challenge

Tier 3 findings get questioned — by vendors, by management, by the next engineer. Make your work defensible. Capture exact timestamps so you can correlate across logs, dumps, and traces. Hash any artefact you rely on so nobody can claim it was altered.

# Hash a captured dump or log so the evidence is tamper-evident
Get-FileHash -Algorithm SHA256 'C:\evidence\MEMORY.DMP'

# Record a precise, sortable timestamp for your notes
Get-Date -Format 'o'

A defensible RCA is one another engineer can reproduce from your notes alone. If they cannot rebuild your reasoning, you have a story, not a root cause.

Knowing when to stop

Not every cause can be proven with the evidence available. The mark of maturity is a bounded unknown: you document every hypothesis tested, what ruled it out, the monitored mitigation you applied, and exactly what new data would let you resume. That is honest engineering. An abandoned ticket dressed up as “intermittent” is not.

Your task

Take a real recurring incident from your queue (or a deliberately broken test VM). Write a one-page RCA scope: list the symptom, your candidate proximate causes, and at least three falsifiable hypotheses for the root cause. For each hypothesis, name the single cheapest test that would disprove it, and rank them so the first test eliminates the most theories. Collect a System event timeline with Get-WinEvent, hash one artefact, and record ISO-8601 timestamps. Conclude with either a proven root cause or an explicitly bounded unknown.