Incident Investigation

This is a real investigation, anonymised. Names, hostnames and service accounts have been changed; the metrics and the reasoning are unaltered. It is published as a sample because the shape of the work is more instructive than any description of it: what was asked, what was looked at, what the evidence supported, and what was recommended.

Environment. Aurora MySQL production cluster, primary plus two replicas (multi-AZ and DR). Observability via Datadog for metrics, logs, monitors and APM. Sixta connected through read-only Datadog API access only. No direct database access was used at any point in this investigation.

01. The question

A database engineer asked Sixta to investigate a production MySQL cluster, services-primary, across a fifteen-minute window on 16 March between 15:45 and 16:00 local time. Something had happened in that window and the team wanted to understand it.

Sixta began by querying Datadog metrics across the window: query throughput, per-statement counters for SELECT, UPDATE, INSERT and DELETE, disk I/O, CPU, InnoDB buffer pool, lock metrics, thread activity and replication lag. The first pass completed in under two minutes.

[table-1]

02. Initial findings

Sixta identified three distinct events inside and immediately after the window of interest. The first two were precursors. The third was the main event.

[table-2]

SELECT traffic remained flat throughout, so this was a pure write event. The UPDATE bursts showed a rhythmic pattern, spiking every 90 to 120 seconds, which is the signature of a batch job processing in chunks rather than of organic application load.

03. Root cause analysis

Sixta deepened the investigation by pulling connection logs, slow query log errors, replication metrics and handler statistics. Four independent lines of evidence converged on the same cause.

Root cause. A scheduled LMS integration batch job, running as svc_lms_integration, opened a large connection pool at around 15:54 UTC and ran sustained UPDATE operations across the cluster for more than 25 minutes. It processed in chunks initially, producing the rhythmic spikes, then switched to sustained non-throttled writes from 16:00 onwards.

The evidence chain

Connection pool signature. More than twenty connections from the same service account expired via wait_timeout, set to 900 seconds, simultaneously between 16:22 and 16:25. Simultaneous expiry means simultaneous opening: they were created together at around 15:54 for a batch job and left idle afterwards.

Chunked processing pattern. The UPDATE rate spiked sharply to 100 to 200 per second, then dropped to 6 to 8 per second, repeating every 90 to 120 seconds from 15:31 through 15:57. At 16:00 the throttle came off and writes became sustained at 60 to 150 per second continuously.

Table scan evidence. handler_read_rnd_next, which counts sequential scan reads, hit 100,000 to 160,000 per second during the burst windows against a baseline of 8,000 to 50,000. The UPDATE WHERE clauses appear to lack adequate indexes, forcing full table scans simply to locate the rows to be updated.

Handler writes confirmation. Row-level writes rose from a baseline of 1,500 to 2,000 per second to 18,000 to 19,000 per second at peak, confirming the scale of the write activity.

Downstream effects

[table-3]

04. Follow-up investigation

The engineer asked a second question: if 80% disk usage is an operational risk, how fast is it growing? Sixta pulled the seven-day disk trend.

Finding. The disk is not at 80%. It has been stable at around 70% for seven days with no growth trajectory. The 80.4% alert most likely reflects a different metric calculation, raw used over total rather than system.disk.in_use, which accounts for reserved blocks. The batch job is not accumulating disk over time; binary logs are being purged on schedule.

This is worth dwelling on. The alert that fired was, in the narrow sense, correct. It was also misleading, and had the team responded to it by provisioning storage they would have spent money solving a problem they did not have. The useful output of an investigation is often a decision not to act.

Recommendations

[table-4]

What this shows

The investigation was conducted entirely through a read-only Datadog integration. No direct database access was required, no agent was installed on the instances, and nothing was changed. The full root-cause analysis completed in under ten minutes from the initial question, against a window that would typically absorb an afternoon of an experienced engineer's time.

Questions about this sample, or about running the same investigation against your own cluster: ewen@sixta.ai.