Design Log Rotation

Problem Statement: Fleet Log Rotation Platform

Log rotation at fleet scale is the discipline of reclaiming disk before it fills, without losing a single forensic byte. The naive answer — a cron job running logrotate on each host — collapses across thousands of heterogeneous nodes: some apps reopen their log file on a signal, some hold an open file descriptor forever, and container runtimes write through a json-file driver that ignores your rename entirely. The platform we design is three layers: a policy plane where operators define versioned, auditable rotation rules; an executor (an agent on every node) that enforces them inode-safely; and an archive pipeline that ships compressed, checksummed segments to S3 and Elastic before anything is deleted. The non-negotiable invariant is that the active file is never removed until its archive's checksum has verified.

What to state early

• Disk-full incidents outpace central log shipping on bursty nodes
• Policies must be versioned, auditable, and pushed without SSH
• Rotation must preserve forensic bytes for Elastic and S3 ingest

The three metrics that define success

Every design choice ties back to three numbers: rotate_success_rate (did the rotation complete inode-safely?), upload_lag_p95 (how far behind is the archive pipeline?), and disk_forecast_error (how accurately do we predict a host filling up?). Inode-safety is the detail that separates a real answer from logrotate-on-cron: when you rename a file, a writer holding the old file descriptor keeps writing to the now-unlinked inode, silently filling the disk while the new file stays empty. The platform exists to make that class of bug impossible.

When it degrades

The failure that matters is the archive pipeline stalling — S3 throttling, a network partition — while disks keep filling. The correct response is to shed gracefully: pause policy rollouts, drop non-critical upload queues, and page the moment any host projects a full disk within six hours. The one thing the platform must never do under pressure is delete an active log before its archive checksum verifies, because that trades a recoverable disk-space problem for permanent data loss.

Cost and operability

Contrast ad-hoc cron logrotate with a control plane that understands inode stickiness, Kubernetes container logs, and compliance retention tiers.

Deep dive

Contrast ad-hoc cron logrotate with a control plane that understands inode stickiness, Kubernetes container logs, and compliance retention tiers.

How to open this one

The framing that signals depth on log rotation is inode-safety over cron: a rename leaves a writer that holds an open descriptor writing into an unlinked inode, so the disk fills while the new file stays empty. Lead with the writer-behavior decision tree (reopen-on-signal versus copytruncate) and the failure story that proves it: a legacy JVM keeps its file descriptor open through a rename, and the host runs out of disk an hour later with an apparently-empty log. That shows you understand rotation is a correctness problem, not a cleanup script.