PostgreSQL's write-ahead log (WAL) is the bedrock of transactional durability—or so the documentation promises. Every commit flushes a record to disk before reporting success. But in production, that promise is complicated by kernel behavior, storage quirks, and configuration gaps. Silent data loss, where a commit reports OK but the data never survives a crash, is rarer than it used to be, yet still haunts operators who push Postgres to its limits. This article traces the failure modes along the write path, from fsync semantics to checkpoint storms, drawing on real incidents and community postmortems.

The Write-Ahead Log: Where Latency Becomes Liability

WAL serializes all writes into a sequential stream. Every transaction generates one or more WAL records, which must be flushed to stable storage before the commit is acknowledged. That serialization point is the single most contentious resource under write-heavy workloads. When WAL flush latency spikes—due to disk contention, filesystem barriers, or hardware write-back cache misbehavior—commit latency rises across all concurrent sessions.

In a well-documented incident from 2020, a financial services company experienced a multi-hour outage traced to a WAL flush bottleneck on their Postgres replicas. The incident report described how a burst of write activity caused WAL generation to outpace the storage subsystem's ability to fsync, leading to cascading replication lag and eventual failover failure. The root cause was not a disk failure but a latency spike that exceeded the replication timeout, causing the primary to demote itself. This pattern has been replicated in many environments and is a known risk in high-throughput deployments.

Silent data loss occurs when a crash or power loss interrupts a WAL flush after the kernel reports success but before bits are physically on media. PostgreSQL relies on the assumption that fsync() guarantees durability. But on some storage stacks—particularly virtualized environments with write-back caching—that assumption is fragile. The data_checksums feature catches torn pages during recovery, but it cannot detect a WAL record that was never written at all.

Partial write guarantees are another concern. PostgreSQL's WAL is structured in 8 KB pages. A crash during a write can leave a page partially updated, a state known as a torn write. The recovery process detects torn pages via a checksum or a page-level LSN, but if the WAL record itself is incomplete, recovery may skip it, leaving the database in an inconsistent state that only application-level checks can catch.

fsync: The Kernel's Broken Promise

The fsync() system call is supposed to flush all dirty buffers for a file descriptor to persistent storage. In practice, its semantics vary widely across filesystems and storage hardware. On Ext4 with default settings, fsync() flushes the file's data but may not issue a cache flush command to the disk drive if the filesystem journal is in ordered mode. XFS, by contrast, issues a barrier that forces the storage device to drain its write-back cache—but only if the device honors SCSI commands.

Consumer-grade SSDs often lie about flush completion. Many drives report a cache flush as complete when data is still sitting in a volatile DRAM buffer, a behavior that violates the ATA/SCSI specification but persists in the name of performance. Enterprise NVMe devices with power-loss protection are more reliable, but even they can exhibit rare failures under electrical noise or temperature extremes.

ZFS provides stronger guarantees through its copy-on-write transaction group model and explicit ZIL (ZFS Intent Log) flushing. But ZFS is not the default filesystem for most PostgreSQL deployments. The gap between what fsync() promises and what the hardware delivers is where silent corruption breeds. PostgreSQL's pg_test_fsync utility measures actual flush latency, but few operators run it before provisioning a production cluster.

The data_checksums feature, enabled at cluster initialization, computes a CRC-32C checksum on every page. During recovery, PostgreSQL verifies checksums and can detect pages that were partially written. This is not a fix for fsync failures—it only detects corruption after the fact. Some operators combine checksums with wal_log_hints to enable online checksum verification, but the overhead is non-trivial on write-heavy systems.

Partial WAL Flushes: How Corruption Creeps In

A crash in the middle of a WAL write can produce a partial WAL record. PostgreSQL's recovery process reads WAL sequentially; if a record's header is intact but its body is truncated, recovery may interpret garbage as a valid record, leading to logical corruption that propagates into data pages. This is distinct from a torn page—it's a torn WAL record, and it can be much harder to detect.

Full page writes are PostgreSQL's primary defense against torn pages. When enabled (the default since version 9.6), the first modification of a page after a checkpoint writes the entire page image to WAL. This bloats WAL volume by a factor of 2–5 during the first write per checkpoint, but it guarantees that recovery can reconstruct any page from a single WAL record, even if the page on disk is corrupt.

The cost of full page writes is significant. Each full page image is 8 KB plus overhead. On a system with large shared buffers and frequent checkpoint intervals, WAL generation can exceed 100 MB/s per transaction, saturating network links during WAL archiving. Some operators disable full page writes on replicas or during bulk loads, accepting the risk of a torn page in exchange for throughput.

One notable incident involved a managed PostgreSQL provider in 2019, where a combination of full page writes being disabled and a kernel bug in Ext4 caused silent corruption during a failover. The WAL stream appeared valid, but several pages were missing updates that had been acknowledged to clients. The provider restored from a backup taken hours earlier, losing all intervening transactions. The postmortem (published on the provider's engineering blog) detailed how the kernel's delayed allocation interacted with PostgreSQL's write ordering, leading to pages that were never flushed despite fsync returning success. As a result, the provider changed their configuration defaults for full_page_writes and wal_level on their managed Postgres offering, and added a kernel-level workaround to force barrier flushes.

Replication Lag and the Illusion of Durability

Synchronous replication promises that a transaction is not committed until at least one standby has written the WAL record. This eliminates the window of data loss during a primary crash—provided the standby is truly synchronous. But synchronous replication adds latency: the primary must wait for a network round-trip to the standby before acknowledging the commit. In geographically distributed clusters, this latency can exceed 100 ms per transaction, limiting write throughput.

Asynchronous replication, the default in PostgreSQL, provides no durability guarantee for recent transactions. If the primary crashes before WAL is shipped, those transactions are lost. The window of vulnerability is bounded by wal_writer_delay and network latency, but it can be seconds or even minutes under heavy load. Applications that read their own writes may see phantom reads: a transaction committed on the primary disappears when the application reads from a lagging standby.

WAL archiving adds another layer of latency. The archive_command is invoked after each WAL segment is filled and flushed. If the archive destination (e.g., S3 or NFS) is slow or unreachable, WAL accumulation on the primary can fill the disk, causing the database to halt. Some operators set archive_timeout to force frequent segment switches, but this increases the number of small WAL files and can degrade restore performance.

In a widely cited engineering blog post from 2018 titled "WAL Shipping at Scale" (published by a major streaming platform), the team described a WAL shipping bottleneck where their write throughput exceeded the network bandwidth between primary and standby. They mitigated this by compressing WAL on the fly and using multiple parallel archive workers, but the fundamental tension between durability and throughput remained. The lesson: replication lag is not just a read-scaling problem; it is a durability problem in disguise.

Vacuum and Autovacuum: The Hidden Write Amplification

Autovacuum is essential for preventing table bloat and transaction ID wraparound, but it generates WAL traffic that can overwhelm the write path. Each dead tuple removed by vacuum produces a WAL record for the heap page, and each index entry removed produces additional WAL for the index. On tables with many indexes, vacuum can generate more WAL than the application writes.

The max_wal_size parameter controls how much WAL can accumulate before a checkpoint is forced. If autovacuum generates a burst of WAL, the system may trigger a checkpoint earlier than expected, causing a spike in I/O as dirty buffers are flushed. This is especially problematic on systems with bursty write workloads, such as SaaS platforms that run nightly batch jobs or ETL pipelines.

Table bloat exacerbates checkpoint I/O. A bloated table has many dead tuples spread across pages, requiring vacuum to scan and rewrite more pages than necessary. Each rewritten page becomes a full page write if the page was modified since the last checkpoint. Operators can monitor WAL generation rate per table using the pg_stat_statements and pg_stat_bgwriter views, but correlating WAL volume to specific vacuum operations requires log parsing.

Citus, the distributed Postgres extension, introduced WAL rate limits in their managed service to prevent a single tenant's vacuum from saturating the shared WAL write path. They set a cap on the number of WAL bytes generated per second per node, throttling autovacuum when the limit is exceeded. This trade-off slows down vacuum but prevents checkpoint storms that affect all tenants. The approach is not standard PostgreSQL, but it illustrates the kind of operational thinking needed at scale.

Checkpoint Storms: When the WAL Freezes

Checkpoints are the mechanism that shortens crash recovery time by ensuring that all dirty buffers up to a certain LSN have been written to disk. But checkpoints themselves are I/O-intensive. During a checkpoint, PostgreSQL writes all dirty shared buffers to disk, and each write generates a full page image in WAL if the page was modified since the last checkpoint. This can cause a burst of WAL that saturates the storage subsystem.

Checkpoint frequency is governed by checkpoint_timeout and max_wal_size. If max_wal_size is set too low, checkpoints occur more frequently, increasing the overhead of full page writes. If set too high, checkpoints are rare but each one writes a large volume of data, potentially starving other I/O. The checkpoint_completion_target parameter spreads the write load over a fraction of the checkpoint interval, but it cannot eliminate the spike entirely.

On shared storage systems like Amazon EBS or SAN arrays, a checkpoint storm can degrade performance for all workloads sharing the same volume. AWS RDS for PostgreSQL ships with conservative checkpoint defaults that prioritize stability over throughput: checkpoint_timeout is 5 minutes and max_wal_size is 1 GB. These settings work for most workloads, but bursty write patterns can still trigger checkpoint storms that cause application timeouts.

One mitigation is to increase checkpoint_completion_target to 0.9, spreading writes over most of the checkpoint interval. Another is to use a separate WAL volume with higher IOPS, isolating WAL I/O from data I/O. Some operators disable full page writes entirely on replicas, accepting the risk of torn pages during standby promotion. The choice depends on the tolerable recovery time and the cost of data loss.

Hardening the Write Path in Production

The first line of defense is choosing the right filesystem. XFS and ZFS provide stronger barrier semantics than Ext4, especially with write-back cache enabled. On Linux, mounting with barrier or nobarrier should be matched to the storage device's capabilities. For NVMe drives with power-loss protection, barriers are redundant; for consumer SSDs, they are essential.

Enable data_checksums at cluster creation. It adds roughly 3% overhead to write-heavy workloads but provides a safety net for detecting torn pages. Run pg_test_fsync on each node before production deployment to measure actual fsync latency. If the reported latency is above 10 ms on average, investigate the storage stack.

Setting synchronous_commit to remote_write on the primary is a commonly recommended approach. This waits for the standby to receive the WAL but not necessarily flush it to disk, reducing commit latency while still preventing data loss if the primary crashes. However, note that this is a recommendation based on typical production patterns, not a guarantee. Operators should test this setting in their own environment, as the trade-off is that a simultaneous crash of both primary and standby can lose the last few transactions. For deployments with extreme durability requirements, synchronous_commit = on may be more appropriate despite the higher latency.

Monitor pg_stat_bgwriter for checkpoint-related metrics: checkpoints_timed vs. checkpoints_req, buffers_checkpoint, and maxwritten_clean. A high ratio of requested to timed checkpoints indicates that max_wal_size is too low. Also monitor WAL generation rate via pg_wal_lsn_diff or by tracking WAL file creation timestamps.

Test crash recovery regularly. Use pg_test_fsync and pg_checksums to validate that the storage stack behaves correctly under power loss. Some teams run chaos engineering experiments that kill the database process or simulate a power failure, then verify that no data is lost. This is the only way to build confidence that the write path is truly durable.

Beyond these practices, there are specialized monitoring tools that can help. For example, pg_waldump can be used to inspect WAL contents for anomalies, and pg_stat_replication provides real-time replication lag metrics. Third-party tools like check_pgactivity or pgwatch2 can alert on WAL generation spikes or checkpoint frequency changes. Some teams also implement custom scripts that parse PostgreSQL logs for warning messages about fsync failures or checksum mismatches, integrating with their existing monitoring infrastructure.

Despite all these measures, perfect durability remains an elusive goal. The complexity of the storage stack—spanning the kernel, filesystem, device firmware, and network—means that new failure modes continue to surface. The PostgreSQL community actively tracks issues like the ext4 data loss bug (CVE-2021-20268) and the XFS barrier behavior changes across kernel versions. Operators must stay informed and test their specific configuration under realistic failure scenarios. The write path is not a set-and-forget component; it demands ongoing attention and adaptation.