Server Monitoring Essentials: CPU, Memory, Disk, and Network

Server monitoring forms the backbone of every reliable infrastructure operation. Whether you manage a single application server or orchestrate hundreds of nodes across multiple data centers, understanding the four fundamental resource pillars — CPU, memory, disk, and network — determines whether you catch degradation before users notice or scramble to respond after the damage is done. This guide provides the technical depth needed to implement monitoring that genuinely protects your systems.

Why Server Monitoring Matters

Every production outage has a timeline. A database query starts running slowly. Memory pressure builds as connection pools expand. Disk I/O latency spikes as the write-ahead log competes with reads. Network throughput drops as TCP retransmissions climb. These signals appear minutes or hours before users experience errors, but only if you are collecting and evaluating the right metrics at the right granularity.

The cost of inadequate monitoring extends beyond downtime. Silent performance degradation erodes user trust gradually, manifesting as higher bounce rates and lower conversion rather than explicit error reports. Capacity planning without historical data becomes guesswork, leading to either over-provisioned infrastructure burning budget or under-provisioned systems collapsing under traffic spikes.

Modern monitoring architecture separates collection, storage, and alerting into distinct layers. Agents running on each server collect raw metrics at intervals typically between 10 and 60 seconds. These metrics flow into a time-series database optimized for high-cardinality write workloads. Alerting rules evaluate conditions against this stored data and trigger notifications through escalation policies.

CPU Monitoring in Depth

CPU metrics carry more nuance than a single utilization percentage suggests. The kernel tracks multiple states for each processor core, and understanding these states separates actionable alerts from noise.

CPU States Explained

The Linux kernel reports CPU time across several categories. User time represents cycles spent executing application code. System time covers kernel operations including system calls, interrupt handling, and context switches. I/O wait measures time the CPU spends idle while waiting for disk operations to complete — a metric frequently misinterpreted as CPU load when it actually signals storage bottlenecks.

Steal time appears exclusively in virtualized environments and indicates cycles the hypervisor allocated to other virtual machines. A rising steal percentage on cloud instances signals that your VM is contending with noisy neighbors or that the instance type provides insufficient dedicated compute.

# Key CPU metrics from /proc/stat
# user    nice   system  idle      iowait  irq  softirq  steal
cpu  74132  1230   18423   892345    4521    892  1204     312

# Per-core breakdown reveals asymmetric load
cpu0 18742  308    4612    223041    1130    223  301      78
cpu1 18534  311    4589    223142    1128    224  298      82
cpu2 18623  305    4611    223080    1131    222  303      76
cpu3 18233  306    4611    223082    1132    223  302      76

Load Average vs. CPU Utilization

Load average and CPU utilization measure different things. Load average counts the total number of processes in runnable and uninterruptible states, averaged over 1, 5, and 15 minutes. A load average of 4.0 on a 4-core machine means every core has exactly one runnable process — full utilization without queuing. A load average of 8.0 on the same machine means processes are queuing for CPU time.

CPU utilization, by contrast, measures the percentage of time each core spends in non-idle states. A system can show 100% CPU utilization with a load average of 4.0 (no queuing) or with a load average of 40.0 (severe queuing). The load average reveals the queuing depth that raw utilization cannot.

CPU Alerting Thresholds

Setting CPU alert thresholds requires understanding your application profile. A batch processing server running at 95% utilization during scheduled jobs is behaving correctly. A web application server at 85% sustained utilization during normal traffic is approaching the point where latency increases non-linearly as queuing begins.

Rather than alerting on instantaneous values, evaluate sustained conditions. A threshold of "CPU utilization above 80% for more than 5 minutes" eliminates transient spikes from garbage collection or cron jobs while catching genuine saturation. Pair this with a synthetic monitoring check on application response time to correlate resource pressure with user-visible impact.

Memory Monitoring Architecture

Memory monitoring in Linux requires careful interpretation. The numbers reported by the kernel do not map intuitively to how much memory your application actually needs, and naive monitoring of "used memory" produces misleading alerts.

Understanding Linux Memory Management

Linux aggressively uses available memory for disk caching. A server showing 95% memory utilization might have 60% consumed by application processes and 35% dedicated to page cache — memory the kernel will reclaim instantly when applications need it. The metric that matters is available memory, introduced in kernel 3.14, which estimates how much memory can be allocated without swapping.

# /proc/meminfo key fields
MemTotal:       32768000 kB
MemFree:         1024000 kB    # Truly unused (often low, that's fine)
MemAvailable:    8192000 kB    # What can be allocated without swapping
Buffers:          512000 kB    # Block device cache
Cached:         12288000 kB    # Page cache (reclaimable)
SwapTotal:       8388608 kB
SwapFree:        8100000 kB
Committed_AS:   24576000 kB    # Total committed memory

Swap Behavior and OOM Events

Swap activity is a stronger signal than memory utilization for detecting memory pressure. Minor swap-in and swap-out rates during idle periods are normal on systems with swap enabled. Sustained swap activity during production traffic indicates that working set size exceeds physical memory, forcing the kernel to page out active data.

The Out-of-Memory (OOM) killer activates when the system has exhausted both physical memory and swap. It selects a process to terminate based on a scoring algorithm that considers memory usage, process age, and an adjustable oom_score_adj parameter. Monitoring for OOM events requires watching kernel logs for "Out of memory" messages, since the terminated process cannot report its own death.

Configure your monitoring agent to track MemAvailable rather than calculated "used memory" to avoid false alerts when page cache is high. Set warning thresholds when available memory drops below 20% and critical thresholds at 10%. Track swap I/O rate (si and so from vmstat) with alerts on sustained values above zero during peak traffic.

Disk I/O Monitoring

Disk performance monitoring splits into two concerns: capacity (how much space remains) and performance (how fast reads and writes complete). Both require different metrics and different alerting strategies.

Capacity Monitoring

Disk space alerts are deceptively simple to implement and notoriously prone to false negatives. A percentage-based threshold (warn at 80%, critical at 90%) works poorly on modern servers where a 2TB disk at 80% still has 400GB free while a 50GB root partition at 80% has only 10GB remaining. Absolute thresholds based on remaining space better reflect operational risk.

Inode exhaustion is the overlooked sibling of disk space monitoring. A filesystem can have gigabytes of free space but zero free inodes if the application creates millions of small files. Monitor inode usage alongside block usage, particularly on servers running applications that generate per-request log files, session files, or cache entries.

I/O Performance Metrics

The kernel exposes detailed I/O statistics through /proc/diskstats and the iostat utility. The critical metrics for performance monitoring include:

• IOPS (I/O Operations Per Second) — the number of read and write operations the disk handles per second. SSDs typically sustain 10,000-100,000+ IOPS while spinning disks deliver 100-200 IOPS.
• Throughput (MB/s) — the volume of data read or written per second. Sequential workloads maximize throughput; random workloads maximize IOPS.
• Latency (await) — the average time in milliseconds for an I/O request to complete, including queue time. This is the most user-visible metric. SSD latency typically stays below 1ms; values above 10ms indicate saturation.
• Queue depth (avgqu-sz) — the average number of I/O requests waiting in the device queue. A growing queue depth signals that the storage subsystem cannot keep pace with request volume.
• Utilization (%util) — the percentage of time the device was busy handling requests. For single-queue devices (spinning disks), 100% indicates saturation. For multi-queue NVMe devices, this metric is misleading because the device can process many requests simultaneously.

Network Monitoring Fundamentals

Network monitoring captures both the volume of traffic flowing through server interfaces and the quality of that traffic as measured by error rates, retransmissions, and connection states.

Bandwidth and Throughput

Interface-level metrics from /proc/net/dev report bytes and packets transmitted and received on each network interface. The delta between consecutive readings, divided by the sampling interval, yields throughput in bytes per second. Comparing this throughput against the interface capacity (1 Gbps, 10 Gbps, 25 Gbps) gives utilization percentage.

Sustained interface utilization above 70% warrants investigation. While modern NICs handle high throughput efficiently, sustained high utilization reduces headroom for traffic spikes and increases tail latency as kernel network buffers fill. For applications behind a load balancer, correlate per-server network utilization with server response time to determine whether bandwidth saturation contributes to latency increases.

TCP Connection State Monitoring

The distribution of TCP connections across states reveals application-level behavior that raw throughput metrics cannot. A growing count of TIME_WAIT connections indicates high connection churn, common in HTTP/1.1 applications making many short-lived connections to backend services. A large number of CLOSE_WAIT connections signals that your application is failing to close sockets after the remote end disconnects — typically a resource leak in connection handling code.

Track the total number of established connections alongside the rate of new connections per second. A web server handling 10,000 concurrent connections with 500 new connections per second is operating differently from one with 500 concurrent connections and 500 new connections per second. The first scenario involves long-lived connections (perhaps WebSocket or HTTP/2 streams) while the second involves rapid connection cycling.

# TCP connection state summary
$ ss -s
Total: 14523
TCP:   12847 (estab 9521, closed 1203, orphaned 42, timewait 1892)

# Detailed state breakdown
$ ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn
   9521 ESTAB
   1892 TIME-WAIT
    623 CLOSE-WAIT
    411 FIN-WAIT-2
    284 SYN-RECV
    116 LAST-ACK

Network Error Monitoring

Network errors fall into categories that indicate different infrastructure problems. Interface errors (rx_errors, tx_errors) on physical interfaces suggest cabling, NIC hardware, or driver issues. Dropped packets (rx_dropped, tx_dropped) indicate kernel buffer exhaustion, often caused by interrupt processing falling behind packet arrival rates. TCP retransmissions signal packet loss somewhere in the network path and directly impact application latency.

A retransmission rate above 1% demands investigation. Even 0.5% retransmissions can add 200-500ms of latency to affected connections as TCP waits for retransmission timeouts. Correlate retransmission rates with distributed trace data to identify whether packet loss occurs within your network infrastructure or on external paths.

Monitoring Agent Selection and Configuration

The monitoring agent running on each server determines what metrics you can collect and the overhead imposed on the host system. Three agent architectures dominate the current landscape.

Pull-Based Collection (Prometheus Model)

Prometheus node_exporter runs as a lightweight HTTP server on each host, exposing a /metrics endpoint that the Prometheus server scrapes at configured intervals. This model centralizes collection scheduling and requires no agent-side configuration of destinations. The agent itself consumes minimal resources: typically under 10MB of RSS and negligible CPU.

Push-Based Collection (Telegraf / StatsD Model)

Push-based agents collect metrics locally and transmit them to a central collector on a schedule. Telegraf supports over 200 input plugins and can output to dozens of destinations simultaneously. This model works better in ephemeral environments where instances spin up and terminate before a scraper discovers them. The tradeoff is more complex agent configuration and the need for buffering logic when the destination is temporarily unreachable.

OpenTelemetry Collector

The OpenTelemetry Collector represents the convergence of monitoring and APM collection into a single agent. It receives metrics, traces, and logs through multiple protocols (OTLP, Prometheus, StatsD, syslog) and exports them to any backend. For teams running both infrastructure monitoring and application performance monitoring, consolidating on a single collector reduces operational complexity and resource usage.

Capacity Planning with Historical Data

Monitoring data becomes most valuable when analyzed over weeks and months to reveal growth trends. Linear regression on daily peak CPU utilization over 90 days projects when existing capacity will be exhausted. Seasonal patterns in memory usage — higher during business hours, lower on weekends — inform autoscaling policies that provision resources before demand arrives rather than after.

Store monitoring data at decreasing resolution over time: 15-second intervals for the last 24 hours, 1-minute intervals for the last week, 5-minute intervals for the last month, and 1-hour intervals for the last year. This downsampling strategy balances query performance with historical depth, keeping storage costs manageable while preserving the ability to identify long-term trends.

Correlation between resource metrics and application-level KPIs transforms raw infrastructure data into business intelligence. When you can demonstrate that a 10% increase in p99 page load time correlates with a 2% decrease in conversion rate, infrastructure investment decisions gain quantitative justification.

Building a Monitoring Baseline

Effective alerting requires a baseline understanding of normal system behavior. Collect at least two weeks of monitoring data before setting alert thresholds. During this period, annotate known events — deployments, traffic spikes, batch job schedules — alongside the metric data. These annotations separate expected patterns from anomalies that warrant investigation.

Document your monitoring configuration in version-controlled infrastructure code. Alert thresholds, dashboard definitions, and agent configurations stored alongside application code ensure that monitoring evolves with the systems it protects. When a new service deploys, its monitoring configuration ships with it rather than being added as an afterthought.

The gap between collecting metrics and acting on them defines your operational maturity. Raw metrics without well-designed alerting produce dashboards that nobody watches. Alerts without runbooks produce pages that nobody can resolve efficiently. The monitoring system is only as valuable as the response process it enables.