6 min read Intermediate 20.6k

Optimize the Linux kernel for a production VPS

Published on May 17, 2026 at 12:21 PM

Tune your Linux VPS for the best network and system performance. This guide covers sysctl, TCP BBR (Google's congestion controller), file descriptors, I/O scheduler and VPS-specific optimizations.

Introduction

Default Linux kernel values are designed to work everywhere, from Raspberry Pi to enterprise server. On a production VPS hosting a website, database or game server, these values are suboptimal.

This guide gives you proven optimizations that:

Improve network latency and throughput (BBR, fast open)
Allow more simultaneous connections (file descriptors, ports)
Reduce memory consumption (swappiness)
Prevent classic attacks (anti-SYN flood)

Prerequisites

Linux VPS (Debian 12 / Ubuntu 22.04+)
Root access
Kernel 5.4+ (for TCP BBR)

Check your version:

uname -r

Step 1: Backup and structure

sudo cp /etc/sysctl.conf /etc/sysctl.conf.bak
sudo mkdir -p /etc/sysctl.d/

All modifications will go into /etc/sysctl.d/99-tuning.conf (better for maintenance than editing /etc/sysctl.conf).

sudo nano /etc/sysctl.d/99-tuning.conf

Step 2: Enable TCP BBR

TCP BBR (Bottleneck Bandwidth and Round-trip propagation time) is the congestion controller developed by Google. It replaces the default CUBIC algorithm and drastically improves throughput, especially on variable-latency connections.

Before tuning: ~20-50 Mbps on a mobile connection, packet loss > 1%. After BBR: ~80-150 Mbps on the same connection.

Add to 99-tuning.conf:

# === TCP BBR ===
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Apply:

sudo sysctl --system

Verify:

sysctl net.ipv4.tcp_congestion_control

Should return bbr.

Step 3: Increase network buffers

On VPS that transfer lots of data (massive download/upload, streaming):

# === Network buffers ===
# Recv buffer max
net.core.rmem_max = 67108864
net.core.rmem_default = 262144

# Send buffer max
net.core.wmem_max = 67108864
net.core.wmem_default = 262144

# TCP buffer
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432

# Queue length
net.core.netdev_max_backlog = 16384

Step 4: Optimize TCP

# === TCP optimizations ===
# Enable TCP Fast Open (reduces handshake latency)
net.ipv4.tcp_fastopen = 3

# Fast socket reuse in TIME_WAIT
net.ipv4.tcp_tw_reuse = 1

# TIME_WAIT duration
net.ipv4.tcp_fin_timeout = 15

# Faster keepalive to detect dead connections
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5

# Increase max pending connections
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Enable MTU probing
net.ipv4.tcp_mtu_probing = 1

# Prevent throughput drop after idle
net.ipv4.tcp_slow_start_after_idle = 0

Step 5: Basic anti-DDoS protection

Netrix handles the big stuff, but a few sysctls complement at kernel level:

# === Anti-DDoS ===
# SYN flood protection
net.ipv4.tcp_syncookies = 1

# Ignore ping broadcasts
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0

# Anti-spoofing (rp_filter)
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Log martian packets
net.ipv4.conf.all.log_martians = 1

# Disable source routing
net.ipv4.conf.all.accept_source_route = 0

Step 6: Expand ephemeral port range

By default, Linux uses about 28K ephemeral ports. For heavily loaded servers (proxies, load balancers), this is limiting:

# === Wider ephemeral port range ===
net.ipv4.ip_local_port_range = 1024 65535

Step 7: Memory optimizations

# === Memory / Swap ===
# Reduce swap usage (prefer RAM)
vm.swappiness = 10

# More aggressive filesystem cache
vm.vfs_cache_pressure = 50

# Dirty pages ratio before disk write
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5

vm.swappiness = 10: only swap if > 90% RAM used. On SSD/NVMe, lower to 1 to minimize wear.

Step 8: File descriptors

Many servers hit the default 1024 file descriptors per process limit. Increase:

# === File descriptors ===
fs.file-max = 2097152
fs.nr_open = 1048576

And shell-side, in /etc/security/limits.conf:

sudo nano /etc/security/limits.conf

Add at the end:

* soft nofile 1048576
* hard nofile 1048576
* soft nproc 65535
* hard nproc 65535
root soft nofile 1048576
root hard nofile 1048576

And enable in PAM:

sudo nano /etc/pam.d/common-session

Add:

session required pam_limits.so

For systemd services (Nginx, MySQL, etc.):

sudo mkdir -p /etc/systemd/system.conf.d
sudo nano /etc/systemd/system.conf.d/limits.conf

[Manager]
DefaultLimitNOFILE=1048576
DefaultLimitNPROC=65535

Step 9: Optimize I/O scheduler

On SSD/NVMe, the default scheduler (mq-deadline) is fine, but none can be faster:

# View current scheduler
cat /sys/block/sda/queue/scheduler

# Change (temporary, lost on reboot)
echo none | sudo tee /sys/block/sda/queue/scheduler

To persist, create a udev rule:

sudo nano /etc/udev/rules.d/60-scheduler.rules

# For SSD/NVMe
ACTION=="add|change", KERNEL=="sd[a-z]|nvme[0-9]n[0-9]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"

Step 10: Apply everything at once

After editing 99-tuning.conf:

sudo sysctl --system

Check no value triggers an error:

sudo sysctl -p /etc/sysctl.d/99-tuning.conf 2>&1 | grep -i error

Step 11: Verify performance

Network throughput test

# iPerf3 between your VPS and another server
sudo apt install -y iperf3

# On test server
iperf3 -s

# On your VPS
iperf3 -c OTHER_SERVER_IP -t 30

TCP latency test

# Time to first byte over 100 connections
time for i in {1..100}; do curl -o /dev/null -s "https://google.com"; done

Verify BBR active

sysctl net.ipv4.tcp_congestion_control
# bbr

cat /proc/sys/net/ipv4/tcp_available_congestion_control
# Should list bbr in addition to cubic

Count sockets

ss -s

Step 12: Complementary application optimizations

Kernel tuning gives the max if your apps are also tuned. See:

Nginx tuning (workers, keepalive): /docs/article/nginx-tuning
MySQL/MariaDB tuning (buffer pool): /docs/article/mysql-tuning
PHP-FPM (pm.max_children): adjust based on RAM

Troubleshooting

Worse performance after tuning

Rare but possible. Compare before/after with iperf3. If worse:

sudo cp /etc/sysctl.conf.bak /etc/sysctl.conf
sudo mv /etc/sysctl.d/99-tuning.conf /etc/sysctl.d/99-tuning.conf.bak
sudo sysctl --system

"Operation not permitted" on certain sysctl

Kernel doesn't support some options. Check:

uname -r

For BBR, kernel 4.9+ required. For tcp_fastopen, 4.0+.

"Too many open files" limit persists

The service must be restarted to take new limits:

sudo systemctl restart nginx mysql php8.3-fpm

Check a process's limit:

cat /proc/$(pidof nginx | awk '{print $1}')/limits | grep "Max open files"

Useful commands

# View all active sysctl
sudo sysctl -a

# View specific value
sudo sysctl net.ipv4.tcp_congestion_control

# Apply temporary change
sudo sysctl -w net.core.somaxconn=65535

# Reload all sysctl config
sudo sysctl --system

# Current user limits
ulimit -a

# Open file descriptors of a process
ls /proc/PID/fd | wc -l

# Current TCP connections
ss -tn state established | wc -l

Conclusion

With these optimizations, your Linux VPS is ready to handle big volumes. Typical observed benefits:

+50 to +150% TCP throughput thanks to BBR
2-5x more simultaneous connections thanks to file descriptors
Reduced latency on new TCP setups

Going further:

Enable NUMA awareness if your VPS has 8+ cores
Tune IRQ affinity to bind network interrupts
Use eBPF/XDP for ultra-fast networking (extreme cases)

Resources

Kernel sysctl documentation: https://www.kernel.org/doc/Documentation/sysctl/
TCP BBR (Google): https://github.com/google/bbr