SSH Secure Remote Access on Linux: Complete Guide

SSH (Secure Shell) is the cornerstone of remote Linux administration. Every sysadmin spends a significant portion of their day using SSH. Understanding SSH deeply — not just "how to connect" but the cryptographic model, the authentication mechanisms, the sshd_config directives, and the troubleshooting methodology — separates entry-level admins from RHCA-certified professionals.

Why SSH Replaced Telnet

Before SSH (invented in 1995 by Tatu Ylönen), Telnet and rsh/rlogin were used for remote administration. These protocols transmit everything — including passwords and commands — in plain text. Anyone with network access and a packet sniffer (tcpdump, Wireshark) could capture your root password.

SSH solves this by establishing an encrypted tunnel before any authentication happens. Even your username and password traverse an encrypted channel. SSH also provides:

Host authentication: Verifies you are connecting to the right server (not an impersonator)
Data integrity: Detects if data is tampered with in transit (via MACs — Message Authentication Codes)
Compression: Optional data compression for slow connections
Port forwarding: Tunnel arbitrary TCP connections through the encrypted SSH channel

SSH Architecture and Protocol Flow

SSH version 2 (the current standard) uses a layered protocol:

Connection Phase

TCP connection: Client opens TCP connection to server port 22.
Version exchange: Both sides announce their SSH protocol version.
Algorithm negotiation: Client and server negotiate: key exchange algorithm, host key algorithm, symmetric cipher, MAC algorithm, compression. They choose the highest-priority algorithm supported by both.
Key exchange: Using Diffie-Hellman (or ECDH), both sides generate a shared secret without ever transmitting it. This secret becomes the session encryption key. Even if someone records all traffic, they cannot decrypt it without knowing the private keys.
Server authentication: Server sends its host key signature. Client verifies against ~/.ssh/known_hosts. If the key has changed since last connection, SSH warns you (possible MITM attack).

Authentication Phase

Server announces supported authentication methods.
Client tries each method in order: publickey → keyboard-interactive → password.
If authentication succeeds, the session opens.

Authentication Methods In Depth

Password Authentication

The simplest method. The client sends the password encrypted through the SSH tunnel. Weaknesses:

Susceptible to brute-force attacks (attackers try millions of passwords)
Users often reuse weak passwords
Requires password management (changing, rotation)

Public Key Authentication (Key-Based)

The gold standard for SSH authentication. Based on asymmetric cryptography:

How it works:

You generate a key pair: private key (kept secret, never shared) and public key (can be shared freely).
You place the public key in ~/.ssh/authorized_keys on the server.
When connecting, the server generates a random challenge and encrypts it with your public key.
Only someone with the private key can decrypt this challenge.
The client decrypts the challenge and sends the response.
The server verifies the response — if correct, you are authenticated.

The private key never leaves your machine. Even if the server is compromised, the attacker cannot steal your private key.

# Generate RSA key pair (4096-bit, more secure):
# ssh-keygen -t rsa -b 4096 -C "admin@company.com"
# ssh-keygen -t ed25519 -C "admin@company.com"  # newer, smaller, equally secure

# Files created:
# ~/.ssh/id_rsa         private key (NEVER share this)
# ~/.ssh/id_rsa.pub     public key (copy this to servers)

# Protect private key with passphrase:
# The passphrase encrypts the private key file locally
# If someone steals your private key file, the passphrase protects it

# Copy public key to remote server:
# ssh-copy-id -i ~/.ssh/id_rsa.pub user@server
# This appends your public key to /home/user/.ssh/authorized_keys

# Manual method (if ssh-copy-id not available):
# cat ~/.ssh/id_rsa.pub | ssh user@server "mkdir -p ~/.ssh; cat >> ~/.ssh/authorized_keys; chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys"

# Use specific key:
# ssh -i ~/.ssh/id_rsa_server2 user@server2

Certificate-Based Authentication

Scales key-based auth across hundreds of servers. Instead of copying public keys to every server, you use a Certificate Authority (CA) to sign keys. Servers trust the CA certificate.

# Create CA key:
# ssh-keygen -t rsa -b 4096 -f /etc/ssh/ca_key

# Sign user public key:
# ssh-keygen -s /etc/ssh/ca_key -I "john_cert" -n john,root -V +1w ~/.ssh/id_rsa.pub

# Configure server to trust CA:
# echo "TrustedUserCAKeys /etc/ssh/ca_key.pub" >> /etc/ssh/sshd_config
# systemctl restart sshd

sshd_config — Complete Reference

The SSH daemon configuration file at /etc/ssh/sshd_config controls every aspect of the SSH service. The most security-critical directives:

# --- NETWORKING ---
Port 22                             # change to non-standard port to reduce brute force noise
# (example: Port 2222)
ListenAddress 0.0.0.0               # bind to all interfaces (default)
# ListenAddress 192.168.1.10        # bind only to specific IP

# --- AUTHENTICATION ---
PermitRootLogin no                  # CRITICAL: never allow direct root login
# Options: yes, no, prohibit-password (key only), forced-commands-only

PasswordAuthentication yes          # allow password auth (set to no for key-only)
PubkeyAuthentication yes            # enable public key auth (default yes)
AuthorizedKeysFile .ssh/authorized_keys  # where to find authorized keys

PermitEmptyPasswords no             # never allow empty passwords

MaxAuthTries 3                      # lock after 3 failed attempts (prevents brute force)
LoginGraceTime 60                   # seconds to authenticate before disconnect

# --- ACCESS CONTROL ---
AllowUsers alice bob admin          # whitelist (only these users can SSH in)
DenyUsers guest backup              # blacklist (these users cannot SSH in)
AllowGroups sysadmins developers    # only members of these groups
DenyGroups legacy noremote          # these group members cannot SSH

# Note: AllowUsers takes precedence over DenyUsers
# Note: If AllowUsers is set, ALL other users are denied by default

# --- SESSION ---
X11Forwarding yes                   # allow GUI forwarding (enable for workstations)
TCPKeepAlive yes                    # send keepalives to detect dead connections
ClientAliveInterval 300             # send keepalive every 5 minutes
ClientAliveCountMax 2               # disconnect after 2 missed keepalives (10 min idle)

# --- LOGGING ---
SyslogFacility AUTH
LogLevel INFO                       # options: QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG

# --- SECURITY ---
UseDNS no                           # skip reverse DNS lookup (faster connections)
StrictModes yes                     # check permissions of home dir and .ssh files

# After any change:
# sshd -t                           # test configuration syntax
# systemctl reload sshd             # apply without dropping active sessions

TCP Wrappers — IP-Based Access Control

TCP Wrappers adds an access control layer around services that are linked against libwrap (which includes sshd). It checks /etc/hosts.allow first, then /etc/hosts.deny.

# Processing order:
# 1. Check /etc/hosts.allow — if match found, ALLOW and stop
# 2. Check /etc/hosts.deny  — if match found, DENY and stop
# 3. No match in either file — ALLOW (permissive default)

# /etc/hosts.deny — examples:
sshd: ALL                          # deny all SSH by default
sshd: 192.168.5.0/24               # deny specific subnet
sshd: .evil.com                    # deny all hosts from evil.com domain
ALL: 10.0.0.100                    # deny this IP for ALL services

# /etc/hosts.allow — examples:
sshd: 192.168.1.0/24               # allow local subnet
sshd: 10.0.0.50 10.0.0.51          # allow specific IPs
sshd: ALL EXCEPT 192.168.5.0/24    # allow all except this subnet

# Typical hardened setup:
# /etc/hosts.deny:
ALL: ALL                            # deny everything by default

# /etc/hosts.allow:
sshd: 192.168.1.0/24               # allow SSH from management network
sshd: 10.0.0.0/8                   # allow internal network

# Check if a service uses TCP Wrappers:
# ldd /usr/sbin/sshd | grep libwrap

SSH Port Forwarding (Tunneling)

SSH can tunnel other protocols through its encrypted channel. This is useful for accessing services on remote networks or bypassing firewalls securely.

# Local port forwarding:
# Access a remote service via a local port
# Format: ssh -L local_port:remote_host:remote_port user@ssh_server
# ssh -L 3306:db.internal:3306 user@jumphost
# Now connect to localhost:3306 to reach db.internal:3306

# Remote port forwarding:
# Expose a local service on the remote server
# Format: ssh -R remote_port:local_host:local_port user@ssh_server
# ssh -R 8080:localhost:80 user@public-server
# Now port 8080 on public-server routes to your local port 80

# Dynamic port forwarding (SOCKS proxy):
# ssh -D 1080 user@ssh_server
# Configure browser to use SOCKS proxy localhost:1080
# All browser traffic goes through SSH tunnel

SSH File Transfer Tools

# scp — Secure Copy (uses SSH protocol):
# scp source user@host:dest          local to remote
# scp user@host:source dest          remote to local
# scp -r /local/dir user@host:/remote/  directory (recursive)
# scp -P 2222 file user@host:        non-standard port
# scp user1@host1:file user2@host2:  server to server

# sftp — Interactive Secure FTP:
# sftp user@host
sftp> ls                            # list remote files
sftp> lls                           # list local files
sftp> pwd                           # remote working dir
sftp> lpwd                          # local working dir
sftp> get remote_file local_file    # download
sftp> put local_file remote_file    # upload
sftp> mget *.log                    # download multiple files
sftp> bye                           # exit

# rsync — Efficient Incremental Sync:
# rsync -avz source/ user@host:/dest/
# Options: -a = archive (preserve everything), -v = verbose, -z = compress
# --delete        = delete files in dest that don't exist in source
# --progress      = show per-file progress
# --exclude=.git  = skip pattern
# --dry-run       = preview without making changes
# -e "ssh -p 2222" = use non-standard SSH port

# Full mirror (exact copy, deletes extras):
# rsync -avz --delete /data/ user@backup:/backup/data/

SSH Key Management Best Practices

# Use ssh-agent to hold decrypted private keys in memory:
# eval $(ssh-agent)                  # start agent
# ssh-add ~/.ssh/id_rsa              # load key (prompted for passphrase once)
# ssh-add -l                         # list loaded keys
# ssh-add -d ~/.ssh/id_rsa           # remove specific key
# ssh-add -D                         # remove all keys

# ~/.ssh/config — per-host SSH settings:
Host jumphost
    Hostname jump.company.com
    User admin
    Port 2222
    IdentityFile ~/.ssh/jump_rsa

Host internal-db
    Hostname db.internal
    User root
    ProxyJump jumphost                # route through jumphost
    IdentityFile ~/.ssh/db_rsa

# Now just: ssh internal-db          # automatically uses all settings

# File permissions (MUST be correct):
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa              # private key (owner read only)
chmod 644 ~/.ssh/id_rsa.pub          # public key
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/config

SSH Troubleshooting Methodology

# Step 1: Can you reach the server at all?
# ping server_ip                     # basic IP connectivity
# telnet server_ip 22                # check port 22 is open (or netcat)
# nc -zv server_ip 22

# Step 2: Verbose SSH output:
# ssh -v user@server                 # verbose (one level)
# ssh -vvv user@server               # maximum verbosity (three levels)
# Key lines to look for:
# "Connecting to" → resolves hostname
# "Server host key:" → gets server key
# "Authentications that can continue" → server-accepted methods
# "Trying private key" → attempting public key auth
# "Permission denied" → auth failed

# Step 3: Check server logs:
# journalctl -u sshd -n 50           # last 50 sshd log entries
# tail -f /var/log/secure             # RHEL 6
# grep "sshd" /var/log/secure | grep "Failed\|refused\|Invalid"

# Step 4: Check sshd configuration:
# sshd -t                            # syntax test
# sshd -T                            # dump effective config

# Common issues and fixes:
# "Host key verification failed"
# → Server key changed (or MITM). Remove old key:
# ssh-keygen -R hostname_or_ip

# "Permission denied (publickey)"
# → Wrong key, wrong authorized_keys permissions, or PermitRootLogin no
# → Check: ls -la ~/.ssh/authorized_keys (must be 600, not 644)
# → Check: AllowUsers in sshd_config — is your user listed?

# "Connection refused"
# → sshd not running or wrong port
# → systemctl start sshd; ss -tulnp | grep sshd

# "Too many authentication failures"
# → ssh-agent has too many keys loaded; add -o IdentitiesOnly=yes
# → Or: ssh-add -D to clear agent keys

Fail2ban — Automatic Brute Force Protection

# Install:
# yum install fail2ban -y

# Configure SSH jail:
# vim /etc/fail2ban/jail.local
[sshd]
enabled = true
port    = ssh
filter  = sshd
logpath = /var/log/secure
maxretry = 5                        # ban after 5 failures
bantime  = 3600                     # ban for 1 hour
findtime = 600                      # within 10 minutes

# Start:
# systemctl start fail2ban
# systemctl enable fail2ban

# Check banned IPs:
# fail2ban-client status sshd