SSH Hardening - Troubleshooting Guide

Common issues, solutions, and SSH lockout recovery procedures.

Quick Emergency Recovery

⚠️ LOCKED OUT OF SSH?

If you still have one working SSH session:

# Revert hardening config
sudo rm /etc/ssh/sshd_config.d/99-ssh-hardening.conf
sudo systemctl restart ssh

# Test from another terminal
ssh your-server

If completely locked out:
- Use console access (physical, VPS console, or rescue mode)
- Boot into single-user mode or rescue mode
- Remove: /etc/ssh/sshd_config.d/99-ssh-hardening.conf
- Restart SSH: systemctl restart ssh

Common Issues

1. Permission Denied (publickey)

Symptom:

Permission denied (publickey).

Causes:

SSH keys not configured
Wrong file permissions
Wrong username
Key type not accepted

Debug:

# Verbose SSH connection
ssh -vvv your-server

# Check auth log on server
sudo journalctl -u ssh -n 50 | grep "Failed\|Accepted"

# Verify authorized_keys exists
ls -la ~/.ssh/authorized_keys

Solutions:

Solution 1: Configure SSH keys

# On client:
ssh-copy-id your-server

# Or manually:
cat ~/.ssh/id_ed25519.pub | ssh your-server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Solution 2: Fix permissions

# On server:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chown -R $USER:$USER ~/.ssh

Solution 3: Check accepted key types

# On server:
sudo sshd -T | grep pubkeyacceptedkeytypes

# If your key type not listed, add to override:
echo "PubkeyAcceptedKeyTypes +rsa-sha2-512" | sudo tee /etc/ssh/sshd_config.d/50-custom.conf
sudo systemctl restart ssh

2. SSH Service Fails to Start

Symptom:

sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded
   Active: failed

Causes:

Syntax error in config
Conflicting directives
Invalid value

Debug:

# Check syntax
sudo sshd -t
# Shows specific error

# Check systemd logs
sudo journalctl -u ssh -n 50 --no-pager

# Test config manually
sudo /usr/sbin/sshd -ddd
# Runs SSH in debug mode (foreground)

Solutions:

Solution 1: Fix syntax error

# Find error line
sudo sshd -t -f /etc/ssh/sshd_config.d/99-ssh-hardening.conf

# Edit config
sudo nano /etc/ssh/sshd_config.d/99-ssh-hardening.conf

# Test again
sudo sshd -t

Solution 2: Remove conflicting override

# List all configs
ls /etc/ssh/sshd_config.d/*.conf

# Remove problematic override
sudo rm /etc/ssh/sshd_config.d/[problematic-file].conf

# Restart
sudo systemctl restart ssh

3. "Host key verification failed"

Symptom:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Cause: Server host keys changed (after running generate-hostkeys.sh or re-install)

Solution:

# Remove old entry
ssh-keygen -R your-server

# Or remove specific IP
ssh-keygen -R 192.168.1.100

# Accept new fingerprint on next connection
ssh your-server

Verify fingerprint (on server):

# Show current fingerprint
sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

4. X11 Forwarding Not Working

Symptom:

ssh -X your-server
# X11 forwarding request failed

Causes:

X11Forwarding disabled in config
X11UseLocalhost misconfigured
xauth not installed
DISPLAY variable not set

Debug:

# Check if X11 enabled
sudo sshd -T | grep x11forwarding

# Check xauth installed
which xauth

# Check DISPLAY variable
echo $DISPLAY

Solutions:

Solution 1: Enable X11 in override

# Deploy development override
sudo cp drop-ins/20-development.conf /etc/ssh/sshd_config.d/
sudo systemctl restart ssh

Solution 2: Install xauth

sudo apt update
sudo apt install xauth

Solution 3: Test X11

# Connect with X11
ssh -X your-server

# Test GUI app
xclock
# Should show clock window

5. TCP Port Forwarding Blocked

Symptom:

ssh -L 8080:localhost:80 your-server
# Warning: remote port forwarding failed

Cause: AllowTcpForwarding disabled (base config default)

Solution: Deploy gateway or development override

# For gateway (TCP forwarding)
sudo cp drop-ins/10-gateway.conf /etc/ssh/sshd_config.d/
sudo systemctl restart ssh

# Test
ssh -L 8080:localhost:80 your-server
curl http://localhost:8080

6. "Too many authentication failures"

Symptom:

Received disconnect from server: 2: Too many authentication failures

Cause: MaxAuthTries exceeded (set to 3)

Solutions:

Solution 1: Specify key explicitly

# Use specific key
ssh -i ~/.ssh/id_ed25519 your-server

# Or configure in ~/.ssh/config:
Host your-server
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

Solution 2: Temporarily increase MaxAuthTries (not recommended)

# On server (emergency only)
echo "MaxAuthTries 6" | sudo tee /etc/ssh/sshd_config.d/50-temp-maxauth.conf
sudo systemctl restart ssh

# Remove after fixing client config
sudo rm /etc/ssh/sshd_config.d/50-temp-maxauth.conf
sudo systemctl restart ssh

7. Connection Times Out

Symptom:

ssh your-server
# Connection timed out

Causes:

Firewall blocking SSH (port 22)
SSH not running
Wrong IP/hostname
Network issue

Debug:

# Check if port 22 open
nc -zv your-server 22

# Check SSH running (on server)
sudo systemctl status ssh

# Check firewall (on server)
sudo ufw status
sudo iptables -L -n | grep 22

Solutions:

Solution 1: Open firewall (on server)

sudo ufw allow 22/tcp
sudo ufw reload

Solution 2: Start SSH

sudo systemctl start ssh
sudo systemctl enable ssh

8. "sshd_config: line X: Bad configuration option"

Symptom:

/etc/ssh/sshd_config.d/99-ssh-hardening.conf: line 42: Bad configuration option: [option]

Causes:

Typo in directive name
Unsupported option (older OpenSSH version)
Wrong value format

Solutions:

Solution 1: Check OpenSSH version

ssh -V
# OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020

Solution 2: Comment out unsupported option

sudo nano /etc/ssh/sshd_config.d/99-ssh-hardening.conf

# Find problematic line and comment:
# UnsupportedOption value

sudo systemctl restart ssh

Solution 3: Check syntax

sudo sshd -t -f /etc/ssh/sshd_config.d/99-ssh-hardening.conf
# Shows exact line number

SSH Lockout Recovery

Scenario 1: One Working Session

Recovery:

Keep working session open (don't close!)

Revert config:

sudo rm /etc/ssh/sshd_config.d/99-ssh-hardening.conf
sudo systemctl restart ssh

Test from new terminal:
```
ssh your-server
```
If works, investigate issue and re-deploy correctly

Scenario 2: Complete Lockout (Console Access Available)

Recovery:

Access console (physical, VPS console, or rescue mode)
Login as root or sudo user

Remove hardening config:

rm /etc/ssh/sshd_config.d/99-ssh-hardening.conf
systemctl restart ssh

Test SSH from remote:
```
ssh your-server
```
If works, fix issue and re-deploy

Scenario 3: Complete Lockout (No Console Access)

Recovery:

Contact hosting provider for console access
Or use rescue mode (VPS/cloud providers)
Mount system disk

Edit config:

mount /dev/sda1 /mnt
rm /mnt/etc/ssh/sshd_config.d/99-ssh-hardening.conf

Reboot into normal mode
SSH should work with default config

Validation & Testing

Pre-Deployment Validation

Critical checks before deploying:

# 1. Syntax check
sudo sshd -t
echo "Exit code: $?"  # Must be 0

# 2. Baseline compliance
./scripts/validate-sshd-config.sh --config /etc/ssh/sshd_config

# 3. Permission check
stat -c "%a %n" /etc/ssh/sshd_config.d/*.conf
# Should be 644

# 4. SSH keys configured
ls -la ~/.ssh/authorized_keys
# Should exist and not be empty

# 5. Test connection with key
ssh -o PreferredAuthentications=publickey localhost
# Should succeed

Post-Deployment Testing

Safe testing workflow:

Keep existing SSH session open
Deploy config in that session
Test new connection from different terminal:
```
ssh your-server
```

If test fails: Revert in existing session

sudo rm /etc/ssh/sshd_config.d/99-ssh-hardening.conf
sudo systemctl restart ssh

Debugging Commands

Check Active Configuration

# Show all active settings
sudo sshd -T

# Filter specific settings
sudo sshd -T | grep -E "(password|pubkey|permit|forwarding)"

# Check specific directive
sudo sshd -T | grep allowtcpforwarding

Check Authentication Logs

# Recent auth attempts
sudo journalctl -u ssh -n 50

# Failed logins
sudo journalctl -u ssh | grep "Failed"

# Successful logins
sudo journalctl -u ssh | grep "Accepted"

# Real-time monitoring
sudo journalctl -u ssh -f

Check File Permissions

# SSH config files
ls -la /etc/ssh/sshd_config*
ls -la /etc/ssh/sshd_config.d/

# User SSH directory
ls -la ~/.ssh/

# Authorized keys
ls -la ~/.ssh/authorized_keys

Check SSH Service

# Status
sudo systemctl status ssh

# Is it running?
sudo systemctl is-active ssh

# Is it enabled?
sudo systemctl is-enabled ssh

# Restart
sudo systemctl restart ssh

Common Error Messages

"Permission denied, please try again."

Cause: Password auth disabled, no SSH keys
Fix: Configure SSH keys (ssh-copy-id)

"Connection closed by remote host"

Cause: MaxAuthTries exceeded, IP banned (fail2ban)
Fix: Use correct key, check fail2ban (sudo fail2ban-client status sshd)

"Port 22: Connection refused"

Cause: SSH not running, firewall blocking
Fix: Start SSH (sudo systemctl start ssh), open firewall

"No supported authentication methods available"

Cause: Client doesn't have accepted key type
Fix: Generate Ed25519 key (ssh-keygen -t ed25519)

Best Practices to Avoid Issues

Always validate before restarting: sudo sshd -t
Always keep one session open during testing
Always backup config before changes
Test from multiple clients (different OSes)
Use console access for initial deployment (if possible)
Document custom changes (comments in configs)
Monitor auth logs after deployment
Have recovery plan (console access, rescue mode)

Getting Help

Gather Diagnostic Info

# SSH version
ssh -V

# OpenSSH version
sudo sshd -V

# OS version
cat /etc/os-release

# Active config
sudo sshd -T

# Auth logs
sudo journalctl -u ssh -n 100 --no-pager

# File permissions
ls -la /etc/ssh/sshd_config*
ls -la ~/.ssh/

Report Issue

Include in bug report:

SSH version
OS version
Active config (sshd -T)
Auth logs (last 50 lines)
Exact error message
Steps to reproduce

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSH Hardening - Troubleshooting Guide

Quick Emergency Recovery

Common Issues

1. Permission Denied (publickey)

2. SSH Service Fails to Start

3. "Host key verification failed"

4. X11 Forwarding Not Working

5. TCP Port Forwarding Blocked

6. "Too many authentication failures"

7. Connection Times Out

8. "sshd_config: line X: Bad configuration option"

SSH Lockout Recovery

Scenario 1: One Working Session

Scenario 2: Complete Lockout (Console Access Available)

Scenario 3: Complete Lockout (No Console Access)

Validation & Testing

Pre-Deployment Validation

Post-Deployment Testing

Debugging Commands

Check Active Configuration

Check Authentication Logs

Check File Permissions

Check SSH Service

Common Error Messages

"Permission denied, please try again."

"Connection closed by remote host"

"Port 22: Connection refused"

"No supported authentication methods available"

Best Practices to Avoid Issues

Getting Help

Gather Diagnostic Info

Report Issue

See Also

FilesExpand file tree

TROUBLESHOOTING.md

Latest commit

History

TROUBLESHOOTING.md

File metadata and controls

SSH Hardening - Troubleshooting Guide

Quick Emergency Recovery

Common Issues

1. Permission Denied (publickey)

2. SSH Service Fails to Start

3. "Host key verification failed"

4. X11 Forwarding Not Working

5. TCP Port Forwarding Blocked

6. "Too many authentication failures"

7. Connection Times Out

8. "sshd_config: line X: Bad configuration option"

SSH Lockout Recovery

Scenario 1: One Working Session

Scenario 2: Complete Lockout (Console Access Available)

Scenario 3: Complete Lockout (No Console Access)

Validation & Testing

Pre-Deployment Validation

Post-Deployment Testing

Debugging Commands

Check Active Configuration

Check Authentication Logs

Check File Permissions

Check SSH Service

Common Error Messages

"Permission denied, please try again."

"Connection closed by remote host"

"Port 22: Connection refused"

"No supported authentication methods available"

Best Practices to Avoid Issues

Getting Help

Gather Diagnostic Info

Report Issue

See Also