The TCP Three-Way Handshake

The TCP three-way handshake is one of the most fundamental concepts in networking. It's the process that establishes a reliable connection between two devices before they exchange data. Understanding this handshake is critical for cybersecurity—many attacks exploit or manipulate this process, from port scanning to SYN floods to session hijacking.

What You'll Learn

How the three-way handshake works step-by-step
TCP flags and their meanings
Sequence and acknowledgment numbers
Connection establishment and termination
How attackers exploit the handshake
SYN flood attacks and defenses
Port scanning techniques
How to analyze handshakes in Wireshark

Why TCP Needs a Handshake

UDP is connectionless—just send packets and hope they arrive.

TCP is connection-oriented—it guarantees:

Reliability: Packets arrive
Order: Packets arrive in sequence
Error checking: Corrupted packets are retransmitted

To provide these guarantees, TCP must establish a connection first. That's where the three-way handshake comes in.

The Three-Way Handshake Explained

Client                          Server
  │                               │
  │         SYN (seq=100)         │
  │──────────────────────────────>│ "Let's connect! My sequence is 100"
  │                               │
  │      SYN-ACK (seq=200,        │
  │       ack=101)                │
  │<──────────────────────────────│ "OK! My sequence is 200, I got your 100"
  │                               │
  │         ACK (seq=101,         │
  │          ack=201)             │
  │──────────────────────────────>│ "Great! I got your 200, let's go!"
  │                               │
  │    [Connection Established]   │
  │                               │
  │         Data Transfer         │
  │<─────────────────────────────>│

Step 1: SYN (Synchronize)

Client → Server

TCP Header:
  Source Port: 45678 (random client port)
  Dest Port: 80 (server's HTTP port)
  Flags: SYN
  Sequence Number: 100 (random initial number)
  Acknowledgment: 0 (not acknowledging anything yet)

What's happening:

Client says "I want to connect!"
Client picks random initial sequence number (ISN)
Sets SYN flag to 1

Real example:

You type: http://example.com
Your browser initiates connection to example.com:80
Sends: SYN packet

Step 2: SYN-ACK (Synchronize-Acknowledge)

Server → Client

TCP Header:
  Source Port: 80
  Dest Port: 45678
  Flags: SYN, ACK (both flags set!)
  Sequence Number: 200 (server's random ISN)
  Acknowledgment: 101 (client's ISN + 1)

What's happening:

Server says "OK, I received your SYN!"
Server picks its own random ISN
Server acknowledges client's sequence (100 + 1 = 101)
Sets both SYN and ACK flags

Step 3: ACK (Acknowledge)

Client → Server

TCP Header:
  Source Port: 45678
  Dest Port: 80
  Flags: ACK
  Sequence Number: 101
  Acknowledgment: 201 (server's ISN + 1)

What's happening:

Client says "I got your SYN-ACK!"
Client acknowledges server's sequence (200 + 1 = 201)
Connection is now ESTABLISHED
Data transfer can begin

TCP Flags Deep Dive

TCP header has 6 important flags (control bits):

┌─────┬──────────────────────────────────────┐
│ URG │ Urgent pointer field significant     │
│ ACK │ Acknowledgment field significant     │
│ PSH │ Push function (send data now)        │
│ RST │ Reset connection                     │
│ SYN │ Synchronize sequence numbers         │
│ FIN │ Finish (no more data)                │
└─────┴──────────────────────────────────────┘

Flag Combinations:

Flags	Meaning
SYN	Start connection
SYN-ACK	Acknowledge connection request
ACK	Acknowledge data
FIN	Close connection gracefully
RST	Abort connection immediately
PSH-ACK	Data delivery + acknowledgment

Sequence and Acknowledgment Numbers

Why Random Initial Sequence Numbers?

Security: Prevents attackers from easily predicting sequence numbers and hijacking connections.

Old vulnerability: Predictable ISNs allowed session hijacking

Attacker could predict next sequence number
Inject packets pretending to be legitimate client/server

Modern systems: Use cryptographically random ISNs

How Sequence Numbers Work:

Client ISN: 1000
Server ISN: 2000

Client sends: seq=1000, data="GET / HTTP/1.1" (16 bytes)
Server acknowledges: ack=1016 (1000 + 16)

Server sends: seq=2000, data="HTTP/1.1 200 OK" (1000 bytes)
Client acknowledges: ack=3000 (2000 + 1000)

Each byte of data increments the sequence number.

Connection Termination: Four-Way Handshake

Closing a TCP connection is a bit more involved:

Client                          Server
  │         FIN (seq=500)        │
  │──────────────────────────────>│ "I'm done sending data"
  │                               │
  │         ACK (ack=501)         │
  │<──────────────────────────────│ "OK, I got that"
  │                               │
  │         FIN (seq=800)         │
  │<──────────────────────────────│ "I'm done too"
  │                               │
  │         ACK (ack=801)         │
  │──────────────────────────────>│ "OK, goodbye!"
  │                               │
  │    [Connection Closed]        │

Why 4 steps instead of 3?

Each side must independently close its transmission
Full-duplex connection (data flows both ways)

Alternative: RST

Immediate termination
No graceful close
Used for errors or forced closure

TCP State Diagram

CLOSED
  │
  ├─ (passive open) ──> LISTEN
  │                       │
  └─ (active open) ───> SYN-SENT
                          │
                          ├─ (receive SYN-ACK, send ACK) ──> ESTABLISHED
                          │
      LISTEN ─────────────┤
        │                 │
        └─ (receive SYN, send SYN-ACK) ──> SYN-RECEIVED
                                              │
                                              └─ (receive ACK) ──> ESTABLISHED
                                                                      │
                                                                      │
                                                          ┌───────────┴──────────┐
                                                          │                      │
                                                    (close, send FIN)      (receive FIN)
                                                          │                      │
                                                       FIN-WAIT-1           CLOSE-WAIT
                                                          │                      │
                                                   (receive ACK)           (close, send FIN)
                                                          │                      │
                                                       FIN-WAIT-2             LAST-ACK
                                                          │                      │
                                                   (receive FIN)           (receive ACK)
                                                          │                      │
                                                       TIME-WAIT ───────────> CLOSED
                                                          │
                                                    (timeout)
                                                          │
                                                       CLOSED

Viewing TCP Connections

Linux:

# View all TCP connections
netstat -an | grep tcp
ss -tan

# View established connections
netstat -ant | grep ESTABLISHED
ss -tan state established

# View listening ports
netstat -tln
ss -tln

# Modern replacement for netstat
ss -tuln

Windows:

# View all connections
netstat -an

# View established connections
netstat -an | findstr ESTABLISHED

# View listening ports
netstat -an | findstr LISTENING

# Detailed connection info
netstat -ano  # Shows PID

Connection States:

# Watch connection progress
watch -n 1 'netstat -ant | grep 192.168.1.5'

# You might see:
SYN-SENT        # Waiting for SYN-ACK
SYN-RECEIVED    # Waiting for final ACK
ESTABLISHED     # Connection active
FIN-WAIT-1      # Closing connection
TIME-WAIT       # Waiting before full close
CLOSED          # Connection terminated

Wireshark Analysis

Let's capture and analyze a real three-way handshake:

Capture Setup:

# Start Wireshark
sudo wireshark

# Filter for specific connection
tcp.port == 80 and ip.addr == 93.184.216.34

# Generate traffic
curl http://example.com

What You'll See:

Packet 1: SYN

Flags: 0x002 (SYN)
    .... .... ..1. = Syn: Set
Sequence Number: 0 (relative)
Acknowledgment: 0
Window: 65535

Packet 2: SYN-ACK

Flags: 0x012 (SYN, ACK)
    .... ...1 .... = Acknowledgment: Set
    .... .... ..1. = Syn: Set
Sequence Number: 0 (relative)
Acknowledgment: 1 (relative)
Window: 65535

Packet 3: ACK

Flags: 0x010 (ACK)
    .... ...1 .... = Acknowledgment: Set
Sequence Number: 1 (relative)
Acknowledgment: 1 (relative)

Follow TCP Stream: Right-click packet → Follow → TCP Stream

See complete conversation
HTTP request and response

Attacks Exploiting TCP Handshake

1. SYN Flood Attack (DoS)

How it works:

Attacker sends thousands of SYN packets
Server responds with SYN-ACK
Attacker NEVER sends final ACK
Server keeps half-open connections waiting
Eventually: Server runs out of connection slots
Result: Legitimate users can't connect (DoS)

Execution:

# Using hping3
hping3 -S --flood -p 80 192.168.1.5

# Using nmap
nmap -sS --scanflags SYN --scan-delay 0 192.168.1.5

Defense:

SYN Cookies: Don't allocate resources until final ACK
Rate limiting: Limit SYNs per second
Firewall rules: Drop excessive SYNs from same source
Increase backlog: Allow more half-open connections

Enable SYN cookies (Linux):

# Check if enabled
sysctl net.ipv4.tcp_syncookies

# Enable
sysctl -w net.ipv4.tcp_syncookies=1

# Make permanent
echo "net.ipv4.tcp_syncookies = 1" >> /etc/sysctl.conf

2. TCP Port Scanning

Different scan types manipulate the handshake:

SYN Scan (Stealth Scan)

nmap -sS 192.168.1.5

How it works:

Scanner → Target: SYN
Target → Scanner: SYN-ACK (port open) or RST (port closed)
Scanner → Target: RST (abort connection)

Why "stealth":

Doesn't complete handshake
May not be logged by some systems
Faster than full connect

TCP Connect Scan

nmap -sT 192.168.1.5

How it works:

Complete three-way handshake
Then immediately close connection (FIN)

When to use:

Don't have raw socket privileges
Want to be less suspicious (complete connections)

FIN Scan

nmap -sF 192.168.1.5

How it works:

Send FIN packet (no SYN)
Open port: No response
Closed port: RST

Bypasses some firewalls that only block SYN packets.

XMAS Scan

nmap -sX 192.168.1.5

How it works:

Send packet with FIN, PSH, URG flags (lit up like Christmas tree)
Open port: No response
Closed port: RST

NULL Scan

nmap -sN 192.168.1.5

How it works:

Send packet with NO flags
Open port: No response
Closed port: RST

3. TCP Session Hijacking

Old attack (harder now with random ISNs):

1. Attacker sniffs connection between Client and Server
2. Observes sequence numbers
3. Predicts next sequence number
4. Injects packet with correct seq/ack
5. Server thinks it's from legitimate client

Modern defenses:

Random ISNs
Encryption (TLS/SSL)
IPsec

4. TCP Reset Attack

How it works:

Attacker sends RST packet with correct seq/ack
Connection immediately terminated

Use case: Force disconnect users

Example:

# tcpkill terminates TCP connections
tcpkill -i eth0 host 192.168.1.5

Real-world: Some countries use RST injection to block websites.

Analyzing Handshake Timing

Normal Timing:

SYN          → (RTT/2)
SYN-ACK      ← (RTT/2)
ACK          → (RTT/2)
Data         ↔

Total setup time: ~1.5 × RTT

RTT (Round Trip Time): Time for packet to reach destination and return.

# Measure RTT with ping
ping google.com

# Output:
# 64 bytes from google.com: icmp_seq=1 ttl=117 time=15.2 ms
#                                                    ↑
#                                                   RTT

Detecting SYN Flood:

# Check for many SYN-RECEIVED states
netstat -an | grep SYN_RECV | wc -l

# If abnormally high (hundreds/thousands), likely under attack

TCP Handshake Options

During handshake, both sides can negotiate options:

MSS (Maximum Segment Size):

SYN: MSS=1460
"I can receive segments up to 1460 bytes"

Window Scaling:

SYN: Window Scale=7
"Multiply window size by 2^7 for better performance"

Timestamps:

SYN: Timestamp
"For RTT calculation and protection against wrapped sequence numbers"

Selective Acknowledgment (SACK):

SYN: SACK Permitted
"I can acknowledge non-contiguous data blocks"

View in Wireshark:

TCP Options section in SYN packet

Try It Yourself

Exercise 1: Capture Your Own Handshake

# Start Wireshark
sudo wireshark

# Filter
tcp.flags.syn == 1

# Generate connection
curl http://example.com

# Observe:
# - Packet 1: SYN only
# - Packet 2: SYN + ACK
# - Packet 3: ACK only

Exercise 2: Check Connection States

# Terminal 1: Start web server
python3 -m http.server 8080

# Terminal 2: Watch connections
watch -n 1 'ss -tan | grep 8080'

# Terminal 3: Connect
curl http://localhost:8080

# Observe state changes:
# LISTEN → SYN-RECEIVED → ESTABLISHED → TIME-WAIT → CLOSED

Exercise 3: SYN Scan

# Full three-way handshake (-sT)
sudo nmap -sT -p 80 example.com

# Stealth SYN scan (-sS)
sudo nmap -sS -p 80 example.com

# Compare:
# - Speed
# - Logs generated
# - Packets captured in Wireshark

Exercise 4: Identify Port States

# Scan with reason
nmap -sS --reason 192.168.1.5

# Output examples:
# 80/tcp   open     syn-ack ttl 64
# 22/tcp   closed   reset ttl 64
# 443/tcp  filtered no-response

Interpretation:

syn-ack: Server responded with SYN-ACK (port open)
reset: Server sent RST (port closed)
no-response: Packet dropped (likely firewall)

Handshake Timing Attacks

Timing Analysis:

# Measure response time
hping3 -S -p 80 -c 1 192.168.1.5

# Compare:
# Open port: Fast SYN-ACK response
# Closed port: Fast RST response
# Filtered port: No response (timeout)

Slow Handshake Attack (Slowloris):

Open many connections
Send partial HTTP requests
Keep connections alive
Server runs out of connection slots
DoS condition

Defense:

Connection timeouts
Rate limiting per IP
Minimum data rate requirements

Real-World Scenarios

Scenario 1: Website Won't Load

Troubleshoot:

# Can you reach the server?
ping example.com

# Can you complete TCP handshake?
telnet example.com 80
# or
nc -vz example.com 80

# If handshake fails:
# - Port closed/filtered
# - Firewall blocking
# - Server down

Scenario 2: Firewall Testing

# Test if firewall allows SYN
hping3 -S -p 443 firewall.example.com

# Responses:
# SA (SYN-ACK): Port open, firewall allows
# RA (RST-ACK): Port closed, but reachable
# No response: Firewall dropping packets

Scenario 3: Detecting Port Knocking

Some servers require "knocking" on specific ports before opening:

# Knock sequence: 7000, 8000, 9000
nc -z server 7000
nc -z server 8000
nc -z server 9000

# Now try the real port
nc -vz server 22  # Should be open now

Summary Diagram

Three-Way Handshake:
┌────────┐                              ┌────────┐
│ Client │                              │ Server │
└───┬────┘                              └───┬────┘
    │                                       │
    │  1. SYN (seq=X)                      │
    │─────────────────────────────────────>│
    │     "Let's connect!"                 │
    │                                       │
    │  2. SYN-ACK (seq=Y, ack=X+1)         │
    │<─────────────────────────────────────│
    │     "OK, I'm ready!"                 │
    │                                       │
    │  3. ACK (seq=X+1, ack=Y+1)           │
    │─────────────────────────────────────>│
    │     "Great, let's go!"               │
    │                                       │
    │═════════ ESTABLISHED ═══════════════│
    │                                       │
    │  Data Transfer                       │
    │<────────────────────────────────────>│
    │                                       │

Resources

Tools:

Wireshark: Analyze handshakes - wireshark.org
hping3: Craft custom TCP packets
nmap: Port scanning with various handshake techniques
tcpdump: Command-line packet capture

Reading:

RFC 793: TCP Specification
Cloudflare: TCP Handshake

Practice:

Capture handshakes in Wireshark
Experiment with different nmap scan types
Set up your own SYN flood defense (in a lab!)

Next Steps

Master Wireshark filters for TCP analysis
Learn port scanning techniques in depth
Understand firewalls and how they inspect handshakes
Study SYN flood defenses (SYN cookies, rate limiting)
Explore TCP options (window scaling, timestamps, SACK)

The three-way handshake is happening millions of times per second across the internet. Understanding it deeply will help you troubleshoot connection issues, scan networks efficiently, and detect/prevent attacks!

"SYN, SYN-ACK, ACK - The digital handshake that connects the world."