Bash: Regular Expressions (Regex)

Regular expressions (regex) are patterns used to match text. They're incredibly powerful for searching logs, parsing data, extracting information, and validating input. Mastering regex will make you exponentially faster at data analysis during penetration tests.

What You'll Learn

Basic regex syntax
Character classes
Quantifiers
Anchors
Grouping and capturing
Practical grep usage
sed and awk with regex
Real pentesting scenarios

What Are Regular Expressions?

Regular Expression = Pattern matching language
- Match patterns in text
- Extract specific data
- Replace text
- Validate formats

Example:

# Find IP addresses
grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' file.txt

# Find email addresses
grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' file.txt

Basic Regex Syntax

Literal Characters

# Match exact string
grep "hello" file.txt

# Case-insensitive
grep -i "hello" file.txt

Special Characters (Metacharacters)

.  = Any single character (except newline)
^  = Start of line
$  = End of line
*  = 0 or more of previous character
+  = 1 or more of previous character (ERE)
?  = 0 or 1 of previous character (ERE)
|  = OR (ERE)
() = Grouping (ERE)
[] = Character class
\  = Escape special character

Character Classes

Predefined Classes

.      = Any character except newline
[abc]  = a, b, or c
[^abc] = NOT a, b, or c
[a-z]  = Lowercase letters
[A-Z]  = Uppercase letters
[0-9]  = Digits
[a-zA-Z] = Any letter
[a-zA-Z0-9] = Alphanumeric

Examples:

# Match any digit
grep '[0-9]' file.txt

# Match any letter
grep '[a-zA-Z]' file.txt

# Match NOT a digit
grep '[^0-9]' file.txt

# Match vowels
grep '[aeiou]' file.txt

# Match hex characters
grep '[0-9a-fA-F]' file.txt

POSIX Character Classes

[:alnum:]  = Alphanumeric [a-zA-Z0-9]
[:alpha:]  = Alphabetic [a-zA-Z]
[:digit:]  = Digits [0-9]
[:lower:]  = Lowercase [a-z]
[:upper:]  = Uppercase [A-Z]
[:space:]  = Whitespace [ \t\n\r]
[:punct:]  = Punctuation
[:xdigit:] = Hex digits [0-9a-fA-F]

Examples:

# Match any digit
grep '[[:digit:]]' file.txt

# Match any letter
grep '[[:alpha:]]' file.txt

# Match alphanumeric
grep '[[:alnum:]]' file.txt

Quantifiers

Basic Quantifiers

*      = 0 or more
+      = 1 or more (ERE: grep -E)
?      = 0 or 1 (ERE: grep -E)
{n}    = Exactly n times
{n,}   = n or more times
{n,m}  = Between n and m times

Examples:

# 0 or more 'a'
grep -E 'a*' file.txt

# 1 or more 'a'
grep -E 'a+' file.txt

# 0 or 1 'a'
grep -E 'a?' file.txt

# Exactly 3 digits
grep -E '[0-9]{3}' file.txt

# 3 or more digits
grep -E '[0-9]{3,}' file.txt

# Between 3 and 5 digits
grep -E '[0-9]{3,5}' file.txt

# Phone number (xxx-xxx-xxxx)
grep -E '[0-9]{3}-[0-9]{3}-[0-9]{4}' file.txt

Anchors

Line Anchors

^  = Start of line
$  = End of line

Examples:

# Lines starting with "Error"
grep '^Error' file.txt

# Lines ending with "failed"
grep 'failed$' file.txt

# Lines containing only "root"
grep '^root$' file.txt

# Empty lines
grep '^$' file.txt

# Non-empty lines
grep -v '^$' file.txt

Word Boundaries

\b = Word boundary
\B = Not word boundary

Examples:

# Match whole word "test"
grep '\btest\b' file.txt

# NOT whole word (test in contest)
grep '\Btest\B' file.txt

Grouping and Alternation

Grouping with ()

# Group patterns
grep -E '(abc)+' file.txt  # abc, abcabc, abcabcabc

# Capture groups
grep -E '(user|admin)' file.txt  # user OR admin

Alternation with |

# Match "cat" OR "dog"
grep -E 'cat|dog' file.txt

# Match "error" OR "warning" OR "critical"
grep -E 'error|warning|critical' file.txt

# Match file extensions
grep -E '\.(jpg|png|gif)$' file.txt

Practical Examples

1. IP Addresses

# Simple IP match
grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' file.txt

# More accurate (0-255)
grep -oE '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)' file.txt

2. Email Addresses

# Basic email
grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' file.txt

# Extract emails from text
grep -oE '\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b' file.txt

3. URLs

# HTTP/HTTPS URLs
grep -oE 'https?://[a-zA-Z0-9./?=_-]*' file.txt

# Domain names
grep -oE '[a-zA-Z0-9.-]+\.(com|net|org|edu|gov)' file.txt

4. Phone Numbers

# US format: (123) 456-7890
grep -oE '\([0-9]{3}\) [0-9]{3}-[0-9]{4}' file.txt

# Alternative: 123-456-7890
grep -oE '[0-9]{3}-[0-9]{3}-[0-9]{4}' file.txt

5. Dates

# YYYY-MM-DD
grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}' file.txt

# MM/DD/YYYY
grep -oE '[0-9]{2}/[0-9]{2}/[0-9]{4}' file.txt

6. Credit Cards

# 16-digit cards (spaces or dashes)
grep -oE '[0-9]{4}[- ]?[0-9]{4}[- ]?[0-9]{4}[- ]?[0-9]{4}' file.txt

7. Passwords (patterns)

# Find password= in configs
grep -i 'password[[:space:]]*=' config.txt

# API keys (32 hex characters)
grep -oE '[0-9a-fA-F]{32}' file.txt

# JWT tokens
grep -oE 'eyJ[a-zA-Z0-9_-]*\.[a-zA-Z0-9_-]*\.[a-zA-Z0-9_-]*' file.txt

grep with Regex

Basic vs Extended Regex

Basic Regex (BRE):

grep 'pattern' file.txt
# Metacharacters: . ^ $ * [ ] \
# Need escaping: + ? | ( )

Extended Regex (ERE):

grep -E 'pattern' file.txt
# Or: egrep 'pattern' file.txt
# Metacharacters: . ^ $ * + ? | ( ) [ ] \
# No escaping needed for + ? | ( )

Perl Regex (PCRE):

grep -P 'pattern' file.txt
# Most powerful, Perl-compatible
# Supports lookahead/lookbehind

Useful grep Flags

-E  = Extended regex
-P  = Perl regex
-i  = Case-insensitive
-v  = Invert match
-o  = Only matched part
-n  = Line numbers
-c  = Count matches
-l  = Files with matches
-r  = Recursive
-A3 = 3 lines after
-B3 = 3 lines before
-C3 = 3 lines context (before+after)

sed with Regex

Find and Replace

# Basic substitution
sed 's/old/new/' file.txt

# Replace all occurrences (g = global)
sed 's/old/new/g' file.txt

# Case-insensitive
sed 's/old/new/gI' file.txt

# Edit file in-place
sed -i 's/old/new/g' file.txt

Examples:

# Replace IP addresses
sed 's/192\.168\.1\./10.0.0./g' file.txt

# Remove leading whitespace
sed 's/^[[:space:]]*//' file.txt

# Remove trailing whitespace
sed 's/[[:space:]]*$//' file.txt

# Extract text between quotes
sed -n 's/.*"\(.*\)".*/\1/p' file.txt

# Delete lines matching pattern
sed '/pattern/d' file.txt

# Delete empty lines
sed '/^$/d' file.txt

awk with Regex

Pattern Matching

# Print lines matching pattern
awk '/pattern/' file.txt

# Print specific field if pattern matches
awk '/error/ {print $1, $3}' file.txt

# Regex in condition
awk '$3 ~ /[0-9]+/' file.txt

Examples:

# Extract IPs from log
awk '/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/ {print $1}' access.log

# Print lines where field 3 is a number
awk '$3 ~ /^[0-9]+$/' file.txt

# Count 404 errors
awk '/404/ {count++} END {print count}' access.log

Practical Pentesting Scenarios

Scenario 1: Extract IPs from Logs

# Extract all IPs
grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' access.log | sort -u

# Extract IPs with failed logins
grep "Failed password" /var/log/auth.log | grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | sort | uniq -c | sort -rn

Scenario 2: Find Credentials

# Search for password patterns
grep -riE 'password[[:space:]]*=[[:space:]]*[^;]+' /var/www/

# API keys
grep -rioE '[0-9a-fA-F]{32}' /var/www/

# Database connection strings
grep -riE '(mysql|postgres|mongodb)://[^"]+' /var/www/

Scenario 3: Parse Nmap Output

# Extract open ports
grep "open" nmap.txt | grep -oE '^[0-9]+' | sort -n

# Extract service versions
grep -oE '[0-9]+/tcp.*open.*' nmap.txt

# Find specific services
grep -iE '(ssh|http|ftp|mysql)' nmap.txt

Scenario 4: Analyze Web Logs

# Find SQL injection attempts
grep -iE "(union|select|insert|update|delete|drop|exec|script)" access.log

# Path traversal attempts
grep -E '\.\./\.\./' access.log

# XSS attempts
grep -iE '(<script|<iframe|javascript:|onerror=)' access.log

# Admin login attempts
grep -E '/admin|/login|/wp-admin' access.log | grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | sort | uniq -c | sort -rn

Scenario 5: Extract Subdomains

# From certificate transparency
curl -s "https://crt.sh/?q=%25.example.com&output=json" | grep -oE '"name_value":"[^"]*"' | cut -d'"' -f4 | sort -u

# From DNS enumeration output
grep -oE '[a-zA-Z0-9.-]+\.example\.com' subdomains.txt | sort -u

Practical Exercises

Exercise 1: Match Patterns

# Create test file
cat << EOF > test.txt
192.168.1.1
[email protected]
10.0.0.1
[email protected]
Invalid IP: 999.999.999.999
EOF

# Extract IPs
grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' test.txt

# Extract emails
grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' test.txt

Exercise 2: Log Analysis

# Extract failed SSH attempts
sudo grep "Failed password" /var/log/auth.log | \
  grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | \
  sort | uniq -c | sort -rn | head -10

Exercise 3: Find and Replace

# Create file
echo "The password is: password123" > secrets.txt

# Replace passwords (hide them)
sed 's/password[^[:space:]]*/[REDACTED]/gi' secrets.txt

Exercise 4: Extract Data

# Parse /etc/passwd
awk -F: '/\/bin\/bash$/ {print $1}' /etc/passwd

# Extract UIDs > 1000
awk -F: '$3 > 1000 {print $1, $3}' /etc/passwd

Quick Reference

Basic Patterns:
.       = Any character
^       = Start of line
$       = End of line
*       = 0 or more
+       = 1 or more (ERE)
?       = 0 or 1 (ERE)
|       = OR (ERE)

Character Classes:
[abc]   = a, b, or c
[^abc]  = NOT a, b, or c
[a-z]   = Range
[0-9]   = Digits
.       = Any character

Quantifiers:
{n}     = Exactly n
{n,}    = n or more
{n,m}   = Between n and m

Common Patterns:
IP: [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}
Email: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
URL: https?://[a-zA-Z0-9./?=_-]*
Phone: [0-9]{3}-[0-9]{3}-[0-9]{4}

grep Options:
grep -E   = Extended regex
grep -P   = Perl regex
grep -o   = Only matched
grep -i   = Case-insensitive
grep -v   = Invert match

Resources

Man Pages:

man grep
man sed
man awk
man regex

Online Tools:

Regex101: https://regex101.com/ (test regex)
RegExr: https://regexr.com/ (interactive regex)
Debuggex: https://www.debuggex.com/ (visualize regex)

Practice:

RegexOne: https://regexone.com/
HackerRank Regex: https://www.hackerrank.com/domains/regex

Next Steps

Practice daily with real logs
Use regex101.com to test patterns
Learn sed and awk for advanced text processing
Master grep flags (-E, -P, -o, -A, -B, -C)
Build regex library for common patterns (IPs, emails, etc.)

Regex is one of the most powerful tools in your arsenal. Master it and you'll parse data 100x faster!

"Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems." - Jamie Zawinski

(But seriously, learn regex. It's worth it.)