Log Analysis & Threat Hunting

⚡ Quick Start

Need to investigate something right now? These are the first commands you'd run when triaging a suspicious system — checking for unauthorized access, privilege escalation, and persistence. Linux commands use grep against log files; Windows uses PowerShell's Get-WinEvent to query the Event Log database:

# === LINUX: Quick investigation ===
# Check for failed login attempts (brute force)
grep "Failed password" /var/log/auth.log | tail -20

# Check for successful logins (compromised accounts)
grep "Accepted" /var/log/auth.log | tail -20

# Check for sudo usage (privilege escalation)
grep "sudo:" /var/log/auth.log | tail -20

# Recently modified system files
find /etc -mtime -1 -type f 2>/dev/null

# Check for new user accounts
grep "useradd\|adduser\|new user" /var/log/auth.log

# === WINDOWS: Quick investigation (PowerShell) ===
# Failed logins in the last 24 hours
Get-WinEvent -FilterHashtable @{LogName='Security';Id=4625;StartTime=(Get-Date).AddDays(-1)}

# Successful logins
Get-WinEvent -FilterHashtable @{LogName='Security';Id=4624;StartTime=(Get-Date).AddDays(-1)}

# New processes created (requires audit policy)
Get-WinEvent -FilterHashtable @{LogName='Security';Id=4688;StartTime=(Get-Date).AddHours(-4)}

# New services installed (common malware persistence)
Get-WinEvent -FilterHashtable @{LogName='System';Id=7045;StartTime=(Get-Date).AddDays(-7)}

That's the quick and dirty version. Now let's build a deep understanding of log analysis that will make you effective at finding attacks.

📝 Why Logs Matter

Logs are timestamped text records that systems generate automatically whenever something happens — a user logs in, a process starts, a file is modified, a network connection is made, or an error occurs. They're the black box recorder of your systems: a chronological trail of every event, stored in files (like /var/log/auth.log) or databases (like Windows Event Logs). Different systems generate different types of logs: authentication logs track who logged in and when, web access logs track every HTTP request to your web server, firewall logs track allowed and blocked network traffic, and application logs track what your software does internally. Without logs, you're blind — you can't detect attacks, investigate incidents, or prove compliance.

The average time to detect a data breach (known as "dwell time") is 194 days (IBM 2024). Organizations with mature log analysis and threat hunting programs detect breaches in weeks, not months. The difference between a minor incident and a catastrophic breach often comes down to how quickly you can find the attacker in your logs.

Logs serve three critical functions:

• Detection: Real-time alerting on suspicious events
• Investigation: Reconstructing what happened during an incident
• Compliance: Proving you meet regulatory requirements (PCI-DSS, HIPAA, SOX)

🐧 Linux Log Sources

Linux logs are primarily stored in /var/log/ and managed by either the traditional syslog daemon or systemd's journal. Understanding which log contains what information is fundamental to investigation.

Key Log Files

Authentication logs — THE most important log for security. Contains: SSH logins, sudo usage, su commands, PAM events, user creation/deletion, password changes.

/var/log/auth.log        # Debian/Ubuntu
/var/log/secure          # RHEL/CentOS/Fedora

System logs — general system events. Contains: service start/stop, hardware events, kernel messages, network interface changes.

/var/log/syslog          # Debian/Ubuntu
/var/log/messages        # RHEL/CentOS

Kernel logs — Contains: hardware errors, kernel module loading (rootkit indicator), firewall messages, storage events.

/var/log/kern.log        # Kernel-specific messages
dmesg                    # Kernel ring buffer (current boot)

Application-specific logs:

/var/log/apache2/        # Apache web server
/var/log/nginx/          # Nginx web server
/var/log/mysql/          # MySQL database
/var/log/postgresql/     # PostgreSQL database

Other important logs:

# Cron logs
/var/log/cron            # RHEL/CentOS
grep CRON /var/log/syslog  # Debian/Ubuntu

# Package management
/var/log/apt/            # Debian/Ubuntu package installs
/var/log/yum.log         # Legacy (RHEL 7 and earlier). Modern RHEL 8+/Fedora use /var/log/dnf.log

# Boot and reboot history
/var/log/boot.log
last -x reboot

Journalctl — The Modern Way

On systems using systemd, journalctl provides powerful structured log access:

# View all logs (newest first)
journalctl -r

# View logs from current boot (use -b -1 for previous boot)
journalctl -b

# View logs for a specific service
journalctl -u sshd
journalctl -u nginx
journalctl -u cron

# Filter by time range
journalctl --since "2024-03-15 14:00" --until "2024-03-15 16:00"
journalctl --since "1 hour ago"
journalctl --since yesterday

# Filter by priority (0=emerg to 7=debug)
journalctl -p err        # Errors and above
journalctl -p warning    # Warnings and above

# Follow logs in real-time (like tail -f)
journalctl -f
journalctl -f -u sshd    # Follow specific service

# Output as JSON (great for parsing)
journalctl -o json-pretty
journalctl -u sshd -o json --since "1 hour ago" | jq '.MESSAGE'

# Disk usage of journal
journalctl --disk-usage

# Kernel messages only
journalctl -k

💡 Pro Tip: journalctl supports structured data. Use -o json and pipe to jq for powerful filtering: journalctl -u sshd -o json | jq 'select(._COMM == "sshd") | .MESSAGE'. This is far more precise than grepping text logs.

🪟 Windows Event Logs

Windows Event Logs are stored in .evtx format and accessed through Event Viewer or PowerShell. They're organized into channels, with three being critical for security:

Key Event Log Channels

• Security Log — Authentication, authorization, audit events. Contains: Logon/logoff, object access, privilege use, policy changes. Requires audit policies to be enabled.
• System Log — OS component events. Contains: Service installs, driver loading, system state changes.
• Application Log — Application errors, warnings, information.
• Sysmon Log (if installed) — GOLD STANDARD for security. Sysmon (System Monitor) is a free Microsoft tool that dramatically improves Windows logging. Contains: Process creation with command lines, network connections, file creation, registry modification, DNS queries. Must be installed and configured separately.
• PowerShell Logs (Microsoft-Windows-PowerShell/Operational) — Script block logging, module logging, transcription.
• Windows Defender Logs — Malware detections, remediation actions.

PowerShell Event Log Queries

# List all event log channels
Get-WinEvent -ListLog * | Where-Object {$_.RecordCount -gt 0} | Sort-Object RecordCount -Desc

# Query specific event IDs
Get-WinEvent -FilterHashtable @{
    LogName='Security'
    Id=4624,4625
    StartTime=(Get-Date).AddDays(-1)
} | Format-Table TimeCreated, Id, Message -Wrap

# Search for specific text in events
Get-WinEvent -FilterHashtable @{LogName='Security'} |
    Where-Object {$_.Message -like "*administrator*"} |
    Select-Object TimeCreated, Id, Message -First 20

# Export events for analysis
Get-WinEvent -FilterHashtable @{LogName='Security';StartTime=(Get-Date).AddDays(-7)} |
    Export-Csv security_events.csv -NoTypeInformation

# Count events by ID (find the noisiest events)
Get-WinEvent -FilterHashtable @{LogName='Security';StartTime=(Get-Date).AddDays(-1)} |
    Group-Object Id | Sort-Object Count -Desc | Select-Object Count, Name -First 20

# Sysmon: Process creation with command lines
Get-WinEvent -FilterHashtable @{LogName='Microsoft-Windows-Sysmon/Operational';Id=1} |
    Select-Object TimeCreated, @{N='CommandLine';E={$_.Properties[10].Value}} -First 20

🔑 Critical Windows Event IDs

These are the Event IDs every analyst must know. Memorize the top 15 — they appear in virtually every investigation.

Authentication Events (Security Log)

4624 — Successful Logon. Key fields: Logon Type, Account Name, Source IP.

4625 — Failed Logon. Key fields: Failure Reason, Account Name, Source IP.

Other authentication events:

• 4634 — Logoff (session ended)
• 4647 — User-initiated logoff
• 4648 — Logon with explicit credentials (RunAs, pass-the-hash). ⚠️ KEY indicator of lateral movement
• 4672 — Special privileges assigned to logon (admin login)

Account Management Events

• 4720 — User account created. ⚠️ Unexpected creation = potential backdoor
• 4722 — User account enabled
• 4724 — Password reset attempted
• 4725 — User account disabled
• 4726 — User account deleted
• 4728 — Member added to security-enabled global group
• 4732 — Member added to security-enabled local group. ⚠️ Watch for Administrators, Domain Admins
• 4738 — User account changed
• 4756 — Member added to universal group

Process & Service Events

• 4688 — New process created (Security log). ⚠️ Must enable "Audit Process Creation" AND "Include command line in process creation events"
• 7045 — New service installed (System log). ⚠️ Common persistence mechanism. Watch for: random names, paths in Temp directories, services running as SYSTEM

Defense Evasion Events

• 1102 — Audit log cleared. ⚠️ CRITICAL: Attackers clear logs to cover tracks. If you see this, something bad happened.
• 4616 — System time changed. Attackers change time to manipulate log timestamps.
• 4719 — System audit policy changed. Attackers may disable auditing.

💡 Pro Tip: Install Sysmon (System Monitor from Sysinternals) on every Windows system. It provides vastly better logging than native Windows events — including command-line arguments, parent process tracking, network connections, file hash logging, and DNS queries. It's free and transforms your detection capability.

Event ID: 4624
Logon Type: 10
Account Name: admin
Source Network Address: 185.234.72.19
Workstation Name: UNKNOWN

📊 Log Aggregation & SIEM

Individual log files on individual systems are useful for investigation, but to detect attacks across your environment, you need centralized log aggregation (collecting all logs in one place) and a SIEM (Security Information and Event Management) system. A SIEM collects, normalizes, and correlates logs from all your systems, then alerts you when it detects suspicious patterns. Think of it as the security operations nerve center: instead of checking logs on 500 individual machines, everything flows into one dashboard where you can search, correlate, and set up detection rules. Popular SIEMs include Splunk (industry standard, powerful but expensive), ELK Stack (Elasticsearch + Logstash + Kibana — open source), Microsoft Sentinel (Azure-native), and QRadar (IBM). The SIEM's killer feature is correlation — connecting a failed login on one server with a successful login on another, then a suspicious process creation on a third, to reveal an attack chain that no individual log would show.

Log Shipping Architecture

  Source Systems → Collectors → Aggregation → SIEM → Alerts

  Linux ────→ rsyslog/Filebeat ─┐
  Windows ──→ WEF/Winlogbeat ───┼──→ Logstash/Kafka ──→ ELK/Splunk
  Network ──→ syslog ───────────┘

# === RSYSLOG (Linux log forwarding) ===
# On the client (/etc/rsyslog.d/forward.conf):
*.* @@siem.company.com:514    # TCP forwarding (double @)
*.* @siem.company.com:514     # UDP forwarding (single @)

# Forward only auth logs:
auth,authpriv.* @@siem.company.com:514

# === FILEBEAT (Elastic agent for log shipping) ===
# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/auth.log
      - /var/log/syslog
    fields:
      log_type: linux_system

output.elasticsearch:
  hosts: ["https://elk.company.com:9200"]
  username: "filebeat_writer"
  password: "${BEAT_PASSWORD}"

# === WINDOWS EVENT FORWARDING (WEF) ===
# Centralize Windows logs without agents
# On collector: wecutil cs subscription.xml
# On sources: winrm quickconfig (GPO-configurable)

Basic Splunk Queries

# Search for failed logins across all Windows systems
index=wineventlog EventCode=4625
| stats count by src_ip, Account_Name
| sort -count

# Find brute force attempts (>10 failed logins from same IP)
index=wineventlog EventCode=4625
| stats count by src_ip
| where count > 10
| sort -count

# Detect new service installations
index=wineventlog EventCode=7045
| table _time, host, Service_Name, Service_File_Name, Service_Type

# Find cleared security logs
index=wineventlog EventCode=1102
| table _time, host, Account_Name

# Detect lateral movement (Logon Type 3 from unusual sources)
index=wineventlog EventCode=4624 Logon_Type=3
| stats count by src_ip, dest, Account_Name
| sort -count

# Timeline of events on a specific host
index=wineventlog host="DC01"
| sort _time
| table _time, EventCode, Message

Basic ELK Queries (KQL)

ELK (Elasticsearch, Logstash, Kibana) is an open-source SIEM alternative. Kibana is its web interface, and KQL (Kibana Query Language) is how you search logs in it:

# Kibana Query Language (KQL) examples

# Failed logins
event.code: "4625" and event.provider: "Microsoft-Windows-Security-Auditing"

# Successful RDP logins
event.code: "4624" and winlog.event_data.LogonType: "10"

# New processes with command line
event.code: "4688" and process.command_line: *powershell*

# Service installations
event.code: "7045"

# Log cleared
event.code: "1102"

# Sysmon: Network connections to external IPs
event.code: "3" and not destination.ip: (10.0.0.0/8 or 172.16.0.0/12 or 192.168.0.0/16)

🔧 Command-Line Log Parsing

You won't always have a SIEM available. Master these command-line tools and you can analyze logs anywhere:

grep — Pattern Matching

# Find failed SSH logins
grep "Failed password" /var/log/auth.log

# Count failed logins per IP
grep "Failed password" /var/log/auth.log | grep -oP '\d+\.\d+\.\d+\.\d+' | sort | uniq -c | sort -rn | head

# Find successful logins outside business hours (before 8am or after 6pm)
grep "Accepted" /var/log/auth.log | grep -E "(0[0-7]|1[89]|2[0-3]):[0-5][0-9]:[0-5][0-9]"

# Search multiple log files recursively
grep -r "error\|failed\|denied" /var/log/ --include="*.log" 2>/dev/null

# Inverse match — find lines that DON'T match (filter noise)
grep -v "CRON\|systemd" /var/log/auth.log

# Context — show lines before and after match
grep -B2 -A5 "Failed password for root" /var/log/auth.log

awk — Field Processing

# Extract specific fields from Apache access log
# Format: IP - - [timestamp] "METHOD /path HTTP/ver" status size
awk '{print $1, $9, $7}' /var/log/apache2/access.log | head
# Output: 10.0.0.1 200 /index.html

# Count requests per IP
awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -rn | head -20

# Find all 4xx and 5xx errors
awk '$9 >= 400 {print $0}' /var/log/apache2/access.log

# Sum bytes transferred per IP
awk '{bytes[$1] += $10} END {for (ip in bytes) print bytes[ip], ip}' \
    /var/log/apache2/access.log | sort -rn | head

# Extract timestamps from auth.log
awk '{print $1, $2, $3}' /var/log/auth.log | sort | uniq -c | sort -rn

jq — JSON Log Processing

# Parse JSON logs (common in cloud and containerized environments)
# View pretty-printed JSON
cat app.json | jq '.'

# Extract specific fields
cat app.json | jq '.timestamp, .level, .message'

# Filter by log level
cat app.json | jq 'select(.level == "ERROR")'

# Count events by type
cat app.json | jq -r '.event_type' | sort | uniq -c | sort -rn

# Filter and format
cat app.json | jq 'select(.src_ip != null and .status == 401) | {time: .timestamp, ip: .src_ip, path: .path}'

# Process journalctl JSON output
journalctl -u sshd -o json | jq 'select(.MESSAGE | test("Failed")) | {time: .__REALTIME_TIMESTAMP, msg: .MESSAGE}'

One-Liners for Common Investigations

# Top 20 IPs hitting your web server
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Find directory traversal attempts in web logs
grep -E "\.\./|\.\.\\\\|%2e%2e" /var/log/nginx/access.log

# Find SQL injection attempts
grep -iE "(union.*select|or.*1.*=.*1|drop.*table|insert.*into)" /var/log/nginx/access.log

# Find web shell access patterns
grep -E "cmd=|exec=|system\(|passthru|eval\(|shell_exec" /var/log/nginx/access.log

# Extract all unique User-Agent strings
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort -u

# Find large data transfers (potential exfiltration)
awk '$10 > 10000000 {print $1, $10, $7}' /var/log/nginx/access.log | sort -k2 -rn

💡 Pro Tip: Chain commands together for powerful analysis: zgrep "Failed" /var/log/auth.log* | grep -oP '\d+\.\d+\.\d+\.\d+' | sort | uniq -c | sort -rn | head -20. Note zgrep searches compressed (gzipped) log files too — important because logs rotate and compress daily.

🎯 Threat Hunting Methodology

Threat hunting is the proactive search for threats that have evaded existing detection. Unlike alert-driven analysis, hunting starts with a hypothesis and investigates whether the hypothesis is true.

The Hunting Loop

1. HYPOTHESIS — "An attacker may be using PowerShell for lateral movement." Based on: Threat intel, MITRE ATT&CK, industry trends.
2. DATA COLLECTION — Identify data sources: PowerShell logs, Sysmon, Security events. Ensure logging is adequate (enable Script Block Logging).
3. INVESTIGATION — Query logs for indicators:
   • Encoded PowerShell commands (-enc, -EncodedCommand)
   • PowerShell network connections (Invoke-WebRequest, Net.WebClient)
   • PowerShell spawned by unusual parents (Word, Excel)
   • PowerShell running from unusual directories
4. ANALYSIS — Evaluate findings: True positive → Incident response. Benign but suspicious → Watchlist, improve detection. Nothing found → Refine hypothesis, check data coverage.
5. DOCUMENTATION — Record hypothesis, data sources, queries, findings. Create detection rules for what you found. Share with the team.

Hunting Types

• IOC-based hunting: Search for known IOCs (Indicators of Compromise — specific IPs, domains, file hashes linked to known attacks) from threat intelligence feeds. Fast but only finds known threats.
• Behavioral hunting: Search for suspicious behaviors regardless of specific indicators — unusual process trees, abnormal login patterns, rare command-line arguments. Slower but catches unknown threats.
• Anomaly-based hunting: Establish baselines and look for deviations — a system that normally makes 100 DNS queries suddenly making 10,000, or a user who normally logs in from New York logging in from Moscow.

Example Hunt: Finding Encoded PowerShell

# Hypothesis: Attackers are using encoded PowerShell to evade detection
# Data source: Sysmon Event ID 1 (Process Creation), PowerShell Script Block Logging

# Splunk query:
index=sysmon EventCode=1 Image="*powershell.exe"
| where like(CommandLine, "%-enc%") OR like(CommandLine, "%-EncodedCommand%")
| table _time, host, User, ParentImage, CommandLine

# ELK/KQL:
process.name: "powershell.exe" and process.command_line: (*-enc* or *-EncodedCommand* or *FromBase64*)

# Command line (Windows):
Get-WinEvent -FilterHashtable @{LogName='Microsoft-Windows-Sysmon/Operational';Id=1} |
    Where-Object {$_.Properties[10].Value -match "enc|EncodedCommand|FromBase64"} |
    Select-Object TimeCreated, @{N='User';E={$_.Properties[12].Value}},
                  @{N='CommandLine';E={$_.Properties[10].Value}}

# Decode found Base64 commands:
echo "VABoAGkAcwAgAGkAcwAgAGEAIAB0AGUAcwB0AA==" | base64 -d
# Or in PowerShell:
[System.Text.Encoding]::Unicode.GetString([System.Convert]::FromBase64String("VABoAGkAcwA..."))

🔍 Common Attack Patterns in Logs

Brute Force Attack

# Pattern: Many failed logins (4625) from same IP, then success (4624)
# Linux: Multiple "Failed password" entries from same IP

# Detection query (Splunk):
index=wineventlog EventCode=4625
| stats count as failures by src_ip
| where failures > 20
| join src_ip [search index=wineventlog EventCode=4624
    | stats latest(_time) as success_time by src_ip]
| table src_ip, failures, success_time

# Linux one-liner:
grep "Failed password" /var/log/auth.log | \
    awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | \
    while read count ip; do
        if [ "$count" -gt 20 ]; then
            echo "ALERT: $ip had $count failed attempts"
            grep "Accepted.*$ip" /var/log/auth.log
        fi
    done

Lateral Movement

# Pattern: Logon Type 3 (network) from internal IPs to multiple systems
# Event 4648 (explicit credential usage) across multiple destinations

# Detection: Same account logging into many systems in short timeframe
index=wineventlog EventCode=4624 Logon_Type=3
| bin _time span=1h
| stats dc(dest) as unique_targets by _time, Account_Name, src_ip
| where unique_targets > 5
| sort -unique_targets

# PsExec detection: Service installation (7045) with PSEXESVC
index=wineventlog EventCode=7045 Service_Name="PSEXESVC"
| table _time, host, Service_File_Name

Data Exfiltration

# Pattern: Large outbound data transfers, DNS tunneling, unusual protocols

# Large HTTP uploads:
awk '$10 > 50000000 {print $1, $7, $10}' /var/log/nginx/access.log
# (POST requests with large body sizes)

# DNS tunneling (unusually long DNS queries):
# In Zeek/Bro dns.log:
awk -F'\t' 'length($10) > 50 {print $10}' dns.log | sort | uniq -c | sort -rn

# Sysmon network connections to unusual destinations:
index=sysmon EventCode=3 NOT (DestinationIp=10.* OR DestinationIp=172.16.* OR DestinationIp=192.168.*)
| stats sum(BytesSent) as total_bytes by Image, DestinationIp
| where total_bytes > 100000000
| sort -total_bytes

🗺️ MITRE ATT&CK Mapping

MITRE ATT&CK is a publicly available knowledge base of real-world attacker techniques, organized by tactics (what attackers want to achieve) and techniques (how they do it). Each technique has an ID (like T1078) and maps to specific log sources that can detect it. It's the common language security teams use to describe attacks and build detections:

🎯 Real-World Methodology

Here's a practical methodology for log-based investigation of a suspected compromise:

  ┌─────────────────────────────────────────┐
  │     INCIDENT INVESTIGATION FLOW         │
  └──────────────────┬──────────────────────┘
                     ▼
  Step 1: TIMELINE ──── When did it start/end?
                     │   Work backwards from the alert
                     ▼
  Step 2: SCOPE ─────── How far did they get?
                     │   Search compromised account + IP
                     │   across ALL log sources
                     ▼
  Step 3: INITIAL ───── How did they get in?
          ACCESS       Auth logs, web logs, email, VPN
                     │
                     ▼
  Step 4: ACTIONS ───── What did they do?
                     │   Process creation, file access,
                     │   network connections, persistence
                     ▼
  Step 5: IMPACT ────── What's the damage?
                     │   Data accessed/exfiltrated?
                     │   Accounts created/modified?
                     ▼
  Step 6: IOCs ──────── Extract indicators
                        IPs, domains, hashes → hunt elsewhere

⚠️ Common Mistakes

❌ Mistake #1: Not enabling audit policies.
Windows doesn't log most security events by default. You must enable Advanced Audit Policies, Process Creation auditing (with command-line logging), and PowerShell Script Block Logging. Without these, your logs are nearly useless for investigation.

❌ Mistake #2: Not centralizing logs.
If logs only exist on individual systems, an attacker who compromises a system can delete the logs. Centralize logs to a dedicated SIEM or log server that the attacker can't access.

❌ Mistake #3: Insufficient log retention.
The average dwell time is 194 days. If you only keep 30 days of logs, you can't investigate breaches that happened months ago. Retain security logs for at least 1 year.

❌ Mistake #4: Ignoring UTC timestamps.
When correlating logs from multiple systems, timezone confusion creates false timelines. Standardize all logging to UTC. Configure NTP to keep system clocks synchronized.

❌ Mistake #5: Not installing Sysmon.
Native Windows logging has significant gaps — no command-line arguments by default, no parent process tracking, no network connection logging. Sysmon fills these gaps and is free. There's no excuse not to deploy it.

❌ Mistake #6: Alert fatigue from too many rules.
Having 10,000 SIEM rules that generate thousands of alerts daily means nothing gets investigated. Start with high-fidelity rules for critical events (log clearing, new admin accounts, service installs) and expand gradually.

📚 Further Reading

• Incident Response Guide — What to do when you find something bad
• Network Defense Guide — Monitoring and IDS
• Malware Analysis Guide — Analyzing malware found through log analysis
• MITRE ATT&CK — Technique-to-detection mapping
• Windows Logging Cheat Sheets — What to enable and why
• Sigma Rules — Generic, vendor-neutral detection rules that can be converted to work with any SIEM (Splunk, ELK, etc.)
• Learning Path — Where log analysis fits in your journey