kavklaw@llm ~ /guides/incident-response

kavklaw@llm $ cat incident-response-guide.md

Incident Response β€” From Alert to Recovery

🟑 Intermediate

⏱️ 30 min read · Master the NIST IR lifecycle and build effective response capabilities

← Back to Guides

Incident response is as much about access as it is about skill. Before an incident hits, make sure you have:

  • SIEM access β€” Splunk, ELK, Microsoft Sentinel, QRadar, or whatever your org uses for centralized log monitoring. If you don't have a SIEM, you'll be grepping raw log files on individual machines (covered in the Log Analysis Guide).
  • EDR console access β€” CrowdStrike Falcon, SentinelOne, Microsoft Defender for Endpoint, or Carbon Black. EDR shows process trees, network connections, and file activity on endpoints.
  • Log sources β€” Know where logs live before you need them: /var/log/auth.log (Debian/Ubuntu; RHEL uses /var/log/secure), Windows Event Logs (Security, System, Sysmon), firewall logs, VPN logs, DNS query logs, proxy/web access logs.
  • Forensic tools β€” KAPE or CyLR (artifact collection), Volatility 3 (memory analysis), dc3dd/dcfldd (disk imaging), LiME or AVML (Linux memory capture), WinPmem or DumpIt (Windows memory capture).
  • Admin credentials β€” Root/domain admin access to affected systems (or a pre-arranged IR account). You can't investigate a box you can't log into.
  • Out-of-band communication β€” A secure channel that doesn't depend on potentially compromised infrastructure: phone numbers, Signal group, a separate Slack/Teams workspace.

For practice, you can follow along with just a Linux VM and the Log Analysis Guide. Real IR work requires the organizational access listed above.

⚑ Quick Start

An alert just fired and you think you're compromised. These commands cover the critical first 15 minutes β€” verify, isolate, preserve evidence, and document. The order matters: you're fighting data loss from the moment you start.

# 1. Confirm the alert β€” don't panic, verify
# Check the alert source (SIEM, EDR, user report)
#   SIEM = Security Information and Event Management (centralized log monitoring)
#   EDR  = Endpoint Detection and Response (monitors individual computers)
# Correlate with other data: is this a true positive?

# 2. Assess severity and scope
# - What system(s) are affected?
# - What data could be at risk?
# - Is the attack still active?

# 3. Isolate the affected system (if confirmed)
# Network isolation β€” DON'T power off (you'll lose volatile evidence)
# On Linux:
sudo iptables -I INPUT -j DROP
sudo iptables -I OUTPUT -j DROP
# Or disconnect the network cable / disable NIC

# On Windows (PowerShell):
Get-NetAdapter | Disable-NetAdapter -Confirm:$false

# 4. Preserve volatile evidence FIRST
# Memory dump (Linux):
# Note: /dev/mem is restricted on modern kernels β€” use AVML instead:
avml /mnt/usb/memdump.lime

# Running processes, network connections, logged-in users:
ps auxf > /mnt/usb/processes.txt
netstat -tulpn > /mnt/usb/netstat.txt
ss -tulpn > /mnt/usb/ss.txt
who > /mnt/usb/logged_in.txt
last > /mnt/usb/last_logins.txt

# 5. Document everything from this point forward
# Timestamp, who did what, what was found

That's the emergency playbook. Now let's build the full picture of how incident response works from preparation through recovery.

🚨 What Is Incident Response?

Incident Response (IR) is how you handle security breaches, attacks, and anything that threatens an organization's systems or data. It's not just technical β€” it's people, process, and tools all working under pressure.

A security incident is any event that actually or potentially jeopardizes the confidentiality, integrity, or availability of information or systems. This includes:

  • Malware infections β€” ransomware, trojans, rootkits, worms
  • Unauthorized access β€” compromised accounts, privilege escalation
  • Data breaches β€” exfiltration of sensitive data
  • Denial of service β€” attacks that disrupt availability
  • Insider threats β€” malicious or negligent employees
  • Social engineering β€” phishing campaigns, business email compromise

Without a solid IR process, everything goes sideways β€” evidence gets destroyed, containment is late, attackers keep their foothold, and recovery drags on. The average data breach cost in 2024 was $4.88 million (IBM). Orgs with a tested IR plan saved $2.66 million per breach compared to those winging it.

πŸ”„ The NIST IR Lifecycle

NIST SP 800-61 Rev. 3 (supersedes Rev. 2 as of April 2025) is the go-to framework. Four major phases, which we'll expand to six for practical use:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    NIST IR Lifecycle                         β”‚
β”‚                                                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                  β”‚
β”‚  β”‚ Preparation │──▢│ Detection &         β”‚                  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚ Analysis            β”‚                  β”‚
β”‚        β–²           β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                  β”‚
β”‚        β”‚                    β–Ό                               β”‚
β”‚        β”‚           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                  β”‚
β”‚        β”‚           β”‚ Containment,        β”‚                  β”‚
β”‚        β”‚           β”‚ Eradication &       β”‚                  β”‚
β”‚        β”‚           β”‚ Recovery            β”‚                  β”‚
β”‚        β”‚           β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                  β”‚
β”‚        β”‚                    β–Ό                               β”‚
β”‚        β”‚           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                  β”‚
β”‚        └───────────│ Post-Incident       β”‚                  β”‚
β”‚                    β”‚ Activity            β”‚                  β”‚
β”‚                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The lifecycle is ITERATIVE β€” lessons learned feed back into preparation.

Let's break each phase down in detail with real-world examples and actionable steps.

πŸ“‹ Phase 1 β€” Preparation

Preparation is everything you do before an incident happens. This is the most important phase because it determines how effectively you can respond when the pressure is on. You cannot build a fire department after the building is already burning.

Building Your IR Team

An IR team (sometimes called CSIRT β€” Computer Security Incident Response Team) needs clearly defined roles:

  • IR Lead / Incident Commander: Makes decisions, coordinates response, communicates to management
  • Triage Analyst: Performs initial analysis, classifies severity, assigns priority
  • Forensic Analyst: Collects and analyzes evidence, creates timeline
  • Containment Specialist: Implements isolation, blocks attacker access
  • Recovery Specialist: Rebuilds systems, restores services
  • Communications Liaison: Handles internal/external communication
  • Legal Counsel: Advises on regulatory obligations, evidence handling

Creating an IR Playbook

A playbook is a step-by-step guide for responding to specific types of incidents. Every organization needs playbooks for at least these scenarios:

Playbook structure template:

  1. Detection Criteria β€” What alerts/indicators trigger this playbook? What data sources to check?
  2. Initial Triage (5 min) β€” Severity classification, scope assessment, escalation criteria
  3. Containment Actions (15 min) β€” Immediate steps, short-term (preserve evidence), long-term (prepare for eradication)
  4. Evidence Collection β€” What to collect, collection order (volatile β†’ non-volatile), chain of custody
  5. Eradication Steps β€” Remove the threat, indicators to verify removal
  6. Recovery Steps β€” Restore normal operations, verification checks
  7. Communication Template β€” Who to notify, when, template messages for each audience
πŸ’‘ Pro Tip: The best playbooks are tested through tabletop exercises before a real incident. Run quarterly tabletop drills where you walk through a scenario and identify gaps. A playbook that's never been tested is just a wish list.

Essential Preparation Checklist

  • βœ… IR team roles assigned with backup personnel for each role
  • βœ… Playbooks written for top 5 incident types
  • βœ… Communication plan (who contacts who, escalation paths)
  • βœ… Jump bag / IR toolkit ready (forensic tools, write blockers, blank drives)
  • βœ… Logging infrastructure operational (SIEM β€” centralized log monitoring and alerting)
  • βœ… EDR (Endpoint Detection and Response) deployed on critical systems
  • βœ… Network diagrams and asset inventory current
  • βœ… Backup strategy tested (actually restore from backups!)
  • βœ… Legal/compliance contacts identified (breach notification requirements)
  • βœ… Tabletop exercises conducted quarterly
🧠 Knowledge Check β€” IR Foundations
According to NIST SP 800-61, which phase of the IR lifecycle feeds directly back into the Preparation phase?
The NIST IR lifecycle is iterative. Post-Incident Activity (Lessons Learned) feeds directly back into Preparation, improving playbooks, tools, training, and processes for future incidents. Every incident is an opportunity to strengthen your defenses.
You receive an alert that a workstation is communicating with a known C2 (Command and Control) server β€” an attacker's remote server. What is your FIRST action?
Always verify first. False positives happen frequently, and powering off a system destroys volatile evidence (running processes, memory contents, network connections). Verify the alert by correlating with other data β€” check if the IP is actually a C2, look for additional IOCs on the host, and determine if this is a true positive before taking containment action.

πŸ” Phase 2 β€” Detection & Analysis

Detection is how you discover that an incident has occurred or is in progress. Analysis is how you determine the scope, severity, and nature of the incident. This phase is where triage happens β€” the critical first minutes that determine the trajectory of the entire response.

Detection Sources

Incidents can be detected through many channels:

  • SIEM alerts β€” A SIEM (Security Information and Event Management) collects logs from across your network and triggers alerts when it spots suspicious patterns (e.g., multiple failed logins followed by a success)
  • EDR alerts β€” EDR (Endpoint Detection and Response) monitors individual computers for suspicious behavior like process injection, suspicious file creation, or credential dumping
  • IDS/IPS alerts β€” IDS (Intrusion Detection System) and IPS (Intrusion Prevention System) watch network traffic for attack patterns
  • User reports β€” "My computer is acting weird" or "I clicked a link and now…"
  • Threat intelligence feeds β€” Matches against known IOCs (Indicators of Compromise β€” specific IP addresses, file hashes, or domains linked to known attacks)
  • External notification β€” Third parties reporting abuse from your network
  • Proactive threat hunting β€” Hypothesis-driven searching through data

Triage Methodology

When an alert comes in, follow this triage process:

Triage Decision Tree
═════════════════════
1. Is this a TRUE POSITIVE?
   β”œβ”€β”€ Check alert fidelity (is the detection reliable?)
   β”œβ”€β”€ Correlate with other data sources
   β”œβ”€β”€ Check for known false positive patterns
   └── If uncertain β†’ treat as true positive until proven otherwise

2. What is the SEVERITY?
   β”œβ”€β”€ Critical: Active data exfil, ransomware executing, domain admin compromise
   β”œβ”€β”€ High: Confirmed compromise, lateral movement detected
   β”œβ”€β”€ Medium: Suspicious activity, single host compromise (contained)
   └── Low: Policy violation, reconnaissance activity

3. What is the SCOPE?
   β”œβ”€β”€ Single host
   β”œβ”€β”€ Multiple hosts / network segment
   β”œβ”€β”€ Enterprise-wide
   └── External (partners, customers affected)

4. Is the attack STILL ACTIVE?
   β”œβ”€β”€ Yes β†’ Immediate containment needed
   └── No β†’ Evidence preservation, then analysis

Initial Analysis Commands

When triaging a potentially compromised system, gather this data quickly:

# === LINUX TRIAGE ===

# Running processes (look for suspicious names, paths, and parent processes)
ps auxf --sort=-%mem | head -50
ps aux | grep -E "(nc |ncat |/tmp/|/dev/shm/|base64|curl.*\||wget.*\|)"

# Network connections (look for unusual outbound connections)
ss -tulpn                       # Listening ports
ss -tn state established        # Established connections
netstat -antup                  # All connections with PIDs

# Recently modified files (attacker tools, webshells)
find / -mtime -1 -type f 2>/dev/null | grep -v proc | head -50
find /tmp /var/tmp /dev/shm -type f 2>/dev/null

# Scheduled tasks (persistence)
crontab -l
ls -la /etc/cron.*
cat /var/spool/cron/crontabs/*
systemctl list-timers

# User accounts (newly created accounts)
cat /etc/passwd | grep -v nologin | grep -v false
lastlog | grep -v "Never"
last -20

# SSH authorized keys (backdoor access)
find / -name "authorized_keys" 2>/dev/null
cat /root/.ssh/authorized_keys
for user in $(ls /home/); do cat /home/$user/.ssh/authorized_keys 2>/dev/null; done

# === WINDOWS TRIAGE (PowerShell) ===

# Running processes
Get-Process | Sort-Object CPU -Descending | Select-Object -First 30
Get-WmiObject Win32_Process | Select-Object Name,ProcessId,CommandLine

# Network connections
Get-NetTCPConnection | Where-Object {$_.State -eq "Established"}
netstat -ano | findstr ESTABLISHED

# Scheduled tasks
Get-ScheduledTask | Where-Object {$_.State -ne "Disabled"}
schtasks /query /fo LIST /v

# Recently created user accounts
Get-LocalUser | Where-Object {$_.Enabled -eq $true}
net user

# Services (look for newly installed services)
Get-Service | Where-Object {$_.Status -eq "Running"} | Sort-Object DisplayName
πŸ’‘ Pro Tip: Use the order of volatility when collecting evidence. The most volatile data disappears first: registers β†’ cache β†’ RAM β†’ running processes β†’ network connections β†’ disk. Always collect volatile data before touching the disk.

πŸ”’ Phase 3 β€” Containment

Containment is about stopping the bleeding. The goal is to prevent the attacker from causing further damage while preserving evidence for analysis. There are two types of containment:

Short-Term Containment

Immediate actions to stop the attack. Done within minutes of confirmation:

# Network Isolation
# ─────────────────
# Linux: Drop all network traffic
sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
sudo iptables -P FORWARD DROP
# Allow only IR team access
sudo iptables -I INPUT -s 10.0.0.50 -j ACCEPT  # IR workstation
sudo iptables -I OUTPUT -d 10.0.0.50 -j ACCEPT

# Windows: Disable network adapter
Get-NetAdapter | Disable-NetAdapter -Confirm:$false
# Or via firewall
New-NetFirewallRule -DisplayName "IR-BlockAll" -Direction Outbound -Action Block

# Account Containment
# ────────────────────
# Disable compromised accounts (don't delete β€” preserve evidence)
# Linux:
sudo usermod -L compromised_user
sudo passwd -l compromised_user

# Windows (Active Directory):
Disable-ADAccount -Identity compromised_user
# Reset password for all potentially affected accounts
Set-ADAccountPassword -Identity compromised_user -Reset

# Process Containment
# ────────────────────
# Kill malicious processes (after documenting them)
# Linux:
kill -STOP <PID>    # Suspend first (preserves memory)
kill -9 <PID>       # Then kill if needed

# Windows:
Stop-Process -Id <PID> -Force
# Or suspend with SysInternals PsSuspend:
pssuspend.exe <PID>

Long-Term Containment

Sustained measures while you prepare for eradication. This might mean setting up a parallel clean environment:

  • VLAN isolation β€” Move compromised systems to a quarantine VLAN with monitoring
  • Firewall rules β€” Block known C2 (Command and Control β€” the attacker's remote server) IPs/domains at the perimeter
  • DNS sinkholing β€” Redirect malicious domains to internal servers
  • Credential rotation β€” Reset passwords for all accounts in the affected scope
  • Patching β€” Apply patches for the exploited vulnerability on unaffected systems
  • Enhanced monitoring β€” Increase logging and alerting for the affected area
⚠️ Critical: Never power off a compromised system unless absolutely necessary (e.g., active ransomware encrypting). Powering off destroys volatile evidence β€” running processes, memory contents, network connections, and cryptographic keys that may be in RAM. Instead, isolate the system from the network while keeping it running.

🧹 Phase 4 β€” Eradication

Eradication is the process of completely removing the attacker's presence from your environment. This means eliminating all malware, backdoors, persistence mechanisms, and compromised accounts.

Eradication Checklist

Linux persistence locations to check:

/etc/cron.*                    # Cron jobs
/var/spool/cron/               # User crontabs
~/.bashrc, ~/.profile          # Shell startup files
/etc/systemd/system/           # Systemd services
/etc/init.d/                   # Init scripts
~/.ssh/authorized_keys         # SSH keys
/etc/ld.so.preload             # Shared library preloading
kernel modules (lsmod)         # Rootkit modules

Windows persistence locations to check:

HKCU\Software\Microsoft\Windows\CurrentVersion\Run
HKLM\Software\Microsoft\Windows\CurrentVersion\Run
Scheduled Tasks
Services (HKLM\SYSTEM\CurrentControlSet\Services)
WMI subscriptions
Startup folder (%APPDATA%\...\Start Menu\Programs\Startup)
DLL search order hijacking

Remove malicious artifacts: delete malware files (after collecting forensic copies), remove unauthorized user accounts, remove malicious scheduled tasks/cron jobs, remove backdoor SSH keys, remove rogue services, clean registry entries (Windows), remove malicious kernel modules (Linux).

Verify eradication: run full AV/EDR scans, check all persistence locations manually, compare system state to known-good baseline, monitor for re-infection indicators.

When to Reimage vs. Clean

Reimage when:

  • Rootkit detected (you can't trust the OS)
  • Firmware-level compromise suspected
  • You can't confidently identify all persistence mechanisms
  • The system is a standard workstation (faster to reimage than clean)
  • Compliance requirements mandate reimaging

Clean when:

  • The system is a critical server that can't be easily rebuilt
  • The compromise is well-understood and limited in scope
  • You have high confidence in your persistence enumeration
  • Downtime must be minimized
🧠 Knowledge Check β€” Containment & Eradication
Why should you avoid powering off a compromised system during containment?
Volatile evidence exists only in memory and is lost when power is removed. This includes running processes (what malware is active), network connections (C2 communication), encryption keys in RAM, and the contents of memory that forensic analysts need. Network isolation preserves this evidence while still containing the threat.
πŸ“‹ Scenario
A rootkit has been detected on a database server that hosts
your company's customer records. The server runs a custom
application that took 6 months to configure. You need to
decide: reimage or clean?
What is the recommended approach for this compromised database server?
When a rootkit is detected, you must reimage. Rootkits operate at the kernel level and can hide their own presence β€” you cannot trust any output from the compromised OS. Even if the server is complex to rebuild, the risk of residual compromise is too high. Back up data (carefully β€” verify it's not infected), rebuild the server from a known-good image, and restore the application from backup/documentation.

πŸ”„ Phase 5 β€” Recovery

Recovery brings affected systems back to normal operation. This must be done carefully to avoid reintroducing the threat or creating new vulnerabilities.

Recovery Process

  1. Rebuild from clean sources β€” reimage from gold/baseline images, reinstall from vendor media, restore from verified-clean backups
  2. Harden the rebuilt system β€” apply ALL current patches, implement the missing security control, change all credentials (including service accounts), tighten firewall rules, enable enhanced logging
  3. Test before returning to production β€” verify system functions correctly, run vulnerability scan, check monitoring/alerting works, verify backup jobs are running
  4. Monitored reintroduction β€” return to service with enhanced monitoring, watch for original IOCs, maintain heightened alerting for 30-90 days
  5. Verification β€” after 24h: check logs for re-infection; after 7 days: review monitoring data; after 30 days: reduce enhanced monitoring if clean

Backup Verification

Before restoring from backups, you must verify they're clean. Attackers with prolonged access may have compromised backups too:

  • Identify when the initial compromise occurred (dwell time)
  • Only restore from backups before the compromise date
  • Scan backup media for known malware IOCs
  • Restore to an isolated environment first and verify before production

πŸ“ Phase 6 β€” Lessons Learned

This is arguably the most valuable phase β€” and the one most often skipped. A post-incident review (PIR) should happen within 1-2 weeks of resolution while memories are fresh.

Post-Incident Review Meeting

Bring together everyone involved in the response and walk through these questions:

  1. What happened? (Timeline of events)
  2. How was it detected? (Could we detect it faster?)
  3. How did the attacker gain initial access?
  4. What was the attacker's objective?
  5. What systems and data were affected?
  6. How effective was our containment?
  7. What went well in our response?
  8. What went poorly? What would we do differently?
  9. Were our playbooks adequate?
  10. What additional tools/training/resources do we need?
  11. What preventive measures would have stopped this?
  12. What detection improvements would catch this faster?

Output: Written report + action items with owners and deadlines.

πŸ’‘ Pro Tip: Conduct blameless post-mortems. The goal is to improve systems and processes, not to punish individuals. If people fear blame, they'll hide mistakes, and you'll miss the most valuable learning opportunities. Focus on "what failed" not "who failed."

πŸ” Evidence Preservation & Chain of Custody

Proper evidence handling is critical, especially if the incident may lead to legal proceedings, regulatory reporting, or law enforcement involvement.

Order of Volatility

Collect evidence in order from most volatile (disappears fastest) to least volatile (sticks around). The idea is simple: grab the stuff that's about to vanish first, then work your way to the persistent data:

Order of Volatility (RFC 3227)
══════════════════════════════
1. CPU registers, cache            (nanoseconds)
2. RAM / memory contents           (lost on power off)
3. Running processes               (lost on power off)
4. Network connections             (lost on power off)
5. Network traffic (packets)       (transient)
6. OS state (uptime, loaded modules)(lost on reboot)
7. Disk (filesystem, swap)         (persistent)
8. Backups / remote logs           (persistent)
9. Physical evidence               (persistent)

Memory Acquisition:

# Linux (using LiME β€” Linux Memory Extractor):
sudo insmod lime-$(uname -r).ko "path=/mnt/usb/memory.lime format=lime"

# Windows (using WinPmem):
winpmem_mini_x64.exe memory.raw

# Windows (using DumpIt β€” easiest):
DumpIt.exe    # Creates memory dump in current directory

# === DISK IMAGING ===
# Create forensic image with dd (bit-for-bit copy):
sudo dc3dd if=/dev/sda of=/mnt/usb/disk.img hash=sha256 log=imaging.log

# Or use dcfldd for progress and verification:
sudo dcfldd if=/dev/sda of=/mnt/usb/disk.img hash=sha256 hashlog=hashes.txt

# Verify image integrity (these MUST match):
sha256sum /dev/sda
sha256sum /mnt/usb/disk.img

Chain of Custody

Every piece of evidence must have a documented chain of custody. This is essentially a logbook that tracks who had the evidence, when they had it, and what they did with it. Without this, evidence might not hold up in court because there's no proof it wasn't tampered with:

# Chain of Custody Record
# ────────────────────────
Evidence ID: IR-2024-001-MEM-01
Description: Memory dump from workstation WS-FINANCE-12
Source: Dell Optiplex 7080, Asset Tag: FIN-7080-012
Collected by: Jane Smith (IR Analyst)
Date/Time collected: 2024-03-15 14:32 UTC
Hash (SHA-256): a1b2c3d4e5f6...
Storage location: Evidence locker, Room 401, Shelf B

Transfer Log:
| Date/Time          | From          | To             | Purpose          |
|---------------------|---------------|----------------|-------------------|
| 2024-03-15 14:32   | System        | Jane Smith     | Collection       |
| 2024-03-15 15:00   | Jane Smith    | Evidence Locker| Secure storage   |
| 2024-03-16 09:00   | Evidence Lock | Bob Johnson    | Analysis         |
| 2024-03-16 17:00   | Bob Johnson   | Evidence Locker| Return           |

🧰 Essential IR Tools

Have these tools staged and ready before something goes wrong:

Evidence Collection

These tools solve a critical problem: during an incident, you need to quickly grab forensic evidence from live systems before it disappears or gets overwritten. Manual collection is too slow and error-prone β€” these tools automate the process.

# KAPE (Kroll Artifact Parser and Extractor) β€” Windows
# Rapidly collects forensic artifacts from live systems
# Targets: Registry hives, event logs, prefetch, browser history, etc.
kape.exe --tsource C: --tdest E:\Evidence --target !SANS_Triage

# Velociraptor β€” Cross-platform IR and forensics
# Agent-based collection at enterprise scale
# Deploy agent, then hunt across thousands of endpoints:
velociraptor gui    # Web interface for artifact collection

# CyLR β€” Cross-platform collection tool
# Collects forensic artifacts quickly
CyLR.exe --output-path E:\Evidence

# LiME β€” Linux Memory Extractor (kernel module)
sudo insmod lime.ko "path=/evidence/memory.lime format=lime"

# AVML β€” Acquire Volatile Memory for Linux (no kernel module needed)
sudo ./avml /evidence/memory.raw

Analysis Tools

Once you've collected evidence, these tools help you answer the key questions: what did the attacker do, how did they get in, and what did they leave behind?

# Sysinternals Suite (Windows) β€” Essential process/system analysis
# Process Explorer β€” advanced task manager, shows DLL loading, handles
procexp64.exe
# Process Monitor β€” real-time file/registry/network monitoring
procmon64.exe
# Autoruns β€” shows EVERYTHING that starts automatically
autoruns64.exe
# TCPView β€” real-time network connection viewer
tcpview64.exe

# Volatility 3 β€” Memory forensics framework
# Analyzes RAM dumps to find what was running at time of capture:
# malicious processes, injected code, network connections, encryption keys
vol -f memory.raw windows.pslist      # List processes
vol -f memory.raw windows.netscan     # Network connections
vol -f memory.raw windows.malfind     # Find injected code
vol -f memory.raw windows.cmdline     # Command line history

# Timeline Explorer / SRUM Dump β€” Eric Zimmerman tools
# Parse Windows artifacts: $MFT, registry hives, event logs
MFTECmd.exe -f "C:\$MFT" --csv output/
RECmd.exe --bn BatchExamples\RECmd_Batch_MC.reb --csv output/

# Chainsaw β€” Fast Windows Event Log analysis
# Scans .evtx log files against Sigma rules (generic, vendor-neutral
# detection rules that work across different SIEM platforms)
chainsaw hunt evtx_files/ -s sigma/ --mapping mappings/sigma-event-logs-all.yml
πŸ’‘ Pro Tip: Build a "jump bag" β€” a physical kit with a laptop loaded with IR tools, write blockers, external drives, network cables, and documentation. When an incident happens at 3 AM, you don't want to be downloading tools.

🎯 Common Attack Scenarios

Scenario 1: Ransomware

# Detection: Files being encrypted, ransom note appears, AV alerts
# Key indicators:
# - High CPU usage from encrypting process
# - Mass file modifications (.encrypted, .locked extensions)
# - Shadow copies deleted (vssadmin delete shadows /all)
# - Ransom note dropped in multiple directories

# Immediate Response:
1. Isolate affected systems (network disconnect)
2. DO NOT pay the ransom (initially)
3. Identify the ransomware variant (check ransom note, ID Ransomware)
4. Check for available decryptors (nomoreransom.org)
5. Determine scope β€” how many systems are affected?
6. Preserve evidence (memory dump before shutting down)
7. Check backups β€” are they intact and pre-infection?
8. Report to law enforcement (FBI IC3, local CERT)

Scenario 2: Business Email Compromise (BEC)

# Detection: Unusual email rules, impossible travel, forwarding rules
# Key indicators:
# - New inbox rules forwarding email to external addresses
# - Login from unusual geographic location
# - Email sent to finance requesting wire transfer
# - Reply-to address differs from sender

# Immediate Response:
1. Reset compromised account password immediately
2. Revoke all active sessions / OAuth tokens
3. Review and remove malicious inbox rules
4. Check for data exfiltration (email forwarding history)
5. Notify recipients of fraudulent emails
6. Enable MFA if not already active
7. Review sign-in logs for other compromised accounts
8. Check if any wire transfers were initiated

Scenario 3: Lateral Movement Detected

# Detection: EDR alert for credential dumping, unusual RDP/SMB connections
# Key indicators:
# - LSASS memory access (Mimikatz/comsvcs.dll)
# - PsExec / remote service creation on multiple hosts
# - Pass-the-hash / pass-the-ticket activity
# - New admin shares accessed (C$, ADMIN$)

# Immediate Response:
1. Identify the source host and compromised credentials
2. Map all systems the attacker has touched (scope assessment)
3. Isolate ALL potentially compromised systems
4. Reset ALL credentials the attacker could have harvested
5. Check for persistence on every touched system
6. Deploy enhanced monitoring on the affected network segment
7. Consider resetting the KRBTGT account (if domain compromise suspected)

πŸ“’ Communication During Incidents

Effective communication during an incident is as important as the technical response. Poor communication leads to confusion, delays, and reputational damage.

Communication Plan

Internal Communication Tiers:

  • Tier 1 (Immediate β€” within minutes): IR team, SOC (Security Operations Center) manager, CISO (Chief Information Security Officer). Channel: Secure chat (not email if email is compromised!)
  • Tier 2 (Within 1 hour): IT management, affected department heads. Channel: Phone call + secure messaging
  • Tier 3 (Within 4 hours): Executive leadership, legal, PR/communications. Channel: Briefing meeting
  • Tier 4 (As required by regulation): Regulators, law enforcement, affected individuals. Channel: Formal written notification

Communication Templates:

# Initial notification:
"At [TIME] on [DATE], we detected [BRIEF DESCRIPTION].
Current severity: [CRITICAL/HIGH/MEDIUM].
Affected systems: [SCOPE].
Current status: [INVESTIGATING/CONTAINING/ERADICATING].
Next update: [TIME].
Incident Commander: [NAME], reachable at [CONTACT]."

# Status update (every 2-4 hours during active response):
"Incident [ID] Update [#]:
Since last update: [ACTIONS TAKEN].
Current status: [STATUS].
Remaining actions: [NEXT STEPS].
ETA to resolution: [ESTIMATE].
Next update: [TIME]."
⚠️ Critical: If you suspect email is compromised, do not use email for incident communication. The attacker may be monitoring email. Use out-of-band communication β€” phone calls, secure messaging apps, or in-person meetings.
🧠 Knowledge Check β€” IR Process & Communication
What is the correct order for evidence collection based on the order of volatility?
Evidence must be collected from most volatile to least volatile (RFC 3227). RAM/memory contents are lost on power off and should be captured first. Running processes and network connections are also volatile. Disk images are persistent and can be collected later. This ensures you capture evidence that would otherwise be lost.
During a suspected email compromise incident, what communication channel should you use for incident coordination?
If email is compromised, the attacker can read your incident communications, learn your containment strategy, and stay ahead of your response. Use out-of-band channels: phone calls, secure messaging apps (Signal, Teams if not compromised), or in-person meetings. Never tip off the attacker about your response plan through a compromised channel.

🎯 Real-World Methodology

Here's a practical, step-by-step methodology for handling a real incident from detection to closure:

IR Timeline
═══════════════════════════════════════════════════════════════
T+0m     T+5m      T+15m      T+30m     T+1-4h    T+4-24h    T+24-72h   T+1-2wk
 β”‚        β”‚         β”‚          β”‚         β”‚          β”‚          β”‚          β”‚
 β–Ό        β–Ό         β–Ό          β–Ό         β–Ό          β–Ό          β–Ό          β–Ό
Alert β†’ Triage β†’ Analysis β†’ Contain β†’ Deep    β†’ Eradicate β†’ Recover β†’ Lessons
                                       Analysis                         Learned

PHASE 0: Alert Received (T+0) β€” Read the alert carefully. Note timestamp, source, affected system, alert type. Begin incident log.

PHASE 1: Triage (T+5 min) β€” Verify true positive. Classify severity. Determine scope. Decide: escalate or handle? Notify IR lead if High or Critical.

PHASE 2: Initial Analysis (T+15 min) β€” Gather volatile data. Check SIEM for correlated events. Check EDR. Identify initial IOCs. Determine attack vector.

PHASE 3: Containment (T+30 min) β€” Isolate affected systems. Block known-bad IPs/domains. Disable compromised accounts. Preserve evidence (memory dump, disk image).

PHASE 4: Deep Analysis (T+1-4 hours) β€” Analyze evidence. Build attacker timeline. Identify all compromised systems. Determine data at risk. Identify persistence mechanisms.

PHASE 5: Eradication (T+4-24 hours) β€” Remove all malware/backdoors. Reimage compromised systems. Rotate all credentials. Patch exploited vulnerabilities.

PHASE 6: Recovery (T+24-72 hours) β€” Rebuild from clean sources. Restore from verified backups. Return to production with enhanced monitoring. Verify normal operation.

PHASE 7: Lessons Learned (T+1-2 weeks) β€” Post-incident review meeting. Write incident report. Create action items with owners and deadlines. Update playbooks and detection rules.

⚠️ Common Mistakes

❌ Mistake #1: No incident response plan.
Building your IR process during an active incident is like building a fire truck during a fire. Have documented playbooks, trained personnel, and tested tools before something happens.
❌ Mistake #2: Powering off systems immediately.
The instinct is to "pull the plug" β€” but this destroys volatile evidence. Memory contents, running processes, and network connections are all lost. Isolate the network instead.
❌ Mistake #3: Skipping evidence preservation.
In the rush to remediate, teams often jump to eradication without collecting evidence first. This means you can't determine what was compromised, can't improve defenses, and can't support legal proceedings.
❌ Mistake #4: Alerting the attacker.
Communicating about the incident through compromised channels, making obvious changes to the compromised system, or immediately blocking the attacker's C2 (before understanding full scope) can cause the attacker to accelerate their objectives or destroy evidence.
❌ Mistake #5: Not determining full scope before eradication.
If you clean one compromised system but the attacker has persistence on three others, they'll simply re-compromise the cleaned system. Map the full attack path before beginning eradication.
❌ Mistake #6: Skipping lessons learned.
After a stressful incident, the temptation is to "move on." But the post-incident review is how you improve. Every incident you don't learn from is an incident wasted.
❌ Mistake #7: Not testing backups.
"We have backups" is meaningless if you've never tested restoring from them. Regularly test backup restoration and verify that backups are not also compromised (especially with ransomware that targets backup systems).

πŸ“š Further Reading

πŸ† Section Assessment β€” Incident Response Mastery
πŸ“‹ Scenario
Your SOC detects that ransomware is actively encrypting
files on three workstations in the finance department.
Shadow copies are being deleted. The encryption has been
running for approximately 10 minutes.
What should be your FIRST action?
With active ransomware encryption, network isolation is the immediate priority to prevent spread to other systems. Every second counts β€” the ransomware may try to spread via SMB or other network protocols. Isolate first, then assess scope. Don't reimage yet (you need evidence and to understand the full scope). Identification is important but secondary to stopping the spread.
Which tool would you use to rapidly collect forensic artifacts (registry hives, event logs, prefetch files) from a live Windows system?
KAPE is specifically designed for rapid forensic artifact collection from live Windows systems. It can collect registry hives, event logs, prefetch files, browser history, and dozens of other artifacts in minutes. Wireshark captures network traffic, Nmap scans ports, and Metasploit is an exploitation framework β€” none are designed for forensic artifact collection.