kavklaw@llm $ cat incident-response-guide.md
β±οΈ 30 min read Β· Master the NIST IR lifecycle and build effective response capabilities
Incident response is as much about access as it is about skill. Before an incident hits, make sure you have:
/var/log/auth.log (Debian/Ubuntu; RHEL uses /var/log/secure), Windows Event Logs (Security, System, Sysmon), firewall logs, VPN logs, DNS query logs, proxy/web access logs.For practice, you can follow along with just a Linux VM and the Log Analysis Guide. Real IR work requires the organizational access listed above.
An alert just fired and you think you're compromised. These commands cover the critical first 15 minutes β verify, isolate, preserve evidence, and document. The order matters: you're fighting data loss from the moment you start.
# 1. Confirm the alert β don't panic, verify
# Check the alert source (SIEM, EDR, user report)
# SIEM = Security Information and Event Management (centralized log monitoring)
# EDR = Endpoint Detection and Response (monitors individual computers)
# Correlate with other data: is this a true positive?
# 2. Assess severity and scope
# - What system(s) are affected?
# - What data could be at risk?
# - Is the attack still active?
# 3. Isolate the affected system (if confirmed)
# Network isolation β DON'T power off (you'll lose volatile evidence)
# On Linux:
sudo iptables -I INPUT -j DROP
sudo iptables -I OUTPUT -j DROP
# Or disconnect the network cable / disable NIC
# On Windows (PowerShell):
Get-NetAdapter | Disable-NetAdapter -Confirm:$false
# 4. Preserve volatile evidence FIRST
# Memory dump (Linux):
# Note: /dev/mem is restricted on modern kernels β use AVML instead:
avml /mnt/usb/memdump.lime
# Running processes, network connections, logged-in users:
ps auxf > /mnt/usb/processes.txt
netstat -tulpn > /mnt/usb/netstat.txt
ss -tulpn > /mnt/usb/ss.txt
who > /mnt/usb/logged_in.txt
last > /mnt/usb/last_logins.txt
# 5. Document everything from this point forward
# Timestamp, who did what, what was found
That's the emergency playbook. Now let's build the full picture of how incident response works from preparation through recovery.
Incident Response (IR) is how you handle security breaches, attacks, and anything that threatens an organization's systems or data. It's not just technical β it's people, process, and tools all working under pressure.
A security incident is any event that actually or potentially jeopardizes the confidentiality, integrity, or availability of information or systems. This includes:
Without a solid IR process, everything goes sideways β evidence gets destroyed, containment is late, attackers keep their foothold, and recovery drags on. The average data breach cost in 2024 was $4.88 million (IBM). Orgs with a tested IR plan saved $2.66 million per breach compared to those winging it.
NIST SP 800-61 Rev. 3 (supersedes Rev. 2 as of April 2025) is the go-to framework. Four major phases, which we'll expand to six for practical use:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β NIST IR Lifecycle β
β β
β ββββββββββββββ ββββββββββββββββββββββ β
β β Preparation ββββΆβ Detection & β β
β ββββββββββββββ β Analysis β β
β β² ββββββββββ¬ββββββββββββ β
β β βΌ β
β β ββββββββββββββββββββββ β
β β β Containment, β β
β β β Eradication & β β
β β β Recovery β β
β β ββββββββββ¬ββββββββββββ β
β β βΌ β
β β ββββββββββββββββββββββ β
β βββββββββββββ Post-Incident β β
β β Activity β β
β ββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The lifecycle is ITERATIVE β lessons learned feed back into preparation.
Let's break each phase down in detail with real-world examples and actionable steps.
Preparation is everything you do before an incident happens. This is the most important phase because it determines how effectively you can respond when the pressure is on. You cannot build a fire department after the building is already burning.
An IR team (sometimes called CSIRT β Computer Security Incident Response Team) needs clearly defined roles:
A playbook is a step-by-step guide for responding to specific types of incidents. Every organization needs playbooks for at least these scenarios:
Playbook structure template:
π‘ Pro Tip: The best playbooks are tested through tabletop exercises before a real incident. Run quarterly tabletop drills where you walk through a scenario and identify gaps. A playbook that's never been tested is just a wish list.
Detection is how you discover that an incident has occurred or is in progress. Analysis is how you determine the scope, severity, and nature of the incident. This phase is where triage happens β the critical first minutes that determine the trajectory of the entire response.
Incidents can be detected through many channels:
When an alert comes in, follow this triage process:
Triage Decision Tree
βββββββββββββββββββββ
1. Is this a TRUE POSITIVE?
βββ Check alert fidelity (is the detection reliable?)
βββ Correlate with other data sources
βββ Check for known false positive patterns
βββ If uncertain β treat as true positive until proven otherwise
2. What is the SEVERITY?
βββ Critical: Active data exfil, ransomware executing, domain admin compromise
βββ High: Confirmed compromise, lateral movement detected
βββ Medium: Suspicious activity, single host compromise (contained)
βββ Low: Policy violation, reconnaissance activity
3. What is the SCOPE?
βββ Single host
βββ Multiple hosts / network segment
βββ Enterprise-wide
βββ External (partners, customers affected)
4. Is the attack STILL ACTIVE?
βββ Yes β Immediate containment needed
βββ No β Evidence preservation, then analysis
When triaging a potentially compromised system, gather this data quickly:
# === LINUX TRIAGE ===
# Running processes (look for suspicious names, paths, and parent processes)
ps auxf --sort=-%mem | head -50
ps aux | grep -E "(nc |ncat |/tmp/|/dev/shm/|base64|curl.*\||wget.*\|)"
# Network connections (look for unusual outbound connections)
ss -tulpn # Listening ports
ss -tn state established # Established connections
netstat -antup # All connections with PIDs
# Recently modified files (attacker tools, webshells)
find / -mtime -1 -type f 2>/dev/null | grep -v proc | head -50
find /tmp /var/tmp /dev/shm -type f 2>/dev/null
# Scheduled tasks (persistence)
crontab -l
ls -la /etc/cron.*
cat /var/spool/cron/crontabs/*
systemctl list-timers
# User accounts (newly created accounts)
cat /etc/passwd | grep -v nologin | grep -v false
lastlog | grep -v "Never"
last -20
# SSH authorized keys (backdoor access)
find / -name "authorized_keys" 2>/dev/null
cat /root/.ssh/authorized_keys
for user in $(ls /home/); do cat /home/$user/.ssh/authorized_keys 2>/dev/null; done
# === WINDOWS TRIAGE (PowerShell) ===
# Running processes
Get-Process | Sort-Object CPU -Descending | Select-Object -First 30
Get-WmiObject Win32_Process | Select-Object Name,ProcessId,CommandLine
# Network connections
Get-NetTCPConnection | Where-Object {$_.State -eq "Established"}
netstat -ano | findstr ESTABLISHED
# Scheduled tasks
Get-ScheduledTask | Where-Object {$_.State -ne "Disabled"}
schtasks /query /fo LIST /v
# Recently created user accounts
Get-LocalUser | Where-Object {$_.Enabled -eq $true}
net user
# Services (look for newly installed services)
Get-Service | Where-Object {$_.Status -eq "Running"} | Sort-Object DisplayName
π‘ Pro Tip: Use the order of volatility when collecting evidence. The most volatile data disappears first: registers β cache β RAM β running processes β network connections β disk. Always collect volatile data before touching the disk.
Containment is about stopping the bleeding. The goal is to prevent the attacker from causing further damage while preserving evidence for analysis. There are two types of containment:
Immediate actions to stop the attack. Done within minutes of confirmation:
# Network Isolation
# βββββββββββββββββ
# Linux: Drop all network traffic
sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
sudo iptables -P FORWARD DROP
# Allow only IR team access
sudo iptables -I INPUT -s 10.0.0.50 -j ACCEPT # IR workstation
sudo iptables -I OUTPUT -d 10.0.0.50 -j ACCEPT
# Windows: Disable network adapter
Get-NetAdapter | Disable-NetAdapter -Confirm:$false
# Or via firewall
New-NetFirewallRule -DisplayName "IR-BlockAll" -Direction Outbound -Action Block
# Account Containment
# ββββββββββββββββββββ
# Disable compromised accounts (don't delete β preserve evidence)
# Linux:
sudo usermod -L compromised_user
sudo passwd -l compromised_user
# Windows (Active Directory):
Disable-ADAccount -Identity compromised_user
# Reset password for all potentially affected accounts
Set-ADAccountPassword -Identity compromised_user -Reset
# Process Containment
# ββββββββββββββββββββ
# Kill malicious processes (after documenting them)
# Linux:
kill -STOP <PID> # Suspend first (preserves memory)
kill -9 <PID> # Then kill if needed
# Windows:
Stop-Process -Id <PID> -Force
# Or suspend with SysInternals PsSuspend:
pssuspend.exe <PID>
Sustained measures while you prepare for eradication. This might mean setting up a parallel clean environment:
β οΈ Critical: Never power off a compromised system unless absolutely necessary (e.g., active ransomware encrypting). Powering off destroys volatile evidence β running processes, memory contents, network connections, and cryptographic keys that may be in RAM. Instead, isolate the system from the network while keeping it running.
Eradication is the process of completely removing the attacker's presence from your environment. This means eliminating all malware, backdoors, persistence mechanisms, and compromised accounts.
Linux persistence locations to check:
/etc/cron.* # Cron jobs
/var/spool/cron/ # User crontabs
~/.bashrc, ~/.profile # Shell startup files
/etc/systemd/system/ # Systemd services
/etc/init.d/ # Init scripts
~/.ssh/authorized_keys # SSH keys
/etc/ld.so.preload # Shared library preloading
kernel modules (lsmod) # Rootkit modules
Windows persistence locations to check:
HKCU\Software\Microsoft\Windows\CurrentVersion\Run
HKLM\Software\Microsoft\Windows\CurrentVersion\Run
Scheduled Tasks
Services (HKLM\SYSTEM\CurrentControlSet\Services)
WMI subscriptions
Startup folder (%APPDATA%\...\Start Menu\Programs\Startup)
DLL search order hijacking
Remove malicious artifacts: delete malware files (after collecting forensic copies), remove unauthorized user accounts, remove malicious scheduled tasks/cron jobs, remove backdoor SSH keys, remove rogue services, clean registry entries (Windows), remove malicious kernel modules (Linux).
Verify eradication: run full AV/EDR scans, check all persistence locations manually, compare system state to known-good baseline, monitor for re-infection indicators.
Reimage when:
Clean when:
A rootkit has been detected on a database server that hosts
your company's customer records. The server runs a custom
application that took 6 months to configure. You need to
decide: reimage or clean?
Recovery brings affected systems back to normal operation. This must be done carefully to avoid reintroducing the threat or creating new vulnerabilities.
Before restoring from backups, you must verify they're clean. Attackers with prolonged access may have compromised backups too:
This is arguably the most valuable phase β and the one most often skipped. A post-incident review (PIR) should happen within 1-2 weeks of resolution while memories are fresh.
Bring together everyone involved in the response and walk through these questions:
Output: Written report + action items with owners and deadlines.
π‘ Pro Tip: Conduct blameless post-mortems. The goal is to improve systems and processes, not to punish individuals. If people fear blame, they'll hide mistakes, and you'll miss the most valuable learning opportunities. Focus on "what failed" not "who failed."
Proper evidence handling is critical, especially if the incident may lead to legal proceedings, regulatory reporting, or law enforcement involvement.
Collect evidence in order from most volatile (disappears fastest) to least volatile (sticks around). The idea is simple: grab the stuff that's about to vanish first, then work your way to the persistent data:
Order of Volatility (RFC 3227)
ββββββββββββββββββββββββββββββ
1. CPU registers, cache (nanoseconds)
2. RAM / memory contents (lost on power off)
3. Running processes (lost on power off)
4. Network connections (lost on power off)
5. Network traffic (packets) (transient)
6. OS state (uptime, loaded modules)(lost on reboot)
7. Disk (filesystem, swap) (persistent)
8. Backups / remote logs (persistent)
9. Physical evidence (persistent)
Memory Acquisition:
# Linux (using LiME β Linux Memory Extractor):
sudo insmod lime-$(uname -r).ko "path=/mnt/usb/memory.lime format=lime"
# Windows (using WinPmem):
winpmem_mini_x64.exe memory.raw
# Windows (using DumpIt β easiest):
DumpIt.exe # Creates memory dump in current directory
# === DISK IMAGING ===
# Create forensic image with dd (bit-for-bit copy):
sudo dc3dd if=/dev/sda of=/mnt/usb/disk.img hash=sha256 log=imaging.log
# Or use dcfldd for progress and verification:
sudo dcfldd if=/dev/sda of=/mnt/usb/disk.img hash=sha256 hashlog=hashes.txt
# Verify image integrity (these MUST match):
sha256sum /dev/sda
sha256sum /mnt/usb/disk.img
Every piece of evidence must have a documented chain of custody. This is essentially a logbook that tracks who had the evidence, when they had it, and what they did with it. Without this, evidence might not hold up in court because there's no proof it wasn't tampered with:
# Chain of Custody Record
# ββββββββββββββββββββββββ
Evidence ID: IR-2024-001-MEM-01
Description: Memory dump from workstation WS-FINANCE-12
Source: Dell Optiplex 7080, Asset Tag: FIN-7080-012
Collected by: Jane Smith (IR Analyst)
Date/Time collected: 2024-03-15 14:32 UTC
Hash (SHA-256): a1b2c3d4e5f6...
Storage location: Evidence locker, Room 401, Shelf B
Transfer Log:
| Date/Time | From | To | Purpose |
|---------------------|---------------|----------------|-------------------|
| 2024-03-15 14:32 | System | Jane Smith | Collection |
| 2024-03-15 15:00 | Jane Smith | Evidence Locker| Secure storage |
| 2024-03-16 09:00 | Evidence Lock | Bob Johnson | Analysis |
| 2024-03-16 17:00 | Bob Johnson | Evidence Locker| Return |
Have these tools staged and ready before something goes wrong:
These tools solve a critical problem: during an incident, you need to quickly grab forensic evidence from live systems before it disappears or gets overwritten. Manual collection is too slow and error-prone β these tools automate the process.
# KAPE (Kroll Artifact Parser and Extractor) β Windows
# Rapidly collects forensic artifacts from live systems
# Targets: Registry hives, event logs, prefetch, browser history, etc.
kape.exe --tsource C: --tdest E:\Evidence --target !SANS_Triage
# Velociraptor β Cross-platform IR and forensics
# Agent-based collection at enterprise scale
# Deploy agent, then hunt across thousands of endpoints:
velociraptor gui # Web interface for artifact collection
# CyLR β Cross-platform collection tool
# Collects forensic artifacts quickly
CyLR.exe --output-path E:\Evidence
# LiME β Linux Memory Extractor (kernel module)
sudo insmod lime.ko "path=/evidence/memory.lime format=lime"
# AVML β Acquire Volatile Memory for Linux (no kernel module needed)
sudo ./avml /evidence/memory.raw
Once you've collected evidence, these tools help you answer the key questions: what did the attacker do, how did they get in, and what did they leave behind?
# Sysinternals Suite (Windows) β Essential process/system analysis
# Process Explorer β advanced task manager, shows DLL loading, handles
procexp64.exe
# Process Monitor β real-time file/registry/network monitoring
procmon64.exe
# Autoruns β shows EVERYTHING that starts automatically
autoruns64.exe
# TCPView β real-time network connection viewer
tcpview64.exe
# Volatility 3 β Memory forensics framework
# Analyzes RAM dumps to find what was running at time of capture:
# malicious processes, injected code, network connections, encryption keys
vol -f memory.raw windows.pslist # List processes
vol -f memory.raw windows.netscan # Network connections
vol -f memory.raw windows.malfind # Find injected code
vol -f memory.raw windows.cmdline # Command line history
# Timeline Explorer / SRUM Dump β Eric Zimmerman tools
# Parse Windows artifacts: $MFT, registry hives, event logs
MFTECmd.exe -f "C:\$MFT" --csv output/
RECmd.exe --bn BatchExamples\RECmd_Batch_MC.reb --csv output/
# Chainsaw β Fast Windows Event Log analysis
# Scans .evtx log files against Sigma rules (generic, vendor-neutral
# detection rules that work across different SIEM platforms)
chainsaw hunt evtx_files/ -s sigma/ --mapping mappings/sigma-event-logs-all.yml
π‘ Pro Tip: Build a "jump bag" β a physical kit with a laptop loaded with IR tools, write blockers, external drives, network cables, and documentation. When an incident happens at 3 AM, you don't want to be downloading tools.
# Detection: Files being encrypted, ransom note appears, AV alerts
# Key indicators:
# - High CPU usage from encrypting process
# - Mass file modifications (.encrypted, .locked extensions)
# - Shadow copies deleted (vssadmin delete shadows /all)
# - Ransom note dropped in multiple directories
# Immediate Response:
1. Isolate affected systems (network disconnect)
2. DO NOT pay the ransom (initially)
3. Identify the ransomware variant (check ransom note, ID Ransomware)
4. Check for available decryptors (nomoreransom.org)
5. Determine scope β how many systems are affected?
6. Preserve evidence (memory dump before shutting down)
7. Check backups β are they intact and pre-infection?
8. Report to law enforcement (FBI IC3, local CERT)
# Detection: Unusual email rules, impossible travel, forwarding rules
# Key indicators:
# - New inbox rules forwarding email to external addresses
# - Login from unusual geographic location
# - Email sent to finance requesting wire transfer
# - Reply-to address differs from sender
# Immediate Response:
1. Reset compromised account password immediately
2. Revoke all active sessions / OAuth tokens
3. Review and remove malicious inbox rules
4. Check for data exfiltration (email forwarding history)
5. Notify recipients of fraudulent emails
6. Enable MFA if not already active
7. Review sign-in logs for other compromised accounts
8. Check if any wire transfers were initiated
# Detection: EDR alert for credential dumping, unusual RDP/SMB connections
# Key indicators:
# - LSASS memory access (Mimikatz/comsvcs.dll)
# - PsExec / remote service creation on multiple hosts
# - Pass-the-hash / pass-the-ticket activity
# - New admin shares accessed (C$, ADMIN$)
# Immediate Response:
1. Identify the source host and compromised credentials
2. Map all systems the attacker has touched (scope assessment)
3. Isolate ALL potentially compromised systems
4. Reset ALL credentials the attacker could have harvested
5. Check for persistence on every touched system
6. Deploy enhanced monitoring on the affected network segment
7. Consider resetting the KRBTGT account (if domain compromise suspected)
Effective communication during an incident is as important as the technical response. Poor communication leads to confusion, delays, and reputational damage.
Internal Communication Tiers:
Communication Templates:
# Initial notification:
"At [TIME] on [DATE], we detected [BRIEF DESCRIPTION].
Current severity: [CRITICAL/HIGH/MEDIUM].
Affected systems: [SCOPE].
Current status: [INVESTIGATING/CONTAINING/ERADICATING].
Next update: [TIME].
Incident Commander: [NAME], reachable at [CONTACT]."
# Status update (every 2-4 hours during active response):
"Incident [ID] Update [#]:
Since last update: [ACTIONS TAKEN].
Current status: [STATUS].
Remaining actions: [NEXT STEPS].
ETA to resolution: [ESTIMATE].
Next update: [TIME]."
β οΈ Critical: If you suspect email is compromised, do not use email for incident communication. The attacker may be monitoring email. Use out-of-band communication β phone calls, secure messaging apps, or in-person meetings.
Here's a practical, step-by-step methodology for handling a real incident from detection to closure:
IR Timeline
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
T+0m T+5m T+15m T+30m T+1-4h T+4-24h T+24-72h T+1-2wk
β β β β β β β β
βΌ βΌ βΌ βΌ βΌ βΌ βΌ βΌ
Alert β Triage β Analysis β Contain β Deep β Eradicate β Recover β Lessons
Analysis Learned
PHASE 0: Alert Received (T+0) β Read the alert carefully. Note timestamp, source, affected system, alert type. Begin incident log.
PHASE 1: Triage (T+5 min) β Verify true positive. Classify severity. Determine scope. Decide: escalate or handle? Notify IR lead if High or Critical.
PHASE 2: Initial Analysis (T+15 min) β Gather volatile data. Check SIEM for correlated events. Check EDR. Identify initial IOCs. Determine attack vector.
PHASE 3: Containment (T+30 min) β Isolate affected systems. Block known-bad IPs/domains. Disable compromised accounts. Preserve evidence (memory dump, disk image).
PHASE 4: Deep Analysis (T+1-4 hours) β Analyze evidence. Build attacker timeline. Identify all compromised systems. Determine data at risk. Identify persistence mechanisms.
PHASE 5: Eradication (T+4-24 hours) β Remove all malware/backdoors. Reimage compromised systems. Rotate all credentials. Patch exploited vulnerabilities.
PHASE 6: Recovery (T+24-72 hours) β Rebuild from clean sources. Restore from verified backups. Return to production with enhanced monitoring. Verify normal operation.
PHASE 7: Lessons Learned (T+1-2 weeks) β Post-incident review meeting. Write incident report. Create action items with owners and deadlines. Update playbooks and detection rules.
β Mistake #1: No incident response plan.
Building your IR process during an active incident is like building a fire truck during a fire. Have documented playbooks, trained personnel, and tested tools before something happens.
β Mistake #2: Powering off systems immediately.
The instinct is to "pull the plug" β but this destroys volatile evidence. Memory contents, running processes, and network connections are all lost. Isolate the network instead.
β Mistake #3: Skipping evidence preservation.
In the rush to remediate, teams often jump to eradication without collecting evidence first. This means you can't determine what was compromised, can't improve defenses, and can't support legal proceedings.
β Mistake #4: Alerting the attacker.
Communicating about the incident through compromised channels, making obvious changes to the compromised system, or immediately blocking the attacker's C2 (before understanding full scope) can cause the attacker to accelerate their objectives or destroy evidence.
β Mistake #5: Not determining full scope before eradication.
If you clean one compromised system but the attacker has persistence on three others, they'll simply re-compromise the cleaned system. Map the full attack path before beginning eradication.
β Mistake #6: Skipping lessons learned.
After a stressful incident, the temptation is to "move on." But the post-incident review is how you improve. Every incident you don't learn from is an incident wasted.
β Mistake #7: Not testing backups.
"We have backups" is meaningless if you've never tested restoring from them. Regularly test backup restoration and verify that backups are not also compromised (especially with ransomware that targets backup systems).
Your SOC detects that ransomware is actively encrypting
files on three workstations in the finance department.
Shadow copies are being deleted. The encryption has been
running for approximately 10 minutes.