Apache Suspicious Log Scanner — Documentation & Download

ruthner.atApache Suspicious Log Scanner← Back to ruthner.at

↑

⚡ Quick Start Guide

Get up and running in under 5 minutes

Download & Extract

wget https://www.ruthner.at/software/secscanner/apaches_secscanner.tgz
tar -xzf apaches_secscanner.tgz
cd apaches_secscanner

Run Initial Scan (Dry-Run)

python3 apache_suspicious_scan_live.py --days 7 /var/log/apache2/access.log*

Reviews last 7 days without making firewall changes

Create Whitelist (Optional but Recommended)

sudo nano whitelist.txt
# Add trusted IPs:
192.168.1.0/24
10.0.0.0/8

Enable Auto-Blocking

sudo python3 apache_suspicious_scan_live.py \
  --days 7 --block --block-backend apache --score-threshold 5 \
  --whitelist whitelist.txt --whitelist-mode ignore \
  /var/log/apache2/access.log*

Blocks IPs with score > 5, respects whitelist

📋 Requirements

Python 3.6+ (standard library only, no pip packages needed)
Root/sudo access to write the blocklist or firewall rules
Apache with mod_rewrite for the default blocking backend, plus mod_remoteip if Apache runs behind a proxy
nftables or iptables only for the optional kernel backend
Apache 2.x with Combined Log Format
Linux (tested on Debian, Ubuntu, Raspberry Pi OS)

🛡️ Overview & Key Features

The Apache Suspicious Log Scanner is a production-ready security tool that analyzes Apache HTTP access logs to identify and automatically block malicious activity. Built with Python's standard library, it requires no external dependencies and integrates seamlessly with modern Linux firewall systems.

🔍

Intelligent Detection

Multi-heuristic analysis detects SQL injection, path traversal, brute force, and scanning attempts

🚫

Auto-Blocking

Blocks in Apache via RewriteMap by default, or in the kernel via nftables/iptables

📊

Real-Time Dashboard

Live HTML status page with packet counters and colored threat indicators

⚡

Live Mode

Tail active logs and block threats as they happen with rate limiting

🎯

Smart Scoring

Per-IP risk scoring combines multiple behavioral indicators

🔒

Flexible Whitelist

CIDR, wildcards, ranges — protect trusted networks with ease

✅ Production Ready

Zero external dependencies • Handles rotated & gzipped logs • Idempotent firewall setup • Tested on Raspberry Pi to enterprise servers

⚙️ How It Works

The scanner processes Apache Combined Log Format entries and builds a behavioral profile for each IP address:

Log Parsing: Reads access.log files including rotated .gz archives
Pattern Matching: Identifies SQL injection, LFI/RFI, sensitive file access, binary protocols
Behavioral Analysis: Tracks request rates, 4xx ratios, unique paths, User-Agent diversity
Risk Scoring: Combines heuristics into a per-IP threat score (0-20+)
Automatic Response: Blocks high-risk IPs in Apache or via nftables/iptables when the score exceeds the threshold
Continuous Monitoring: Live mode tracks new threats and updates dashboard in real-time

🎓 Note

All detection is based on observable behavior in logs. The scanner does NOT inspect actual application code or database queries — it identifies attack attempts based on HTTP request patterns.

🔎 Detection Methods

The scanner analyzes multiple behavioral indicators to identify malicious activity:

Behavioral Heuristics

Indicator	Threshold	What It Means
PHP File Scanning	> 20 unique .php files	Vulnerability scanner probing for known exploits
4xx Error Ratio	> 50% (min 20 requests)	Path enumeration, forced browsing attacks
Request Rate	> 0.5 req/sec (min 10 req)	Aggressive scanning, DoS attempts
User-Agent Diversity	≥ 5 unique UAs per IP	Botnet activity, proxy rotation
Path Discovery	> 200 unique paths	Extensive reconnaissance, directory brute force
Auth Failures	> 50 (401/403 codes)	Credential stuffing, brute force login

🚩 Per-Request Suspicion Flags

Attack Type	Detection Patterns	Examples
SQL Injection	`UNION SELECT`, `OR 1=1`, `sleep()`	Database enumeration, blind SQLi
LFI/RFI/Traversal	`../`, `%2e%2e`, `php://`	File inclusion, path traversal attacks
Sensitive Files	`.env`, `.git`, `wp-login`	Configuration file access, admin panel probes
Binary/TLS	`\x16\x03` (TLS handshake)	Protocol confusion, HTTPS to HTTP port
Unprintable Chars	Non-ASCII bytes in request	Binary data, malformed requests
Overlong URI	> 1000 characters	Buffer overflow attempts, large payloads

📊 Scoring & Blocking Logic

Each IP receives a cumulative risk score based on observed behavior. Default blocking threshold is score > 5 (customizable via --score-threshold).

Score Calculation

Condition	Points	Severity
> 20 unique PHP files accessed	+5	High
4xx error ratio > 50% (≥20 requests)	+4	High
Request rate > 0.5/sec (≥10 requests)	+4	High
≥ 5 unique User-Agents	+2	Medium
> 200 unique paths accessed	+2	Medium
> 50 authentication failures (401/403)	+1	Medium
SQLi/LFI/Sensitive/Binary patterns	+1 to +5	Variable
Known scanner/attack User-Agent	+6	High
Honeypot path accessed	+8	High
Missing User-Agent	+2	Medium

Immediate Block Triggers

Some signals are conclusive enough to block on the first occurrence, without waiting for the score to accumulate: --block-on-first-sqli, --block-on-sensitive, --block-on-honeypot and --block-on-bad-ua (the last one is enabled by default).

⚠️ A missing User-Agent is deliberately not an immediate trigger

It scores +2 and nothing more. Health checks, simple scripts and some monitoring tools legitimately send no User-Agent — treating that as conclusive blocks real users on a single request, permanently, before they have done anything else.

This matters more than it looks. Blocking by IP punishes an address, not a person: behind a VPN exit, a mobile carrier NAT or a shared office line, one stranger's probe blocks everyone else too. Keep the whitelist current, prefer score thresholds over instant triggers for weak signals, and keep a way back in that does not depend on HTTP — the blocklist only affects the web server, so shell access always remains.

⚠️ Threshold Tuning

Start with threshold 5 and monitor for false positives. High-traffic sites may need 7-10. Use --days 1 initially to test scoring accuracy.

Example Score Scenarios

Score 3: Legitimate bot (GoogleBot) with diverse paths — safe
Score 6: Scanner probing 30 .php files + high 404 ratio — BLOCKED
Score 12: Aggressive attack with SQLi attempts + 2 req/sec — BLOCKED
Score 18: Botnet with 8 User-Agents + LFI probes + auth failures — BLOCKED

🔴 Live Mode & Real-Time Blocking

Monitor active logs and respond to threats instantly with live mode capabilities:

Live Mode Features

Continuous Monitoring: Tail active access.log (like tail -F)
Rate Limiting: Auto-block IPs exceeding req/sec threshold
Instant SQLi Blocking: --block-on-first-sqli for zero-tolerance
Sensitive File Protection: --block-on-sensitive blocks .env, .git access immediately
Status Dashboard: Auto-updating HTML page with live statistics

Live Mode Command

sudo python3 apache_suspicious_scan_live.py \
  --days 7 --live --block --score-threshold 4 \
  --whitelist whitelist.txt \
  --whitelist-mode ignore \
  --status-page /var/www/html/security_scanner/secscanstat.html \
  --status-interval 60 \
  --rate-limit 2 --rate-window 15 \
  /var/log/apache2/access.log

Rate Limiting Parameters

Parameter	Default	Description
`--rate-limit`	None	Max requests per second before auto-block
`--rate-window`	10s	Time window for rate calculation
`--block-on-first-sqli`	Off	Immediate block on first SQL injection attempt
`--block-on-sensitive`	Off	Immediate block on .env, .git access

💡 Production Tip

Run live mode in a screen/tmux session or as a systemd service. Combine with logrotate to handle Apache log rotation seamlessly.

📈 Status Dashboard

The scanner generates a modern, auto-refreshing HTML dashboard showing real-time security status.

🔴 View Live Demo

See a production scanner in action:

Open Live Status Dashboard →

Dashboard Sections

🚫 Blocked IPs

Currently blocked threats with scores, request counts, Hits (how often the block fired), the backend that enforced it, and colored reason tags

⚠️ Yellow Zone

Near-threshold IPs (score 3-5) that warrant monitoring

👀 Top Observed

High-activity IPs below blocking threshold — normal traffic or early reconnaissance

🚩 Recent Flags

Last 50 suspicious events with timestamps and short tags (SQLi, LFI, Sensitive)

Hits Counter

The dashboard displays Hits for each blocked IP — how often the block has taken effect since it was set. With the nft backend that is the firewall packet counter; with the apache backend it is the number of 403 responses the blocklist produced. This helps identify:

Persistent attackers (high hit counts)
One-time scanners (low hit counts)
Active threats (rapidly increasing hits)

🔄 Refresh Interval

Default: 60 seconds. Adjust with --status-interval. Hits are polled via --fw-poll-interval (if supported in your version).

✅ Whitelist Configuration

Protect trusted networks and services from accidental blocking with flexible whitelist formats.

Whitelist File Format

# whitelist.txt
# Lines starting with # are comments

# Full CIDR notation
192.168.1.0/24
10.0.0.0/8
172.16.0.0/12

# Shorthand notation
192.168.1
10

# Wildcards
203.0.113.*
198.51.100.1*

# IP ranges
203.0.113.10-203.0.113.20

# Single IPs
8.8.8.8
1.1.1.1

Whitelist Modes

Mode	Behavior	Use Case
`block-only`	IPs are scored and visible, but never auto-blocked	Default. Monitor internal networks for anomalies
`ignore`	IPs are fully excluded from scoring, display, and blocking	Monitoring systems, load balancers, trusted services

⚠️ Security Recommendation

Always whitelist your own management IPs, monitoring systems, and known partners. Test with block-only mode first to verify no legitimate traffic is flagged.

🔥 Blocking Backends

Where a block is enforced is a decision, not a detail — and it depends entirely on how traffic reaches your server. Select it with --block-backend apache|nft|both.

Behind a CDN, reverse proxy, or tunnel? Use the Apache backend.

When Apache sits behind Cloudflare, nginx, HAProxy, or a tunnel client, every packet arrives from the proxy — frequently from localhost. A packet filter then sees only the proxy's address, so blocking the attacker's IP in nftables silently does nothing: no packet from that address ever reaches your machine. The real client IP survives only in an HTTP header, and only Apache can act on it.

apache — RewriteMap blocklist (default)

The scanner maintains a text map that Apache consults on every request; blocked clients get 403 Forbidden.

# /etc/apache2/sec-blocklist.map
203.0.113.66 blocked
2a02:4780:59:6bca::1 blocked

Add this to every virtual host that should enforce the list:

RewriteMap secblock "txt:/etc/apache2/sec-blocklist.map"
RewriteEngine On
RewriteCond ${secblock:%{REMOTE_ADDR}|-} =blocked
RewriteRule ^ - [F,L]

No reload required. Apache re-reads a txt map when its mtime changes, so blocks take effect within seconds. The scanner replaces the file atomically.
IPv4 and IPv6 alike. The map is keyed by string — no separate sets or rules per address family.
Survives reboots by virtue of being a file.

🚨 Two easy mistakes

The RewriteMap must be declared inside the virtual host. In the global server config it is silently ineffective — the lookup always returns the default and nothing is ever blocked.

The rule belongs before any other RewriteRule in that vhost, or blocked clients can still reach proxy and WebSocket rules further down.

Prerequisite: the real client IP

Behind a proxy, the blocklist only works if mod_remoteip restores the real address into REMOTE_ADDR:

RemoteIPHeader X-Forwarded-For          # or CF-Connecting-IP behind Cloudflare
RemoteIPTrustedProxy 127.0.0.1          # the proxy's address as Apache sees it
RemoteIPTrustedProxy ::1

A locally running tunnel client connects over loopback, so loopback is what has to be trusted. Configuring only the CDN's public ranges is a common and silent failure: the header is discarded, every request is logged as ::1, and both detection and blocking quietly stop working.

Trusting a proxy means anything able to reach Apache from that address can forge the client IP header. Trust only addresses your proxy actually connects from.

nft — kernel packet filter

Right choice when clients connect directly: packets are dropped before they reach Apache, which is cheaper under heavy attack. Note the set is IPv4-only.

nftables Configuration

The scanner creates an idempotent nftables infrastructure:

# Table and set
Table: inet apache_scan
Set: bad_ips (type ipv4_addr, interval flag)

# Chains
input_drop (base chain, priority -5)
per_ip_drop (for individual IP counters)

# Rules
ip saddr @bad_ips drop        # Main blocking rule
ip saddr 1.2.3.4 counter drop # Per-IP stats

Manual nftables Setup (if needed)

# Create infrastructure (idempotent)
sudo nft add table inet apache_scan 2>/dev/null || true
sudo nft add set inet apache_scan bad_ips '{ type ipv4_addr; flags interval; }' 2>/dev/null || true
sudo nft add chain inet apache_scan per_ip_drop {} 2>/dev/null || true

# Recreate base chain
sudo nft delete chain inet apache_scan input_drop 2>/dev/null || true
sudo nft add chain inet apache_scan input_drop '{ type filter hook input priority -5; policy accept; }'
sudo nft add rule inet apache_scan input_drop jump per_ip_drop
sudo nft add rule inet apache_scan input_drop ip saddr @bad_ips drop

🚨 Important

The scanner needs root/sudo privileges to write the blocklist or modify firewall rules. Always test with --days 1 and dry-run before enabling --block in production.

🔧 Troubleshooting

Common Issues & Solutions

Issue: "nft: Could not process rule: No such file or directory"

Solution: The nftables infrastructure doesn't exist. Run the manual setup commands above, or let the scanner create it automatically (requires root).

Issue: Whitelist not working

Solution: Check file path and format. Ensure no trailing spaces. Use --whitelist-mode ignore for complete exclusion.

Issue: No logs processed

Solution: Verify log path with ls -lh /var/log/apache2/access.log*. Check date filter isn't too restrictive.

Issue: False positives blocking legitimate traffic

Solution: Increase --score-threshold to 7-10. Add legitimate bots to whitelist.

Reset Everything

# Remove all blocks
sudo python3 apache_suspicious_scan_live.py --unblock --block-backend apache

# Delete nftables infrastructure
sudo nft delete table inet apache_scan

📄 JSON Report Structure

Every scan generates a machine-readable JSON report for integration with SIEM, dashboards, or custom tools.

Structure Example

{
  "total_lines_attempted": 14523,
  "processed_entries": 14231,
  "unique_ips": 158,
  "top_suspicious": [
    {
      "ip": "203.0.113.57",
      "score": 14,
      "reqs": 412,
      "reasons": ["many_php_files=67", "4xx_ratio=0.86"]
    }
  ],
  "flagged_events": [
    {
      "ip": "203.0.113.57",
      "reason": "Possible LFI/RFI attempt",
      "detail": "/../../etc/passwd"
    }
  ]
}

📖 Command Examples

Basic Analysis

# Dry-run: analyze last 7 days, no firewall changes
python3 apache_suspicious_scan_live.py --days 7 /var/log/apache2/access.log*

# Include all rotated logs (automatic .gz support)
python3 apache_suspicious_scan_live.py --days 30 /var/log/apache2/access.log*

# Custom output file
python3 apache_suspicious_scan_live.py --days 7 --out /tmp/scan_report.json /var/log/apache2/access.log*

Production Deployment

# Standard blocking with whitelist (recommended)
sudo python3 apache_suspicious_scan_live.py \
  --days 7 --block --block-backend apache --score-threshold 5 \
  --whitelist whitelist.txt --whitelist-mode ignore \
  /var/log/apache2/access.log*

# Aggressive blocking (lower threshold)
sudo python3 apache_suspicious_scan_live.py \
  --days 3 --block --score-threshold 3 \
  --whitelist whitelist.txt \
  /var/log/apache2/access.log*

# Conservative blocking (higher threshold)
sudo python3 apache_suspicious_scan_live.py \
  --days 14 --block --score-threshold 8 \
  --whitelist whitelist.txt \
  /var/log/apache2/access.log*

Live Monitoring

# Basic live mode with status page
sudo python3 apache_suspicious_scan_live.py \
  --live --block --score-threshold 5 \
  --status-page /var/www/html/security_scanner/secscanstat.html \
  /var/log/apache2/access.log

# Advanced live mode with rate limiting
sudo python3 apache_suspicious_scan_live.py \
  --days 7 --live --block --score-threshold 4 \
  --whitelist whitelist.txt \
  --whitelist-mode ignore \
  --status-page /var/www/html/security_scanner/secscanstat.html \
  --status-interval 60 \
  --rate-limit 2 --rate-window 15 \
  --block-on-first-sqli \
  /var/log/apache2/access.log

# Run in screen/tmux (recommended for production)
screen -dmS apache-scanner sudo python3 apache_suspicious_scan_live.py \
  --live --block --score-threshold 5 \
  --whitelist whitelist.txt \
  --status-page /var/www/html/security_scanner/secscanstat.html \
  /var/log/apache2/access.log

Maintenance

# Remove all firewall blocks
sudo python3 apache_suspicious_scan_live.py --unblock --block-backend apache

# Analyze specific IP from logs
grep "1.2.3.4" /var/log/apache2/access.log* | python3 apache_suspicious_scan_live.py --days 999 -

# Check what would be blocked (dry-run)
python3 apache_suspicious_scan_live.py --days 1 --score-threshold 3 /var/log/apache2/access.log*

Cron Job (Hourly Scan)

# /etc/cron.d/apache-scanner
0 * * * * root /usr/bin/python3 /home/pi/sec_scanner/apache_suspicious_scan_live.py \
  --days 1 --block --score-threshold 5 \
  --whitelist whitelist.txt \
  /var/log/apache2/access.log* > /var/log/apache_scanner.log 2>&1

📁 File Locations

Path	Purpose	Permissions
`/var/log/apache2/access.log*`	Apache log files (incl. .gz rotations)	644 (readable)
`apache_suspicious_report.json`	Scan results and statistics	644
`whitelist.txt`	Trusted IP whitelist	644
`apache_suspicious_scan_*.py`	Scanner scripts	755 (executable)
`secscanstat.html`	Live status dashboard	644 (web-accessible)

❓ Frequently Asked Questions

Does this work with Nginx or other web servers?

Currently optimized for Apache Combined/Common log format. Nginx support possible with minor modifications to regex patterns.

Will it block search engine bots like GoogleBot?

Legitimate bots typically have low scores (< 3) unless they trigger specific attack patterns. Add known bot IPs to whitelist if needed.

How much CPU/memory does live mode use?

Minimal — typically < 50MB RAM and < 5% CPU on a Raspberry Pi 4. Scales well even with high-traffic sites.

Can I run multiple instances simultaneously?

Yes, but only one should have --block enabled to avoid firewall conflicts. Use different --out paths.

How do I unblock a specific IP?

# nftables
sudo nft delete element inet apache_scan bad_ips { 1.2.3.4 }

# iptables
sudo iptables -D INPUT -s 1.2.3.4 -m comment --comment apache-scan -j DROP

Does it handle log rotation automatically?

Yes — reads all matching files including .gz archives. Live mode gracefully handles Apache log rotation.

Can I integrate with Fail2Ban?

Yes — both can coexist. Fail2Ban handles authentication failures; this scanner handles broader attack patterns.

What about IPv6 support?

Currently IPv4 only. IPv6 support planned for future release.

📦 Download

Apache Suspicious Log Scanner v1.0

Scanner, vhost snippet for the Apache blocking backend, whitelist template and full documentation. Ready for production deployment.

📥 Download apaches_secscanner.tgz

Size: ~28KB • Python 3.6+ • No dependencies • Open Source

Package Contents

apache_suspicious_scan_live.py — The scanner
apache-blocklist.conf.example — vhost snippet for the Apache blocking backend
whitelist.txt.example — Whitelist template
start.sh — Example invocation for live mode
info.html — Full documentation

Installation

# Download and extract
wget https://www.ruthner.at/software/secscanner/apaches_secscanner.tgz
tar -xzf apaches_secscanner.tgz
cd apaches_secscanner

# Make executable
chmod +x apache_suspicious_scan_live.py

# Test installation
python3 apache_suspicious_scan_live.py --help

# Copy whitelist template
sudo cp apache_scan.whitelist.example whitelist.txt
sudo nano whitelist.txt  # Add your trusted IPs

Recommended Deployment

# Create scanner directory
sudo mkdir -p /home/pi/sec_scanner
sudo mv apache_suspicious_scan_*.py /home/pi/sec_scanner/

# Create web-accessible status directory
sudo mkdir -p /var/www/html/security_scanner
sudo chown www-data:www-data /var/www/html/security_scanner

# Initial scan (dry-run)
python3 /home/pi/sec_scanner/apache_suspicious_scan_live.py \
  --days 7 /var/log/apache2/access.log*

# Deploy live monitoring
sudo screen -dmS apache-scanner python3 \
  /home/pi/sec_scanner/apache_suspicious_scan_live.py \
  --live --block --score-threshold 5 \
  --whitelist whitelist.txt \
  --status-page /var/www/html/security_scanner/secscanstat.html \
  /var/log/apache2/access.log

⚖️ License & Disclaimer

✅ Free to Use

This software is provided free of charge for personal, educational, and commercial use.

🚨 Disclaimer

This software is provided AS-IS, without warranty or guarantees of any kind, express or implied. You are solely responsible for:

Correct configuration and testing before production use
Monitoring for false positives that may block legitimate traffic
Compliance with applicable laws and regulations
Any damages, data loss, or service disruption resulting from use

The author assumes no liability for misconfiguration, data loss, service interruption, or any other damages arising from the use of this software.

Use at your own risk. Always test thoroughly in a non-production environment before deploying to production systems.