I’ll write this directly.
Most scraper teams add proxies and rotate user agents, then call it hardening — but without egress filtering and structured audit logging, you have no idea what your scrapers are actually calling out to, and no forensic trail when something goes sideways. That gap is where breaches happen and where compliance audits get painful.
Why Egress Filtering Belongs in Every Scraper Stack
Egress filtering means restricting outbound network traffic from your scraper processes to only the destinations they should be reaching. It sounds obvious, but most deployments skip it entirely. A compromised dependency, a malicious redirect chain, or a misconfigured scraper can exfiltrate data, phone home to a C2 server, or hammer unintended hosts without triggering a single alert.
The threat model is real. In 2025, the requests-html supply chain incident injected a DNS-beaconing payload that called out to a third-party domain from inside scraper containers. Without egress control, it ran silently for weeks across affected pipelines.
Pair egress filtering with Scraper Container Security: Isolation, Image Hardening (2026) to get defense in depth: the container isolation limits lateral movement, and egress filtering limits outbound blast radius.
Implementing Egress Filtering: Tools and Approaches
There are three practical layers to egress control in a scraper fleet:
- Network policy at the cluster level (Kubernetes NetworkPolicy or Cilium) — allowlist by CIDR or FQDN, deny everything else by default.
- Transparent proxy (Squid, mitmproxy in transparent mode) — forces all HTTP/S through a controlled proxy that enforces an allowlist and logs every request.
- eBPF-based syscall filtering (Falco, Tetragon) — catches connections at the kernel level, including non-HTTP traffic that a proxy would miss.
Here’s a minimal Cilium CiliumNetworkPolicy that restricts a scraper pod to only reach your proxy egress pool and a DNS resolver:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: scraper-egress
spec:
endpointSelector:
matchLabels:
app: scraper
egress:
- toEntities:
- "kube-dns"
- toCIDR:
- "10.10.5.0/24" # your proxy egress CIDR
toPorts:
- ports:
- port: "8080"
protocol: TCPEverything not explicitly allowed is dropped. For scrapers hitting residential proxies, the CIDR block is the proxy provider’s exit node range, not the target domain — the scraper never calls the target directly.
Comparing the three approaches:
| Tool | Layer | Pros | Cons |
|---|---|---|---|
| Kubernetes NetworkPolicy | L3/L4 | built-in, no agent | no FQDN matching, no logging |
| Cilium | L3-L7 | FQDN allowlists, metrics, DNS-aware | CNI replacement required |
| Squid (transparent) | L7 HTTP/S | deep logging, header inspection | TLS MITM complexity, latency |
| Falco + Tetragon | Kernel/eBPF | catches all syscalls, not just HTTP | alert-only unless paired with policy |
Cilium is the right default for Kubernetes-native fleets in 2026. Squid adds value when you need per-request HTTP logging with full headers. Use Tetragon on top for process-level telemetry.
Audit Logging: What to Capture and Where
Egress filtering without logging is a locked door with no camera. Audit logs let you reconstruct what a scraper did, prove compliance, and correlate anomalies after the fact. For scraper infrastructure, you need logs at three levels:
- DNS resolution logs — every domain a scraper pod resolved, with timestamp and pod identity
- Connection logs — source IP, destination IP:port, bytes sent/received, TLS SNI
- Application-level request logs — HTTP method, URL, status code, response size, latency
Securing Scraper Infrastructure: Rotating Credentials + Vault Patterns (2026) covers credential hygiene, but credential rotation is only useful if your logs can tell you which credential was used for which request. Make sure your scraper worker ID and the active credential alias appear in every log line.
For log shipping, the practical 2026 stack is Vector (collection and transformation) feeding Loki or OpenSearch. Vector runs as a DaemonSet, tails container stdout and Cilium flow logs, and enriches with pod labels before shipping. Retention: keep raw connection logs for 90 days, aggregated metrics for 1 year.
A structured log line from a scraper worker should look like this:
{
"ts": "2026-05-07T03:14:22Z",
"worker_id": "scraper-pool-7f4b",
"credential_alias": "pool-sg-03",
"proxy_exit": "103.24.77.14:8080",
"target_domain": "example.com",
"method": "GET",
"status": 200,
"bytes_rx": 42310,
"latency_ms": 312
}Structured JSON from the start means you can query by credential_alias or proxy_exit without regex gymnastics later.
Anomaly Detection and Alerting on Top of Logs
Raw logs are a forensic tool. Alerts are an operational tool. The two serve different purposes and need different configurations.
Useful alert rules for scraper egress:
- Scraper pod making a connection to a destination not in the allowlist (Cilium policy violation, fired immediately)
- Outbound data volume from a single worker exceeds 2x rolling 24h average (exfiltration signal or runaway loop)
- DNS resolution for a domain outside your target list (dependency or redirect chain gone rogue)
- More than 50 4xx responses per minute from a single proxy exit IP (IP burned, rotate and flag)
Falco rules work well for the process-level alerts. For volume-based anomalies, a simple Prometheus recording rule on Vector-derived metrics with a 2-sigma threshold catches most cases without drowning you in noise.
When you store scraped output, tie the pipeline back to Encrypting Scraped Data at Rest: KMS, Envelope Encryption (2026) — audit logs should record the KMS key alias used to encrypt each data batch, so a key compromise investigation can scope affected records precisely.
Operationalizing: Making This Stick
Egress filtering decays fast. New target domains get added, allowlists get lazily widened, and “temporary” exceptions become permanent. Prevent that with process:
- Pin egress allowlists in version control alongside scraper configs. Every change needs a PR.
- Run a weekly allowlist audit: compare Cilium policy entries against active scraper job configs. Flag anything in the policy with no corresponding active job.
- Set a hard limit on CIDR block width — no /8 or /16 blocks in egress policy. If a provider needs a wide range, use FQDN matching instead.
- Test egress policy in CI: spin up a policy-applied namespace, run the scraper, assert that connections to out-of-policy destinations are blocked (not just dropped silently).
Rotating credentials, hardening images, encrypting outputs — these are each solved problems in isolation. The discipline is doing them together, with the same rigor you’d apply to a production API.
Bottom line
Egress filtering and structured audit logging are not nice-to-haves for scraper infrastructure: they’re the difference between a repeatable, auditable data pipeline and a black box that you can’t investigate or defend. Start with Cilium NetworkPolicy in deny-all mode, add Vector-to-Loki for log shipping, and wire up at least the volume-anomaly and policy-violation alerts before your next production deploy. dataresearchtools.com covers this full stack — from network hardening to credential management to encrypted storage — so you can build scraper infrastructure that holds up under scrutiny.