Charles Proxy Guide: Debug HTTP Traffic Like a Pro

Charles Proxy is a cross-platform HTTP debugging proxy with a polished GUI that makes inspecting web traffic intuitive. While mitmproxy excels at scripting and automation, Charles shines for visual debugging — breakpoints, request editing, bandwidth throttling, and session recording are all point-and-click.

For web scrapers, Charles is the fastest way to reverse-engineer a website’s API, understand authentication flows, and pinpoint why your requests get blocked.

Installation & Setup

Install Charles

macOS:    brew install --cask charles
Windows:  Download from charlesproxy.com
Linux:    Download .tar.gz from charlesproxy.com

Charles runs as an HTTP proxy on port 8888 by default.

Configure Your System

macOS/Windows: Charles auto-configures as system proxy on launch. Toggle via Proxy → macOS/Windows Proxy.

Manual configuration:

Proxy Host: 127.0.0.1
Proxy Port: 8888

Python scripts:

import requests

proxies = {
    'http': 'http://127.0.0.1:8888',
    'https': 'http://127.0.0.1:8888',
}

# Use Charles CA cert for HTTPS
response = requests.get(
    'https://api.example.com/data',
    proxies=proxies,
    verify='/path/to/charles-ssl-proxying-certificate.pem'
)

SSL Proxying (HTTPS Inspection)

By default, Charles shows HTTPS as encrypted tunnels. To inspect HTTPS content:

Step 1: Install Charles Root Certificate

Go to Help → SSL Proxying → Install Charles Root Certificate

macOS: Installs to Keychain. Open Keychain Access, find “Charles Proxy CA”, double-click, expand Trust, set to “Always Trust.”
Windows: Installs to certificate store automatically.
iOS/Android: Navigate to chls.pro/ssl on the device while proxied through Charles.

Step 2: Enable SSL Proxying

Go to Proxy → SSL Proxying Settings

Add hosts you want to decrypt:

Host: *             Port: 443    (all HTTPS — use carefully)
Host: api.target.com  Port: 443  (specific domain — recommended)
Host: *.example.com   Port: 443  (wildcard subdomain)

Now HTTPS traffic shows decrypted request/response bodies.

Key Features for Web Scraping

1. Structure View

Charles organizes traffic by host in a tree view:

▼ api.example.com
  ▼ /v2
    ▼ /products
      GET /v2/products?page=1&limit=20
      GET /v2/products?page=2&limit=20
    ▼ /search
      POST /v2/search
  ▼ /auth
    POST /auth/token

This immediately reveals API structure and endpoint patterns.

2. Breakpoints

Set breakpoints to pause requests/responses for inspection or modification:

Menu: Proxy → Breakpoint Settings

Enable Breakpoints: ✓
URL Pattern: */api/v2/products*
Request: ✓  (pause outgoing requests)
Response: ✓ (pause incoming responses)

When a matching request fires, Charles pauses execution and lets you:

Edit request headers, body, URL
Edit response status, headers, body
Continue or abort the request

3. Rewrite Tool

Automatically modify requests and responses without breakpoints:

Menu: Tools → Rewrite

Example rules:

Rule: Add User-Agent Header
  Match URL: *
  Type: Add Header
  Where: Request
  Name: User-Agent
  Value: Mozilla/5.0 (Windows NT 10.0; Win64; x64)...

Rule: Remove Rate Limit Header
  Match URL: */api/*
  Type: Remove Header
  Where: Response
  Name: X-RateLimit-Remaining

Rule: Modify Response Body
  Match URL: */api/pricing*
  Type: Body
  Where: Response
  Match: "premium_only": true
  Replace: "premium_only": false

4. Map Remote

Redirect requests to different servers — useful for testing scrapers against local mocks:

Map From:
  Protocol: https
  Host: api.target.com
  Path: /v2/products

Map To:
  Protocol: http
  Host: localhost
  Port: 3000
  Path: /mock/products

5. Map Local

Serve local files instead of making network requests:

Map From:
  URL: https://api.target.com/v2/products*

Map To:
  Local Path: /Users/me/mocks/products.json

6. Throttling

Simulate slow connections to test timeout handling:

Menu: Proxy → Throttle Settings

Enable Throttling: ✓
Preset: 56K modem / 3G / Custom

Custom Settings:
  Bandwidth: 100 KB/s
  Utilization: 80%
  Latency: 200ms
  MTU: 1500

7. Repeat & Advanced Repeat

Right-click any request:

Repeat: Send the exact same request again
Advanced Repeat: Send N times with configurable concurrency
Compose: Copy request to editor for modification before sending

8. Compare Sessions

Record two sessions and compare differences:

Session 1: Requests from Chrome browser (successful)
Session 2: Requests from Python scraper (blocked)

Diff reveals:
- Missing headers (Accept-Language, Sec-Fetch-*)
- Different cookie values
- Different TLS fingerprint
- Missing AJAX preflight requests

Practical Workflow: Reverse-Engineering an API

Open Charles and enable SSL proxying for the target domain
Browse the target website normally in your browser
Watch the Structure view — identify API endpoints
Click each API call — examine:

URL pattern and query parameters
Required headers (Authorization, cookies, CSRF tokens)
Request body format (JSON, form data)
Response structure

Use Repeat to test the API call in isolation
Compose a modified request to understand required parameters
Replicate in your scraper with the exact headers and parameters

Charles vs mitmproxy Comparison

Feature	Charles	mitmproxy
Interface	GUI (polished)	TUI / Web / CLI
SSL Proxying	Point-and-click	CA cert install
Scripting	Limited (Rewrite rules)	Full Python
Automation	Low	High
Price	$50 (30-day trial)	Free, open source
Platform	macOS, Windows, Linux	macOS, Windows, Linux
Mobile debugging	Excellent	Good
WebSocket	Yes	Yes
Best for	Visual debugging, API discovery	Scripting, automation

Internal Links

mitmproxy Tutorial — open-source alternative with Python scripting
Fiddler Proxy Debugging — free Windows alternative
AJAX Request Interception — capture API calls programmatically
How Websites Detect Bots — use Charles to study detection
TLS Fingerprinting Deep Dive — understand TLS inspection implications

FAQ

Is Charles Proxy free?

Charles offers a 30-day free trial with full functionality. After that, a license costs $50 (one-time purchase). During the trial, sessions are limited to 30 minutes before requiring a restart.

Can Charles decrypt all HTTPS traffic?

Charles can decrypt HTTPS traffic from applications that trust its CA certificate. Some apps use certificate pinning, which prevents Charles from intercepting their traffic. Mobile apps often use pinning — tools like Frida can bypass pinning for debugging.

Does using Charles affect my scraper’s TLS fingerprint?

Yes. When Charles performs SSL proxying, the target server sees Charles’s TLS fingerprint instead of your client’s. This is fine for debugging but means your scraper’s behavior through Charles may differ from direct connections.

Can I use Charles and another proxy simultaneously?

Yes. Configure Charles to use an upstream proxy: Proxy → External Proxy Settings. Charles intercepts traffic first, then forwards it through your upstream proxy (e.g., residential proxy). This lets you debug traffic while using a real proxy.

How do I export captured requests to code?

Charles does not have built-in code export, but you can: (1) Copy requests as cURL from the right-click menu, then convert cURL to Python/JavaScript using online converters, or (2) Save the session and parse it programmatically.