mitmproxy & Charles — Docs

Why chain a debugging proxy

When your scraper misbehaves it can be hard to tell whether the problem is in your code, in the proxy gateway, or on the target. A local debugging proxy sits between the two and shows you the raw wire traffic:

Are the Proxy-Authorization headers being sent correctly?
Is the session token making it into the username?
Is the target returning a 302 redirect your code isn't following?
What does the full response body look like before your parser touches it?

The chain is: your app → local debug proxy → JustProxies → target. The debug proxy forwards everything upstream and lets you inspect it in real time.

mitmproxy — upstream mode

mitmproxy's upstream mode forwards every intercepted request to the specified upstream proxy — in this case JustProxies. Install via pip or the official installer, then start it:

shell

# Install
pip install mitmproxy

# Start in upstream mode, forwarding to JustProxies.
# mitmproxy itself listens on port 9090.
mitmproxy \
  --mode upstream:http://USER:[email protected]:8080 \
  --listen-port 9090

Now point your client at localhost:9090:

shell

# curl through mitmproxy → JustProxies → target
curl -x http://localhost:9090 \
     --proxy-cacert ~/.mitmproxy/mitmproxy-ca-cert.pem \
     https://api.ipify.org

# Python requests
import requests
proxies = {"http": "http://localhost:9090", "https": "http://localhost:9090"}
r = requests.get("https://api.ipify.org", proxies=proxies,
                 verify="~/.mitmproxy/mitmproxy-ca-cert.pem")
print(r.text)

The mitmproxy console shows every request and response in real time. Press f to filter by host, e to view an entry in detail, q to quit.

For non-interactive use, mitmdump writes a HAR-style log to stdout. mitmweb opens a browser-based UI. Same --mode upstream flag for all three.

Charles Proxy

Charles is a GUI debugging proxy for macOS and Windows. After installing, configure the external proxy:

Open Proxy → External Proxy Settings.
Check Use external proxy servers.
Under Web Proxy (HTTP) and Secure Web Proxy (HTTPS), enter gw.justproxies.online port 8080.
Check Proxy Authentication and enter your Username and Password.

Charles listens on port 8888 by default. Point your app at localhost:8888 and Charles forwards everything upstream through JustProxies.

Browser setup

To inspect browser traffic, configure the browser to use the local debug proxy:

Chrome / Chromium — launch flagshell

google-chrome \
  --proxy-server="http://localhost:9090" \
  --proxy-bypass-list="<-loopback>" \
  --ignore-certificate-errors

Firefox: Settings → Network Settings → Manual proxy configuration → HTTP Proxy: localhost, Port: 9090. Check Also use this proxy for HTTPS.

TLS and the certificate chain

For HTTPS traffic mitmproxy generates a dynamic certificate signed by its own CA. Your client must trust that CA or TLS verification will fail. mitmproxy generates the CA on first run:

shell

# CA certificate location (after first run)
~/.mitmproxy/mitmproxy-ca-cert.pem    # PEM format
~/.mitmproxy/mitmproxy-ca-cert.cer    # DER / Windows format

# Trust it system-wide on Ubuntu
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy.crt
sudo update-ca-certificates

mitmproxy intercepts TLS from your client to itself, then re-issues a fresh CONNECT to JustProxies for the upstream leg. JustProxies then opens a CONNECT tunnel to the target. The target sees a connection from a JustProxies exit IP — mitmproxy is invisible beyond your local machine.