DataResearchTools

Category: DIY Proxy Infrastructure

  • How to Build a 4G/5G Mobile Proxy Farm with Raspberry Pi

    How to Build a 4G/5G Mobile Proxy Farm with Raspberry Pi

    Building your own mobile proxy farm is one of the most rewarding (and frustrating) technical projects you can undertake. When it works, you have a self-owned infrastructure of mobile IP addresses that you control completely. When it breaks, you are debugging USB power issues, flaky modem firmware, and carrier-specific quirks at 3 AM.

    This guide covers everything from the hardware shopping list to the software configuration, with honest assessments of what works, what does not, and when you should consider using a managed service like DataResearchTools instead.

    Why Build Your Own Mobile Proxy Farm?

    Advantages

    • Full control: You own the hardware, the IPs, and the data path. No third party can see your traffic.
    • Cost efficiency at scale: After the initial hardware investment, ongoing costs are primarily SIM card data plans.
    • Custom configuration: Rotation intervals, sticky sessions, geographic targeting, and authentication are all under your control.
    • Learning experience: Understanding how mobile proxies work at the hardware level makes you a better user of commercial services.

    Disadvantages

    • Upfront cost: Hardware is not cheap, especially when you factor in dongles, SIM cards, powered USB hubs, and cooling.
    • Maintenance burden: Hardware failures, SIM deactivation, firmware updates, and carrier changes require ongoing attention.
    • Limited IP pool: A single SIM card gives you one carrier’s IP pool. Commercial services like DataResearchTools pool thousands of connections across multiple carriers.
    • Geographic limitation: Your proxies are tied to wherever your hardware is physically located.
    • Scalability ceiling: Beyond 50-100 modems, power management, heat, and USB reliability become serious engineering challenges.

    Hardware Requirements

    Core Components

    Component Recommended Quantity (10 Proxy Setup) Approximate Cost
    Raspberry Pi 4B (4GB) Yes 2-3 (each handles 4-5 modems) $35-55 each
    Raspberry Pi 5 (8GB) Better performance 2 (each handles 5-6 modems) $80 each
    4G USB Dongle Huawei E3372h or E3276 10 $15-30 each
    5G USB Dongle Quectel RM520N-GL 10 $80-150 each
    Powered USB Hub 10-port, 60W minimum 2-3 $30-50 each
    SIM Cards Data-only plans preferred 10 Varies by carrier
    MicroSD Cards 32GB+ Class 10 2-3 $8-12 each
    Ethernet Switch Gigabit, 8+ ports 1 $20-30
    Power Supply Reliable UPS recommended 1 $50-100
    Cooling Fan or heat sinks Per Pi $5-15

    Choosing the Right 4G Dongle

    Not all USB dongles work well for proxy purposes. The key requirements are:

    1. AT command support: The dongle must accept AT commands for IP rotation (airplane mode toggle).
    2. Linux compatibility: Must work with Raspberry Pi OS without proprietary drivers.
    3. CDC Ethernet mode: The dongle should present as a network interface, not a serial modem requiring PPP.
    4. Band compatibility: Must support the LTE bands used by your target carrier.

    Recommended dongles for Southeast Asia:

    Dongle Model Mode 4G Bands Linux Support Price
    Huawei E3372h-153 HiLink (Ethernet) B1/3/5/7/8/20 Excellent $15-25
    Huawei E3372h-320 HiLink (Ethernet) B1/3/7/8/20/28 Good $20-30
    ZTE MF833V RNDIS (Ethernet) B1/3/5/7/8/20/28 Good $15-25
    Quectel EC25 QMI/MBIM B1/3/5/7/8/20/28/38/40/41 Excellent $25-40

    LTE Band Reference for Southeast Asia

    Country Major Carriers Primary LTE Bands
    Thailand AIS, DTAC, True B1, B3, B7, B28
    Indonesia Telkomsel, XL, Indosat B1, B3, B5, B8, B40
    Philippines Globe, Smart B1, B3, B5, B7, B28, B40
    Malaysia Maxis, Celcom, Digi B1, B3, B7, B8, B28, B40
    Vietnam Viettel, Mobifone, Vinaphone B1, B3, B7, B38, B40, B41
    Singapore Singtel, StarHub, M1 B1, B3, B7, B8, B28

    Software Setup

    Step 1: Prepare the Raspberry Pi

    # Flash Raspberry Pi OS Lite (64-bit) to the SD card
# Boot and run initial configuration

# Update system
sudo apt update && sudo apt upgrade -y

# Install essential packages
sudo apt install -y \
    usb-modeswitch \
    usb-modeswitch-data \
    network-manager \
    iptables \
    squid \
    dante-server \
    python3-pip \
    python3-flask \
    screen \
    htop \
    usbutils

    Step 2: Configure USB Dongles

    # Check connected dongles
lsusb

# You should see entries like:
# Bus 001 Device 003: ID 12d1:14db Huawei Technologies Co., Ltd. E353/E3131

# If the dongle shows as a storage device (CD-ROM mode),
# usb-modeswitch should handle the switch automatically.
# Check if it created a network interface:
ip addr show

# You should see interfaces like:
# eth1, eth2, etc. (for HiLink dongles)
# or wwan0, wwan1 (for QMI/MBIM dongles)

    Step 3: Set Up Network Interfaces

    For each dongle, create a network configuration:

    # /etc/NetworkManager/system-connections/modem1.nmconnection
[connection]
id=modem1
type=gsm
interface-name=cdc-wdm0
autoconnect=true

[gsm]
apn=internet
number=*99#

[ipv4]
method=auto
route-metric=200

[ipv6]
method=auto

    Step 4: Install Proxy Server Software

    Option A: 3proxy (Lightweight, Recommended)

    # Install 3proxy
cd /tmp
git clone https://github.com/3proxy/3proxy.git
cd 3proxy
make -f Makefile.Linux
sudo make -f Makefile.Linux install

    Configure 3proxy for each modem:

    # /etc/3proxy/3proxy.cfg
daemon
log /var/log/3proxy/3proxy.log

# Authentication
users admin:CL:your_password

# Proxy for modem 1 (eth1)
auth strong
allow admin
proxy -p10001 -i0.0.0.0 -e192.168.8.100
socks -p20001 -i0.0.0.0 -e192.168.8.100

# Proxy for modem 2 (eth2)
proxy -p10002 -i0.0.0.0 -e192.168.9.100
socks -p20002 -i0.0.0.0 -e192.168.9.100

    Option B: Squid (Full-Featured)

    # /etc/squid/squid.conf

# Access control
acl proxy_users proxy_auth REQUIRED
http_access allow proxy_users
http_access deny all

# Authentication
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwords
auth_param basic realm Proxy Server

# Modem 1 listener
http_port 10001
tcp_outgoing_address 192.168.8.100 port 10001

# Modem 2 listener
http_port 10002
tcp_outgoing_address 192.168.9.100 port 10002

    Step 5: IP Rotation Script

    The core feature of a mobile proxy is IP rotation. This is achieved by briefly disconnecting the modem from the cellular network, forcing the carrier to assign a new IP:

    #!/usr/bin/env python3
"""
IP Rotation script for mobile proxy farm.
Rotates IP by toggling the modem's cellular connection.
"""

import subprocess
import time
import requests
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("ip_rotator")

class ModemRotator:
    def __init__(self, modem_config):
        self.config = modem_config

    def get_current_ip(self, interface):
        """Get the current external IP through this interface"""
        try:
            proxy = {"http": f"http://127.0.0.1:{self.config['proxy_port']}"}
            response = requests.get(
                "https://api.ipify.org?format=json",
                proxies=proxy,
                timeout=10
            )
            return response.json()["ip"]
        except Exception as e:
            logger.error(f"Failed to get IP: {e}")
            return None

    def rotate_hilink(self, modem_ip="192.168.8.1"):
        """Rotate IP on Huawei HiLink dongles via their web API"""
        import xml.etree.ElementTree as ET

        # Get session token
        token_response = requests.get(
            f"http://{modem_ip}/api/webserver/SesTokInfo"
        )
        root = ET.fromstring(token_response.text)
        session = root.find("SesInfo").text
        token = root.find("TokInfo").text

        headers = {
            "Cookie": session,
            "__RequestVerificationToken": token,
            "Content-Type": "application/xml"
        }

        # Toggle airplane mode ON
        data = '<?xml version="1.0" encoding="UTF-8"?><request><dataswitch>0</dataswitch></request>'
        requests.post(
            f"http://{modem_ip}/api/dialup/mobile-dataswitch",
            data=data,
            headers=headers
        )

        time.sleep(3)

        # Get new token (session may have changed)
        token_response = requests.get(
            f"http://{modem_ip}/api/webserver/SesTokInfo"
        )
        root = ET.fromstring(token_response.text)
        session = root.find("SesInfo").text
        token = root.find("TokInfo").text

        headers["Cookie"] = session
        headers["__RequestVerificationToken"] = token

        # Toggle airplane mode OFF
        data = '<?xml version="1.0" encoding="UTF-8"?><request><dataswitch>1</dataswitch></request>'
        requests.post(
            f"http://{modem_ip}/api/dialup/mobile-dataswitch",
            data=data,
            headers=headers
        )

        time.sleep(5)  # Wait for new IP assignment

    def rotate_at_command(self, device_path="/dev/ttyUSB0"):
        """Rotate IP using AT commands (for non-HiLink modems)"""
        import serial

        ser = serial.Serial(device_path, 115200, timeout=5)

        # Enable airplane mode
        ser.write(b"AT+CFUN=4\r\n")
        time.sleep(3)

        # Disable airplane mode
        ser.write(b"AT+CFUN=1\r\n")
        time.sleep(8)

        ser.close()

    def rotate_and_verify(self, method="hilink"):
        """Rotate IP and verify the change"""
        old_ip = self.get_current_ip(self.config["interface"])
        logger.info(f"Current IP: {old_ip}")

        if method == "hilink":
            self.rotate_hilink(self.config["modem_ip"])
        elif method == "at_command":
            self.rotate_at_command(self.config["device_path"])

        # Verify new IP
        new_ip = self.get_current_ip(self.config["interface"])
        logger.info(f"New IP: {new_ip}")

        if new_ip and new_ip != old_ip:
            logger.info("IP rotation successful")
            return True
        else:
            logger.warning("IP rotation may have failed")
            return False


# Configuration for a 10-modem setup
MODEM_CONFIGS = [
    {
        "id": "modem_1",
        "interface": "eth1",
        "modem_ip": "192.168.8.1",
        "proxy_port": 10001,
        "socks_port": 20001,
        "method": "hilink"
    },
    {
        "id": "modem_2",
        "interface": "eth2",
        "modem_ip": "192.168.9.1",
        "proxy_port": 10002,
        "socks_port": 20002,
        "method": "hilink"
    },
    # ... additional modems
]

    Step 6: Management API

    Build a simple API to control your proxy farm remotely:

    from flask import Flask, jsonify, request
import threading

app = Flask(__name__)

# Initialize rotators for each modem
rotators = {}
for config in MODEM_CONFIGS:
    rotators[config["id"]] = ModemRotator(config)

@app.route("/api/proxies", methods=["GET"])
def list_proxies():
    """List all available proxies"""
    proxies = []
    for modem_id, rotator in rotators.items():
        ip = rotator.get_current_ip(rotator.config["interface"])
        proxies.append({
            "id": modem_id,
            "http_port": rotator.config["proxy_port"],
            "socks_port": rotator.config["socks_port"],
            "current_ip": ip,
            "interface": rotator.config["interface"]
        })
    return jsonify(proxies)

@app.route("/api/rotate/<modem_id>", methods=["POST"])
def rotate_ip(modem_id):
    """Rotate IP for a specific modem"""
    if modem_id not in rotators:
        return jsonify({"error": "Modem not found"}), 404

    rotator = rotators[modem_id]
    success = rotator.rotate_and_verify()

    return jsonify({
        "modem_id": modem_id,
        "success": success,
        "new_ip": rotator.get_current_ip(rotator.config["interface"])
    })

@app.route("/api/rotate/all", methods=["POST"])
def rotate_all():
    """Rotate IPs for all modems"""
    results = {}
    threads = []

    def rotate_modem(modem_id, rotator):
        success = rotator.rotate_and_verify()
        results[modem_id] = {
            "success": success,
            "new_ip": rotator.get_current_ip(rotator.config["interface"])
        }

    for modem_id, rotator in rotators.items():
        t = threading.Thread(target=rotate_modem, args=(modem_id, rotator))
        threads.append(t)
        t.start()

    for t in threads:
        t.join()

    return jsonify(results)

@app.route("/api/health", methods=["GET"])
def health_check():
    """Check health of all modems"""
    health = {}
    for modem_id, rotator in rotators.items():
        ip = rotator.get_current_ip(rotator.config["interface"])
        health[modem_id] = {
            "status": "online" if ip else "offline",
            "ip": ip
        }
    return jsonify(health)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

    Scaling Your Proxy Farm

    10-Modem Setup (Starter)

    • 2 Raspberry Pi 4B units
    • 2 powered USB hubs (7-port each)
    • 10 Huawei E3372h dongles
    • 10 SIM cards
    • Total hardware cost: approximately $400-600

    25-Modem Setup (Medium)

    • 4-5 Raspberry Pi 4B units
    • 4 powered USB hubs (7-port each)
    • 25 dongles
    • 25 SIM cards
    • Consider a dedicated mini server rack
    • Total hardware cost: approximately $900-1,400

    50+ Modem Setup (Large)

    • Switch to dedicated x86 servers (Intel NUC or similar)
    • Industrial USB hubs with individual port power management
    • Professional rack mounting
    • Dedicated cooling solution
    • UPS power backup
    • Total hardware cost: approximately $2,500-5,000+

    Common Scaling Challenges

    Challenge Cause Solution
    USB disconnections Power insufficient Higher wattage USB hubs, dedicated power per dongle
    Overheating Dense modem packing Active cooling, spacing, air conditioning
    IP reuse Small carrier IP pool Multiple carriers, more SIM cards
    Modem hang Firmware bugs Scheduled daily reboots, watchdog scripts
    Network conflicts Multiple subnets Careful network configuration, VLAN management

    When to DIY vs When to Use a Service

    Build Your Own When:

    • You need complete traffic privacy (no third-party can inspect your data)
    • You require specific carrier IPs that commercial services do not offer
    • You have the technical skills and time for ongoing maintenance
    • Your proxy usage is consistent and predictable (not bursty)
    • You are in a geographic location where commercial proxy services are limited

    Use DataResearchTools When:

    • You need proxies across multiple countries (DataResearchTools covers all major SEA markets)
    • You want instant scalability without hardware procurement delays
    • You need high uptime guarantees (99.9% SLA)
    • Your team lacks the hardware/networking expertise for farm management
    • You need access to thousands of IPs (far more than a small farm provides)
    • You want to avoid the capital expenditure of hardware
    • You need redundancy and failover that a single-location farm cannot provide

    Hybrid Approach

    Many professional users run a small DIY farm for their most sensitive operations (where complete traffic control matters) while using DataResearchTools mobile proxies for high-volume, multi-geography tasks. This gives the best of both worlds: privacy when you need it, and scale when you need it.

    Troubleshooting Common Issues

    Modem Not Detected

    # Check USB devices
lsusb

# Check dmesg for errors
dmesg | tail -50

# Try resetting USB bus
sudo usbreset /dev/bus/usb/001/003

# If modem is in CD-ROM mode, force switch
sudo usb_modeswitch -v 12d1 -p 1f01 -M '55534243123456780000000000000a11062000000000000100000000000000'

    IP Not Rotating

    # Verify modem has network connection
ping -I eth1 8.8.8.8

# Check if AT commands reach the modem
echo -e "AT+CFUN?\r" > /dev/ttyUSB0
cat /dev/ttyUSB0

# Try manual network disconnect/reconnect
nmcli connection down modem1 && sleep 5 && nmcli connection up modem1

    High Latency

    • Check signal strength: AT+CSQ command (values 10-31 are acceptable)
    • Try a different carrier with better coverage at your location
    • Position antennas near windows or use external antennas
    • Avoid USB 2.0 hubs; use USB 3.0 for better throughput

    Security Considerations

    Exposing Your Proxy Farm to the Internet

    If your proxy farm is accessible from the internet:

    1. Always use authentication: Never run an open proxy
    2. Use a VPN or SSH tunnel: Instead of exposing proxy ports directly
    3. Firewall rules: Only allow connections from your known IP addresses
    4. Rate limit management API: Prevent brute-force attacks on your control panel
    5. Log everything: Monitor for unauthorized access attempts

    SIM Card Security

    • Register SIM cards in compliance with local regulations
    • Store SIM PINs securely
    • Monitor for unusual charges or carrier messages indicating misuse
    • Keep spare SIM cards ready for quick replacement

    Conclusion

    Building a 4G/5G mobile proxy farm with Raspberry Pi is a technically satisfying project that gives you complete control over your proxy infrastructure. The key components are reliable USB dongles, properly configured Linux networking, proxy server software, and an IP rotation mechanism.

    However, be realistic about the limitations. A DIY farm of 10-25 modems provides a fraction of the IP diversity and geographic coverage that a commercial service like DataResearchTools offers out of the box. The maintenance burden is real, and hardware failures will happen at inconvenient times.

    For most use cases, the optimal strategy is to start with DataResearchTools mobile proxies for immediate coverage across Southeast Asian markets, and build a small DIY farm alongside it for specialized needs. This way you get the scale and reliability of a professional service while maintaining the control and privacy of self-owned infrastructure for your most sensitive operations.

    Whether you build, buy, or combine both approaches, understanding how mobile proxy farms work at the hardware level makes you a more effective proxy user and helps you make better decisions about your infrastructure investments.

    Related Reading