Skip to content

Integrating Python Scrapy with XProxy

Ask AI

Ready taskAdd this to my code

Coding agents

Claude CodeOpens the desktop appCodexOpens the desktop appCursorOpens the app from the web

Chat assistants

Open in ChatGPTNew tabOpen in ClaudeNew tab

Copy

View as MarkdownPlain text for AI tools
See the prompt
Read https://xproxy.io/document/integrating-scrapy-with-xproxy.md, the XProxy guide "Integrating Python Scrapy with XProxy". If you cannot open links, download them with curl. Then add it to my project: read my code, use the language and libraries it already uses, keep proxy logins and API keys in environment variables, and show me how to test it.
Other XProxy guides, each with a Markdown copy at its URL + .md, are listed in https://xproxy.io/llms.txt.

The app opens with the prompt filled in. Nothing runs until you press Enter.

Send a Scrapy crawler through an XProxy mobile proxy with one downloader middleware, and change the IP every N requests.

In short

One middleware

A small downloader middleware sends every request through your XProxy proxy.

New IP every N requests

Set a number in settings.py. The middleware calls your panel and the modem gets a new IP.

Easy to check

A tiny spider logs the exit IP. It must match the Public IP of that proxy in your panel.

You need: a Scrapy project (pip install scrapy requests), and an XProxy panel with a working modem.

Get your proxy URL

In your XProxy panel open Proxy Management and click Export Proxy. Pick the Plain HTTP preset with user:pass@host:port, turn on Scheme prefix and click Copy All. Each line is one proxy, for example http://USERNAME:[email protected]:4001. The port is the one shown for that modem in Proxy Management.

The address must work from the computer that runs your code: see how to copy a proxy in the right format.

Add the middleware

Add the two settings files below to your project.

settings.py
XPROXY_URL = "http://USERNAME:[email protected]:4001"  # from Export Proxy
XPROXY_ROTATE_URL = "http://192.168.1.10/api/v1/rotate_ip/position/1"  # optional
XPROXY_ROTATE_EVERY = 150  # new IP after this many requests (0 = never)

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.XProxyMiddleware": 350,
}

Change myproject to the name of your Scrapy project. Leave XPROXY_ROTATE_URL out if you do not want to change the IP from the crawler.

Run a spider

This spider fetches your exit IP and logs it. Save it as myproject/spiders/ip.py and run it.

myproject/spiders/ip.py
import scrapy


class IpSpider(scrapy.Spider):
    name = "ip"
    start_urls = ["https://api.ipify.org?format=json"]

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.json()["ip"])
        yield response.json()
Terminal
scrapy crawl ip

Change the IP

The middleware calls the rotation API when XPROXY_ROTATE_EVERY requests have passed, then waits 6 seconds while the modem reconnects. The crawl stops during that time. Position 1 in the URL is the first modem, the one on port 4001 when the starting port is 4000. All rotation calls are in How to change / rotate IP.

Panel asks for a login? If Force API Auth is on, the rotation call needs your dashboard login: use requests.get(self.rotate_url, auth=("USER", "PASS"), timeout=10).

Check it works

  1. Run scrapy crawl ip. The log shows Exit IP: ....
  2. In Proxy Management, look at the Public IP of that proxy (marked 1). It is the same IP.
  3. Set XPROXY_ROTATE_EVERY = 1, run the spider twice and compare. The IP changes.
Proxy Management table with the Public IP of the first proxy marked (demo data)
1 The Public IP your spider should log. Demo data.

Something not working?

You seeWhat to do
407 Proxy Authentication Required, or an authentication errorThe username or password is wrong or missing. Copy the proxy again after saving Proxy Settings → Login. If the password has special characters such as @ or :, URL-encode them in the proxy URL.
Connection refused or timed outThe address is not reachable from your computer. A LAN only address works only on your own network: use Public address or VPS relay (public access). Also check the port in Proxy Management.
The printed IP is your own IPThe request did not go through the proxy. Check that the proxy is passed to the request, the browser or the crawler, and that it is not set only for http or only for https.
The rotation call fails or says status is falseCheck the position number and the panel address. If Force API Auth is on, add your dashboard login to the call.
The IP is the same after rotatingSometimes the carrier gives the modem the same IP. Rotate again. The status call can also answer COLLISION_IP when IP unchanged detect is on in Rotation Settings.
NotConfigured in the log, or the middleware is skippedCheck that XPROXY_URL is set in settings.py and that the path in DOWNLOADER_MIDDLEWARES matches your project name.

What next?

Change the IP on a timer, or copy your proxies in another format.

Still stuck?A technician can look at your device with a one-time PIN, or chat with us on Telegram.