Integrating Python Scrapy with XProxy
Ask AI
Ready taskAdd this to my code
Coding agents
Claude CodeOpens the desktop appCodexOpens the desktop appCursorOpens the app from the webChat assistants
Open in ChatGPTNew tabOpen in ClaudeNew tabCopy
View as MarkdownPlain text for AI toolsSee the prompt
Read https://xproxy.io/document/integrating-scrapy-with-xproxy.md, the XProxy guide "Integrating Python Scrapy with XProxy". If you cannot open links, download them with curl. Then add it to my project: read my code, use the language and libraries it already uses, keep proxy logins and API keys in environment variables, and show me how to test it. Other XProxy guides, each with a Markdown copy at its URL + .md, are listed in https://xproxy.io/llms.txt.
The app opens with the prompt filled in. Nothing runs until you press Enter.
Send a Scrapy crawler through an XProxy mobile proxy with one downloader middleware, and change the IP every N requests.
In short
A small downloader middleware sends every request through your XProxy proxy.
Set a number in settings.py. The middleware calls your panel and the modem gets a new IP.
A tiny spider logs the exit IP. It must match the Public IP of that proxy in your panel.
You need: a Scrapy project (pip install scrapy requests), and an XProxy panel with a working modem.
Get your proxy URL
In your XProxy panel open Proxy Management and click Export Proxy. Pick the Plain HTTP preset with user:pass@host:port, turn on Scheme prefix and click Copy All. Each line is one proxy, for example http://USERNAME:[email protected]:4001. The port is the one shown for that modem in Proxy Management.
The address must work from the computer that runs your code: see how to copy a proxy in the right format.
Add the middleware
Add the two settings files below to your project.
XPROXY_URL = "http://USERNAME:[email protected]:4001" # from Export Proxy
XPROXY_ROTATE_URL = "http://192.168.1.10/api/v1/rotate_ip/position/1" # optional
XPROXY_ROTATE_EVERY = 150 # new IP after this many requests (0 = never)
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.XProxyMiddleware": 350,
}
Change myproject to the name of your Scrapy project. Leave XPROXY_ROTATE_URL out if you do not want to change the IP from the crawler.
Run a spider
This spider fetches your exit IP and logs it. Save it as myproject/spiders/ip.py and run it.
import scrapy
class IpSpider(scrapy.Spider):
name = "ip"
start_urls = ["https://api.ipify.org?format=json"]
def parse(self, response):
self.logger.info("Exit IP: %s", response.json()["ip"])
yield response.json()
scrapy crawl ipChange the IP
The middleware calls the rotation API when XPROXY_ROTATE_EVERY requests have passed, then waits 6 seconds while the modem reconnects. The crawl stops during that time. Position 1 in the URL is the first modem, the one on port 4001 when the starting port is 4000. All rotation calls are in How to change / rotate IP.
requests.get(self.rotate_url, auth=("USER", "PASS"), timeout=10).Check it works
- Run
scrapy crawl ip. The log showsExit IP: .... - In Proxy Management, look at the Public IP of that proxy (marked 1). It is the same IP.
- Set
XPROXY_ROTATE_EVERY = 1, run the spider twice and compare. The IP changes.

Something not working?
| You see | What to do |
|---|---|
| 407 Proxy Authentication Required, or an authentication error | The username or password is wrong or missing. Copy the proxy again after saving Proxy Settings → Login. If the password has special characters such as @ or :, URL-encode them in the proxy URL. |
| Connection refused or timed out | The address is not reachable from your computer. A LAN only address works only on your own network: use Public address or VPS relay (public access). Also check the port in Proxy Management. |
| The printed IP is your own IP | The request did not go through the proxy. Check that the proxy is passed to the request, the browser or the crawler, and that it is not set only for http or only for https. |
The rotation call fails or says status is false | Check the position number and the panel address. If Force API Auth is on, add your dashboard login to the call. |
| The IP is the same after rotating | Sometimes the carrier gives the modem the same IP. Rotate again. The status call can also answer COLLISION_IP when IP unchanged detect is on in Rotation Settings. |
NotConfigured in the log, or the middleware is skipped | Check that XPROXY_URL is set in settings.py and that the path in DOWNLOADER_MIDDLEWARES matches your project name. |
What next?
Change the IP on a timer, or copy your proxies in another format.

