# Integrating Python Scrapy with XProxy

> For AI tools: this is the Markdown copy of https://xproxy.io/document/integrating-scrapy-with-xproxy. Every XProxy guide has one at its URL + `.md`; the index of all guides is https://xproxy.io/llms.txt.

Send a Scrapy crawler through XProxy with one middleware that changes the IP every N requests.

Send a Scrapy crawler through an XProxy mobile proxy with one downloader middleware, and change the IP every N requests.

In short

**One middleware**

A small downloader middleware sends every request through your XProxy proxy.

**New IP every N requests**

Set a number in `settings.py`. The middleware calls your panel and the modem gets a new IP.

**Easy to check**

A tiny spider logs the exit IP. It must match the **Public IP** of that proxy in your panel.

**You need:** a Scrapy project (`pip install scrapy requests`), and an XProxy panel with a working modem.

## Get your proxy URL

In your XProxy panel open **Proxy Management** and click **Export Proxy**. Pick the **Plain HTTP** preset with `user:pass@host:port`, turn on **Scheme prefix** and click **Copy All**. Each line is one proxy, for example `http://USERNAME:PASSWORD@192.168.1.10:4001`. The port is the one shown for that modem in **Proxy Management**.

The address must work from the computer that runs your code: see [how to copy a proxy in the right format](https://xproxy.io/document/how-use-proxy-format-http-socks5).

## Add the middleware

Add the two settings files below to your project.

**settings.py**

```
XPROXY_URL = "http://USERNAME:PASSWORD@192.168.1.10:4001"  # from Export Proxy
XPROXY_ROTATE_URL = "http://192.168.1.10/api/v1/rotate_ip/position/1"  # optional
XPROXY_ROTATE_EVERY = 150  # new IP after this many requests (0 = never)

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.XProxyMiddleware": 350,
}
```

Change `myproject` to the name of your Scrapy project. Leave `XPROXY_ROTATE_URL` out if you do not want to change the IP from the crawler.

**middlewares.py**

```
import time

import requests
from scrapy.exceptions import NotConfigured

class XProxyMiddleware:
    """Send every request through XProxy and rotate the IP every N requests."""

    def __init__(self, proxy_url, rotate_url, rotate_every):
        self.proxy_url = proxy_url
        self.rotate_url = rotate_url
        self.rotate_every = rotate_every
        self.count = 0

    @classmethod
    def from_crawler(cls, crawler):
        proxy_url = crawler.settings.get("XPROXY_URL")
        if not proxy_url:
            raise NotConfigured("XPROXY_URL is not set")
        return cls(
            proxy_url,
            crawler.settings.get("XPROXY_ROTATE_URL"),
            crawler.settings.getint("XPROXY_ROTATE_EVERY", 0),
        )

    def process_request(self, request, spider):
        request.meta.setdefault("proxy", self.proxy_url)
        if self.rotate_url and self.rotate_every:
            self.count += 1
            if self.count >= self.rotate_every:
                self.count = 0
                self.rotate(spider)

    def rotate(self, spider):
        # Blocking on purpose: the crawl pauses while the modem reconnects.
        try:
            r = requests.get(self.rotate_url, timeout=10)
            r.raise_for_status()
            spider.logger.info("XProxy rotation: %s", r.text)
            time.sleep(6)
        except requests.RequestException as e:
            spider.logger.error("XProxy rotation failed: %s", e)
```

Needs `pip install requests`. Scrapy has no built-in SOCKS5 support, so use the **HTTP** address. A request that sets its own `meta["proxy"]` keeps it.

## Run a spider

This spider fetches your exit IP and logs it. Save it as `myproject/spiders/ip.py` and run it.

## Change the IP

The middleware calls the rotation API when `XPROXY_ROTATE_EVERY` requests have passed, then waits 6 seconds while the modem reconnects. The crawl stops during that time. Position **1** in the URL is the first modem, the one on port **4001** when the starting port is 4000. All rotation calls are in [How to change / rotate IP](https://xproxy.io/document/how-change-rotate-ip).

**Panel asks for a login?** If **Force API Auth** is on, the rotation call needs your dashboard login: use `requests.get(self.rotate_url, auth=("USER", "PASS"), timeout=10)`.

## Check it works

1. Run `scrapy crawl ip`. The log shows `Exit IP:...`.
2. In **Proxy Management**, look at the **Public IP** of that proxy (marked 1). It is the same IP.
3. Set `XPROXY_ROTATE_EVERY = 1`, run the spider twice and compare. The IP changes.

![Proxy Management table with the Public IP of the first proxy marked (demo data)](https://media1.xproxy.io/docs/integrations/int-publicip-v1.webp)

The Public IP your spider should log. Demo data.

## Something not working?

| You see | What to do |
| --- | --- |
| 407 Proxy Authentication Required, or an authentication error | The username or password is wrong or missing. Copy the proxy again after saving **Proxy Settings → Login**. If the password has special characters such as `@` or `:`, URL-encode them in the proxy URL. |
| Connection refused or timed out | The address is not reachable from your computer. A **LAN only** address works only on your own network: use **Public address** or **VPS relay** ([public access](https://xproxy.io/document/how-public-access-xproxy)). Also check the port in **Proxy Management**. |
| The printed IP is your own IP | The request did not go through the proxy. Check that the proxy is passed to the request, the browser or the crawler, and that it is not set only for `http` or only for `https`. |
| The rotation call fails or says `status` is false | Check the position number and the panel address. If **Force API Auth** is on, add your dashboard login to the call. |
| The IP is the same after rotating | Sometimes the carrier gives the modem the same IP. Rotate again. The status call can also answer `COLLISION_IP` when **IP unchanged detect** is on in Rotation Settings. |
| `NotConfigured` in the log, or the middleware is skipped | Check that `XPROXY_URL` is set in `settings.py` and that the path in `DOWNLOADER_MIDDLEWARES` matches your project name. |

## What next?

Change the IP on a timer, or copy your proxies in another format.

[Rotate IP: all ways →](https://xproxy.io/document/how-change-rotate-ip) [Proxy formats →](https://xproxy.io/document/how-use-proxy-format-http-socks5)

---

Source: https://xproxy.io/document/integrating-scrapy-with-xproxy
