Published on

Scraping: back off, honour Retry-After, and never retry a 404

Scraping: back off, honour Retry-After, and never retry a 404
Authors
On this page

A scraper that gives up on the first failed request is brittle. One that retries everything is worse, because it keeps hammering a server that already asked it to slow down.

I wanted a retry helper that sits in the middle, so I wrote a small one and tested it against a server that fails on purpose. The rule it follows is short:

  • Retry the failures that can clear on their own: 429 and 5xx, plus connection errors and timeouts.
  • Don't retry the rest. A 404 will still be a 404 in a second.
  • If the server says how long to wait, believe it.
Two small robots: one waits patiently at a door with an hourglass sign, the other shrugs at a brick wall and walks away instead of knocking again.

The helper

import random, time, requests

RETRY = {429, 500, 502, 503, 504}

def get(url, tries=5, base=0.2, cap=5):
    for n in range(1, tries + 1):
        r = requests.get(url, timeout=(3, 10))
        if r.status_code not in RETRY:
            return r, n
        ra = r.headers.get("Retry-After")
        wait = float(ra) if ra and ra.isdigit() else min(cap, base * 2**n) * random.uniform(0.5, 1.0)
        print(f"  attempt {n}: {r.status_code}, waiting {wait:.2f}s")
        time.sleep(wait)
    r.raise_for_status()

Three details matter:

  • Timeouts are a tuple. (3, 10) is 3 seconds to connect and 10 to read. With no timeout, requests waits forever.
  • Jitter. Multiplying by a random factor stops many workers from retrying in lockstep.
  • A cap. Without cap, the wait doubles into minutes.

What it did

I pointed it at a local server that returns 429 with Retry-After: 1 on the first call, 503 on the second, and 200 on the third. A second URL always returns 404:

  attempt 1: 429, waiting 1.00s
  attempt 2: 503, waiting 0.73s
flaky -> 200 {'ok': True, 'attempt': 3} after 3 attempts in 1.74 s
missing -> 404 after 1 attempt in 0.0 s
Timeline of the test run: a 429 with Retry-After waits 1.00 s, a 503 waits 0.73 s of jittered backoff, and the third attempt returns 200 after 1.74 s; the 404 URL returns after one attempt.

The 429 wait was exactly the 1 second the server asked for. The 503 had no header, so it fell back to the jittered backoff. The 404 came back at once, with no retries.

Limits of this sketch

  • It doesn't parse a Retry-After that's an HTTP date, only seconds.
  • It retries GET only. Don't retry a POST unless the endpoint is idempotent.
  • Retries do nothing about your request rate. If you get a 429 often, send fewer requests.

Takeaways

  • Sort failures into "might clear" and "won't clear" before you write any retry loop.
  • Let Retry-After beat your own backoff when the server sends it.
  • Always set a timeout, add jitter, and cap the wait.

Frequently asked questions

Which HTTP status codes should a scraper retry?
Retry 429 and 5xx responses such as 500, 502, 503 and 504, plus connection errors and timeouts. Do not retry a 404, because it will still be a 404 a second later.
How should a scraper handle the Retry-After header?
Believe it. If the server sends Retry-After, wait that many seconds instead of using your own backoff. In my test the 429 wait was exactly the 1 second the server asked for, and a 503 without the header fell back to jittered exponential backoff.
Why add jitter and a cap to retry backoff?
Jitter stops many workers from retrying in lockstep, and a cap stops the doubling wait from growing into minutes. Also set a connect and read timeout, because requests waits forever with none.