124. Retry With Exponential Backoff

🔄 Aryan’s LLM API calls sometimes get a 529 Overloaded response. His retry logic retries forever, always waits 1 second, and re-raises the wrong exception type on final failure.

💡 Fun fact: The exponential backoff algorithm was first described by computer scientist Leonard Kleinrock in 1975 for Ethernet collision resolution. Today every major cloud SDK — AWS Boto3, Google Cloud Client, Azure SDK — builds in exponential backoff automatically. The tenacity Python library provides decorators like @retry(stop=stop_after_attempt(3), wait=wait_exponential()) that add retries to any function in one line. AWS also recommends adding “jitter” (random noise) to backoff delays to prevent the “thundering herd” problem where thousands of clients retry at exactly the same moment after a server restart.

⚠️ Watch out: 4xx HTTP errors (400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found) are NOT retryable — they mean YOUR request is wrong. Retrying a 400 will always return 400. Only 5xx server errors (500, 503) and network exceptions (Timeout, ConnectionError) are transient and worth retrying. The broken code retries ALL non-200 statuses, including 4xx, wasting time and potentially triggering rate-limit bans.

🤔 Think about it: A while True loop that increments attempt but never checks it against max_retries will loop forever if every response is a 5xx. How would you test that a function terminates? What does mock_post.call_count tell you after the function returns? What happens to the caller if the function never raises and never returns?

Learning objectives

• Calculate exponential backoff delays using 2 ** attempt
• Distinguish retryable (5xx, network) from fatal (4xx) errors
• Use a bounded for loop to guarantee termination
• Re-raise as a meaningful custom exception after all retries fail
• Add jitter to backoff to avoid thundering herd in production

Key concepts

• exponential backoff — 2 ** attempt delay growth
• retryable vs fatal errors — 5xx vs 4xx
• bounded retry loop — for range(max_retries)
• jitter — random.uniform() to spread retries
• tenacity — production retry library

Try it

Concept detail

Retry with Exponential Backoff

Transient failures (network hiccups, overloaded servers) are normal. A good retry strategy:

1. Retry only transient errors — 5xx server errors, network timeouts
2. Never retry client errors — 4xx means your request is wrong
3. Wait longer each attempt — exponential backoff
4. Stop after N attempts — don’t loop forever
5. Re-raise clearly — raise a meaningful error after all retries fail

Exponential backoff formula

delay = base * (2 ** attempt)
# attempt 0 → 1s
# attempt 1 → 2s
# attempt 2 → 4s
# attempt 3 → 8s

Add jitter in production to avoid thundering herd:

import random
delay = base * (2 ** attempt) + random.uniform(0, 1)

Which errors to retry?

Using tenacity (the easy way)

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1))
def call_api():
    ...

Solution

import time
from unittest.mock import MagicMock

class ApiError(Exception):
    pass

class _FakeRequests:
    """Simulated requests library for Pyodide environment."""
    class HTTPError(Exception):
        def __init__(self, response=None):
            self.response = response

    class Response:
        def __init__(self, status_code=200, json_body=None):
            self.status_code = status_code
            self._json_body = json_body or {}
        def json(self):
            return self._json_body
        def raise_for_status(self):
            if self.status_code >= 400:
                raise _FakeRequests.HTTPError(response=self)

    @staticmethod
    def post(url, json=None, headers=None, timeout=None):
        raise RuntimeError("Real network call — replace with mock in tests")

requests = _FakeRequests()

def call_with_retry(
    url: str,
    payload: dict,
    headers: dict,
    max_retries: int = 3,
    base_delay: float = 1.0,
) -> dict:
    """Call a JSON API with retry on transient failures."""
    last_exc = None
    for attempt in range(max_retries):
        try:
            resp = requests.post(url, json=payload, headers=headers, timeout=30)

            # 4xx = client error (bad request, auth failure) — do NOT retry
            if 400 <= resp.status_code < 500:
                resp.raise_for_status()   # raises HTTPError immediately

            # 5xx = server error — retryable
            if resp.status_code >= 500:
                delay = backoff_delay(attempt, base_delay)
                time.sleep(delay)
                last_exc = requests.HTTPError(response=resp)
                continue

            return resp.json()

        except requests.HTTPError:
            raise   # 4xx errors re-raised immediately, no retry
        except Exception as exc:
            delay = backoff_delay(attempt, base_delay)
            time.sleep(delay)
            last_exc = exc

    raise ApiError(f'API failed after {max_retries} attempts') from last_exc

def backoff_delay(attempt: int, base: float = 1.0) -> float:
    """Calculate exponential backoff delay for attempt number (0-indexed)."""
    return base * (2 ** attempt)  # 1s, 2s, 4s, 8s ...

Tests

from unittest.mock import MagicMock, patch

def _resp(status):
    r = MagicMock()
    r.status_code = status
    r.json.return_value = {'content': [{'text': 'ok'}]}
    if status >= 400:
        r.raise_for_status.side_effect = requests.HTTPError(response=r)
    else:
        r.raise_for_status.return_value = None
    return r

def test_backoff_delay_is_exponential():
    assert backoff_delay(0) == 1.0
    assert backoff_delay(1) == 2.0
    assert backoff_delay(2) == 4.0
    assert backoff_delay(3) == 8.0

def test_backoff_delay_custom_base():
    assert backoff_delay(0, base=0.5) == 0.5
    assert backoff_delay(1, base=0.5) == 1.0
    assert backoff_delay(2, base=0.5) == 2.0

def test_call_with_retry_success_first_try():
    original_post = requests.post
    mock_post = MagicMock(return_value=_resp(200))
    requests.post = mock_post
    try:
        with patch('time.sleep') as mock_sleep:
            result = call_with_retry('http://x', {}, {})
        assert mock_post.call_count == 1
        mock_sleep.assert_not_called()
    finally:
        requests.post = original_post

def test_call_with_retry_retries_on_503():
    responses = [_resp(503), _resp(503), _resp(200)]
    idx = [0]
    def fake_post(*a, **kw):
        r = responses[idx[0]]
        idx[0] += 1
        return r
    original_post = requests.post
    requests.post = fake_post
    try:
        with patch('time.sleep'):
            result = call_with_retry('http://x', {}, {}, max_retries=3)
        assert result == {'content': [{'text': 'ok'}]}
    finally:
        requests.post = original_post

def test_call_with_retry_no_retry_on_400():
    """4xx errors should NOT be retried — they are client errors."""
    original_post = requests.post
    mock_post = MagicMock(return_value=_resp(400))
    requests.post = mock_post
    try:
        with patch('time.sleep') as mock_sleep:
            try:
                call_with_retry('http://x', {}, {}, max_retries=3)
            except requests.HTTPError:
                pass
        assert mock_post.call_count == 1, '400 errors must not be retried'
        mock_sleep.assert_not_called()
    finally:
        requests.post = original_post

def test_call_with_retry_raises_after_max():
    original_post = requests.post
    requests.post = MagicMock(return_value=_resp(503))
    try:
        with patch('time.sleep'):
            try:
                call_with_retry('http://x', {}, {}, max_retries=3)
                assert False, 'Should raise ApiError'
            except ApiError:
                pass
    finally:
        requests.post = original_post

def test_call_with_retry_finite_attempts():
    """Must stop after max_retries, not loop forever."""
    original_post = requests.post
    mock_post = MagicMock(return_value=_resp(503))
    requests.post = mock_post
    try:
        with patch('time.sleep'):
            try:
                call_with_retry('http://x', {}, {}, max_retries=3)
            except ApiError:
                pass
        assert mock_post.call_count == 3
    finally:
        requests.post = original_post

Resources

• tenacity — retry library
• Exponential Backoff and Jitter — AWS Blog
• requests HTTPError
• ArjanCodes — How to Handle Errors in Python
• HTTP Status Codes Reference — MDN