AI provider boundaries, timeouts and bounded retries

What you will learn

• Separate provider transport failures from invalid responses and wrong meaning.
• Limit total attempts, retry only named transient failures, and stop on rate limits or permanent rejection.
• Test a provider boundary with deterministic fakes without waiting, networking or buying API access.

Theory: a boundary makes failure explicit

An adapter is a small boundary between application rules and an outside dependency. A provider interface describes what the caller needs: prompt text, an explicit timeout value and response text. Python Protocol helps a type checker compare that interface; it does not automatically validate every runtime value or interrupt a slow function.

A class groups behavior and data for one object. ScriptedProvider.init prepares a fake object; self refers to that object. Its outcomes list contains the remaining scripted responses, while calls records what was requested. The constructor copies the caller's list so consuming a response does not remove it from the original list. The small exception classes use pass because their distinct names, not extra methods, identify the failure category. The annotation str | Exception means response text or an exception instance; the constructor also checks those values at runtime.

An attempt is one call, including the first call. A retry is an extra call after failure. Here max_attempts=3 means at most three total calls, not an initial call plus three retries. Only ProviderTimeout and ProviderUnavailable are retried. A rate limit means stop and let a later scheduling policy decide; a rejected request or invalid response is not repaired by blindly sending it again. An unknown programming error propagates instead of being disguised as a successful answer.

A timeout means the caller did not obtain a response in time. It does not prove that the remote service did nothing. Retrying a real POST, paid model call or action can repeat a side effect or charge. Before adding a real adapter, understand its HTTP client/SDK retry defaults, provider idempotency support and whether your application can safely repeat the operation. One retry layer is easier to budget than several layers multiplying calls.

This lab has no real transport. Scripted exceptions imitate named failures; they do not prove real socket cancellation. The timeout value is validated and forwarded, but only a real HTTP client with appropriate connection/read/total-deadline handling can enforce its actual policy. A bounded call count is not a wall-clock deadline. The injected wait function records requested delays in a list; it does not sleep. Production backoff/jitter, Retry-After handling, cancellation, quotas and a total deadline need a separate design and measurement.

Worked example: deterministic provider scenarios

Use Python3.13 standard library. Put this file, fake_triage_provider.py, beside prompt_contract.py from the preceding prompt-contract lesson. Keep both filenames lowercase, including on Ubuntu. The adapter reuses that lesson's build_ticket_prompt and parse_triage functions rather than duplicating validation. No pip dependency, account token or API key is required. Only the supplied dummy ticket is used.

"""Offline provider-boundary exercise. No network, real sleep or model call."""
import math
from collections.abc import Callable
from typing import Protocol

from prompt_contract import build_ticket_prompt, parse_triage


class ProviderTimeout(Exception):
    pass


class ProviderUnavailable(Exception):
    pass


class ProviderRateLimited(Exception):
    pass


class ProviderRejected(Exception):
    pass


class InvalidProviderResponse(Exception):
    pass


class AttemptsExhausted(Exception):
    pass


class TriageProvider(Protocol):
    def generate(self, prompt: str, *, timeout_seconds: float) -> str:
        ...


class ScriptedProvider:
    def __init__(self, outcomes: list[str | Exception]):
        if not isinstance(outcomes, list) or not outcomes:
            raise ValueError("Provide a nonempty list of scripted outcomes.")
        if not all(isinstance(item, (str, Exception)) for item in outcomes):
            raise ValueError("Each outcome must be response text or an exception.")
        self.outcomes = outcomes.copy()
        self.calls: list[dict[str, object]] = []

    def generate(self, prompt: str, *, timeout_seconds: float) -> str:
        self.calls.append({"prompt": prompt, "timeout_seconds": timeout_seconds})
        if not self.outcomes:
            raise AssertionError("The fake fixture ran out of outcomes.")
        outcome = self.outcomes.pop(0)
        if isinstance(outcome, Exception):
            raise outcome
        return outcome


def classify_ticket(
    provider: TriageProvider,
    record: dict[str, str],
    *,
    wait: Callable[[float], None],
    max_attempts: int = 3,
    timeout_seconds: float = 2.0,
) -> dict[str, object]:
    if type(max_attempts) is not int or not 1 <= max_attempts <= 3:
        raise ValueError("Use 1 to 3 total attempts, excluding booleans.")
    if type(timeout_seconds) not in (int, float) or not 0 < timeout_seconds <= 10 or not math.isfinite(timeout_seconds):
        raise ValueError("Use a finite timeout above 0 and at most 10 seconds.")
    if not callable(wait):
        raise ValueError("Provide an explicit wait function.")
    prompt = build_ticket_prompt(record)
    allowed_ids = {record["id"]}
    for attempt in range(1, max_attempts + 1):
        try:
            raw = provider.generate(prompt, timeout_seconds=float(timeout_seconds))
        except (ProviderTimeout, ProviderUnavailable):
            if attempt == max_attempts:
                raise AttemptsExhausted("Transient attempts exhausted.") from None
            wait(round(0.1 * attempt, 1))
            continue
        # Rate limits/rejections and unexpected exceptions deliberately propagate.
        try:
            result = parse_triage(raw, allowed_ids)
        except ValueError:
            raise InvalidProviderResponse("Response contract failed.") from None
        return {"attempts": attempt, **result}
    raise AssertionError("The validated loop must return or raise.")


if __name__ == "__main__":
    record = {"id": "r1", "text": "The task page returns HTTP 500 after valid input."}
    good = '{"label":"bug","evidence":["r1"]}'
    scenarios = [
        ("success", [good]),
        ("timeout-then-success", [ProviderTimeout(), good]),
        ("unavailable-then-success", [ProviderUnavailable(), good]),
        ("rate-limited", [ProviderRateLimited(), good]),
        ("rejected", [ProviderRejected(), good]),
        ("malformed", ['{"label":"bug","evidence":"r1"}', good]),
        ("exhausted", [ProviderTimeout(), ProviderTimeout(), ProviderTimeout()]),
        ("wrong-label", ['{"label":"question","evidence":["r1"]}']),
    ]
    for name, outcomes in scenarios:
        fake = ScriptedProvider(outcomes)
        waits: list[float] = []
        try:
            result = classify_ticket(fake, record, wait=waits.append)
        except (ProviderRateLimited, ProviderRejected, InvalidProviderResponse, AttemptsExhausted):
            print(f"{name}: stopped; calls={len(fake.calls)} waits={waits}")
        else:
            print(f"{name}: attempts={result['attempts']} label={result['label']} waits={waits}")

Windows PowerShell:

python ./fake_triage_provider.py

Ubuntu Bash:

python3 ./fake_triage_provider.py

Expected output:

success: attempts=1 label=bug waits=[]
timeout-then-success: attempts=2 label=bug waits=[0.1]
unavailable-then-success: attempts=2 label=bug waits=[0.1]
rate-limited: stopped; calls=1 waits=[]
rejected: stopped; calls=1 waits=[]
malformed: stopped; calls=1 waits=[]
exhausted: stopped; calls=3 waits=[0.1, 0.2]
wrong-label: attempts=1 label=question waits=[]

The transient success rows contain one recorded wait of 0.1 seconds. The exhausted row contains two requested waits, 0.1 and 0.2, and no wait after the final attempt. The program never actually sleeps. The fake retains its remaining scripted responses so tests can show that rate limits, rejection and malformed responses stop before consuming a later success.

The wrong-label row returns question because it is structurally valid and cites the allowed record. That is still a semantic miss against the dummy ticket's reviewed bug reference. Transport success, response-contract success and label correctness are separate evidence. This function neither awards course credit nor performs any generated action.

Guided lab

1. Run both files from the same practice folder. Match every output line to its script and count total calls separately from retries.
2. Replace the timeout-then-success script with two timeouts then success. Predict three attempts and waits [0.1, 0.2], then run it.
3. Set max_attempts=1. A transient failure should stop after one call and no wait. Try True, 0 and 4; they should be rejected before any call.
4. Try a zero, negative, infinite or boolean timeout. Check that no provider call occurs. A valid timeout such as 1.5 is forwarded unchanged in meaning to each fake call.
5. Return an invented evidence ID or a JSON object with an extra action field. Confirm InvalidProviderResponse and one call only. Never execute generated text.
6. Compare the original record with a copy after success and after failure. Inspect the fake call log only with approved dummy data; never log real prompts, tokens or personal records by default.

Independent exercise

Create a scenario table with input, scripted outcomes, expected total calls, requested waits and final result/error. Include immediate success, transient recovery, exhaustion, rate limit, rejection, invalid JSON, wrong evidence, valid unknown and a structurally valid wrong label. Build a tiny recording wait function and assertions rather than using real sleep or an internet service.

Hint: a later success in the script is not permission to retry every error. Keep the original caller record and the original outcome list intact; the fake owns a shallow list copy. Exception objects themselves are not deep-copied. Add an unexpected RuntimeError and a wait-function failure; both must propagate without another provider call.

For a future real HTTP adapter, write a design note before implementing it: specify provider endpoint selection, server-only credentials, authorized context, connection/read/total time budgets, one retry layer, Retry-After rules, operation idempotency, maximum cost, cancellation and redacted monitoring. If the server asks you to wait longer than your available budget, stop or schedule a later job; do not shorten its requested delay and immediately retry. Do not activate a paid API for this exercise.

Common mistakes and debugging

• Catching every Exception and returning unknown hides programming errors and misrepresents outages as model uncertainty.
• Retrying invalid JSON, forbidden requests or rate limits indiscriminately consumes budget without a defined repair.
• Calling max_attempts=3 three retries creates four calls. Inspect the exact calls and wait lists.
• Assuming a timeout proves no remote action occurred creates duplicate effects. Read the real API's contract first.
• Treating a forwarded timeout or a fake delay as a tested network deadline is an unsupported claim.
• Importing from another folder without prompt_contract.py causes ModuleNotFoundError. Keep the two lesson files together and use the labelled command.

English practice and glossary

Adapter: boundary to a dependency. Transient: possibly temporary failure. Attempt: one call. Retry: another call after failure. Backoff: a delay policy between attempts. Jitter: controlled variation of delays to reduce synchronized retries. Idempotency: repeating an operation preserves its intended effect. Deadline: maximum allowed total elapsed time. Fake: controlled implementation for a test; not evidence of a real provider's behavior.

Say: “The first attempt failed with __. The policy allowed __ total attempts. We recorded __ waits. We did not call __.” Keep exception names and technical terms in English when explaining in Malayalam.

Interview: “Why not retry every AI failure?” Separate transient transport failures, rate limits, permanent rejection, malformed output and wrong meaning. Explain duplicate charges/effects, total budget and what the deterministic tests do not prove.

Recap and next step

Keep provider calls behind a narrow interface. Validate input before calling, retry only explicit temporary failures within a fixed attempt budget, preserve caller data and validate every returned response. Keep operational failure distinct from semantic uncertainty. Next extend the reviewed evaluation set and safe product behavior before choosing any live provider.

Answer key

Two timeouts followed by success with max_attempts=3 produce three calls and waits [0.1, 0.2]. With a one-attempt budget, a first timeout raises AttemptsExhausted with no wait. Rate limit and rejection propagate immediately; malformed output becomes InvalidProviderResponse and is not retried. Invalid input/options never call the provider. Unexpected provider or wait errors propagate. Valid unknown with empty evidence returns structurally; the wrong label can also return structurally and must fail a separate reviewed semantic check. No live model accuracy, network deadline, real Retry-After behavior, API charge or external action was tested.

Exercise outline: use approved dummy fixtures and a recording fake; assert return/error, call count, exact requested waits, forwarded timeout and unchanged input for every scenario. Keep model meaning tests separate. For a live design, stop when budgets are exhausted and verify idempotency before retrying side-effecting calls.