How AI Has Reshaped API Design and Interviews

A team ships a /chat endpoint that looks ordinary. The application sends text, the server returns JSON, and the user waits for a single answer. Then the backend switches from hand-written rules to a language model, and assumptions that held for years start failing in production. One answer arrives in two seconds, and the next takes twelve. Long conversations overflow the context limit and have to be trimmed, some questions trigger a search tool before the model can reply, and users abandon the screen unless the text streams in as it is written. Worst of all, some answers read fluently and are simply wrong. The route never changed, but the design behind it did.

This lesson is about that design. We stay at the product architecture level and ask two questions: what can a client observe, and what must the API promise? Transformer internals and model training are out of scope.

That framing matters because an AI-backed endpoint behaves less like a plain request handler and more like a small protocol. Instead of one final answer, the client may receive partial output, a job identifier, token usage, model metadata, or a refusal.

The new boxes in modern architectures

A conventional endpoint validates input, runs business logic, reads from storage, and returns a single response. An AI-backed endpoint adds several new stages to that path, and many of them shape what clients must handle.

The request still enters through the API gateway, which authenticates the caller, applies rate limits, and routes the request onward. From there, the orchestration layer takes over. It assembles the prompt, attaches system instructions, fetches context, and trims everything to fit a token budgetThe maximum amount of input and output text the request can consume within model limits and cost rules.. Before inference begins, the request may also pass through moderation, tool selection, and provider routing.

The components that usually become contract-visible are listed below.

• Prompt assembly: The server may rewrite or enrich the request. This adds latency and can require truncation metadata in the response.

• Context management: Retrieved passages or earlier chat responses may be dropped, summarized, or reordered, which makes results harder to repeat.

• Tool registry: The orchestrator exposes named tools with parameter schemas, and the model can request them during execution.

• Fallback routing: If the primary model is slow or unavailable, a backup model takes over, which often changes response quality and metadata.

• Usage metering: The system counts prompt and completion tokens so quotas, billing, and client dashboards stay accurate.

• Streaming transport: The server may send partial chunks over Server-Sent Events (SSE) or WebSocket instead of one blocking payload.

Interviewers expect you to use a few terms precisely, i.e, first-token latencyIt is the time from request acceptance until the first streamed token reaches the client, and it is what users feel as responsiveness in a chat interface., finish reasonIt is the machine-readable reason generation stopped, such as a normal stop, a length limit, a refusal, or a tool call., an async taskIt is a long-running request tracked by an identifier and retrieved later., a partial resultAny output delivered before generation is complete., and model version pinningIt means requesting a specific model version so that provider updates don't silently change behavior.. The diagram below shows where these stages sit in the request path and which ones surface in the API.

Where classic assumptions break

With CRUD-style APIs, teams often assumed the same request produced the same response, latency was mostly bounded by infrastructure health, and cost scaled weakly with payload size. An AI-backed system breaks all three assumptions.

The most visible change is variability. Given the same prompt, the system can produce different wording or choose different tools, because sampling, retrieval order, and provider-side updates all affect execution. Testing therefore looks less like checking one exact string and more like judging whether outputs stay within an acceptable range.

What changes in the response

A single JSON envelope is often no longer enough. Depending on the endpoint, the client may need partial chunks, usage counters, finish reasons, model identifiers, or a task handle for later retrieval. The endpoint stops behaving like a one-shot function call and starts behaving like a conversation with states.

New visible states

Clients often need to handle several distinct states during one request life cycle:

• Partial output: The server can emit early tokens before the final answer exists.

• Policy refusal: The model can decline the task even when the transport and infrastructure are healthy.

• Tool interruption: The model may stop generation and request an external tool before continuing.

• Length stop: The system can halt because output reached a configured cap, not because reasoning is complete.

New failure semantics

Failures also become more product-shaped than transport-shaped.

• Timeout after partial data: The connection closes, but the stream already contains useful text.

• Invalid tool arguments: The model requests a tool with malformed parameters, even when the schema is documented.

• Model drift: A provider update changes format, tone, or reliability with no change to client code.

• Fluent wrong answer (hallucination): The request succeeds technically while failing product expectations.

Once those assumptions break, endpoint design stops being a packaging exercise and becomes a set of explicit contract choices.

The new design levers

Teams now choose endpoint shape the way they once chose database indexes or cache strategy. The request path still runs through familiar infrastructure, but the visible design depends on how long the task runs, how costly tokens are, and whether the model may call tools.

Sync vs. Streaming vs. Async

Start with the product's latency budget. A short question in a chat window is usually streamed, so the UI shows progress within a second or two. A long report upload is usually handled as an async task that returns tracking metadata immediately. Each mode fits a different kind of interaction:

• Sync: The server returns one complete response. This works for short, predictable tasks such as classification or extraction, as long as the expected completion time fits the interaction.

• Streaming: The orchestrator forwards chunks as they are generated. The client renders partial text and treats the final done event, which carries usage and finish metadata, as the signal that the response is complete.

• Async: The request is saved under a task_idA unique identifier the client uses to poll, cancel, or fetch the final result of long-running work. , so uploads, retries, and tool execution can continue without holding the connection open. The client retrieves the result later by polling or through a webhook.

Exposing budgets and tool boundaries

AI contracts increasingly expose input and output constraints. The request may include max output tokens, truncation policy, and model tier. The response may include prompt tokens, completion tokens, total tokens, and model version. This is like a taxi meter on the dashboard. The trip can continue only while time and budget remain.

Tool calling extends the contract further. The model receives tool definitions, returns a tool call with typed arguments, and may wait for a tool result before continuing. Side effects require permission boundaries, correlation IDs, and typed error responses so retried calls do not duplicate writes.

The comparison below summarizes the main levers and the client behavior they force.

How the interview changed

Interview prompts sound the same on the surface, but the evaluation bar has moved. If you are asked to design a chat API, naming endpoints and storage is no longer enough. You also need to justify the endpoint shape from how users interact with it.

For interactive chat, a strong answer may choose streaming and explain that first-token latency matters more than full completion time. For long-document summarization, a strong answer may choose async submission, status retrieval, and idempotent retries because processing time and token cost scale with document size. For an assistant that uses search or calendar tools, a strong answer may separate generation from side effects and define tool schemas and failure handling.

Strong answers also name the cost of each choice:

• Streaming: Improves perceived responsiveness, but requires client parsers, cancellation behavior, and rules for partial failures.

• Async jobs: Improve reliability for long tasks, but add status endpoints, task stores, and idempotency keys.

• Larger models: Improve answer quality, but can break p95 latency targets, throughput, or budget.

These choices pull against each other, which is what the triangle below shows.

After this interview lens, the final step is to compress everything into one reusable contract mindset.

The design mindset to carry forward

Treat an AI endpoint as a protocol. The request enters a path that may assemble prompts, trim context, route across models, invoke tools, stream partial output, and finally stop for a typed reason. Every one of those transitions can become client-visible.

That mindset changes what you document and what you defend in interviews. Clients need to understand partial responses, task life cycle, token accounting, model metadata, tool invocation, and probabilistic behavior. Servers need explicit schemas, typed finish reasons, model pinning, fallback paths, usage observability, and deterministic guards around high-risk actions.

A strong architecture answer usually separates two lanes.

• Deterministic control lane: Authentication, permissions, billing, writes, and irreversible actions stay in ordinary validated services.

• Probabilistic generation lane: Drafting, summarization, extraction, classification, and conversational responses run through model orchestration with budgets and fallbacks.

Conclusion

The strongest product architecture answers separate deterministic control paths from model-driven generation and then document the states, metadata, and failure modes clients must handle. In the next lesson, we will extend that shift from API contracts into engineering roles, tooling, and interview expectations.