Calling LLM APIs in Python: Completions, Chat, and Structured Output

Why API calls are the foundation of agents

Every LLM agent, no matter how sophisticated, ultimately rests on one operation: sending a request to a model and parsing its response. Before you build tool-using loops or multi-agent systems, you must be able to call an LLM reliably and get back data your code can trust. This lesson focuses on that primitive.

Setting up your environment

Isolate dependencies in a virtual environment and never hard-code secrets.

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install openai pydantic python-dotenv

Store your key in a .env file (and add .env to .gitignore):

OPENAI_API_KEY=sk-...

Load it in Python:

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI()  # reads OPENAI_API_KEY from the environment

Completions vs. chat

The legacy completions API takes a single text prompt and returns a continuation. It is largely deprecated. The chat completions API takes a list of messages with roles (system, user, assistant) and is the standard for modern instruction-tuned models. Always prefer chat.

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Name three uses of vector databases."},
    ],
    temperature=0.2,
)
print(resp.choices[0].message.content)

Key parameters

• model: which model to call.
• messages: the conversation history; the system message sets behavior.
• temperature: 0 = deterministic, higher = more creative. Use low values for structured tasks.
• max_tokens: cap on the response length.

Getting reliable structured output

Free-form text is hard for code to consume. For agents you usually want JSON. Three escalating techniques:

1. Prompt for JSON — ask the model to reply in JSON. Cheap but unreliable.
2. JSON mode — pass response_format={"type": "json_object"} to guarantee valid JSON syntax (but not a specific shape).
3. Schema-enforced structured output — supply a JSON schema (or a Pydantic model) so the model conforms to your exact fields.

from pydantic import BaseModel

class Movie(BaseModel):
    title: str
    year: int
    genres: list[str]

resp = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Give details for the film Inception."}],
    response_format=Movie,
)
movie = resp.choices[0].message.parsed  # already a validated Movie instance
print(movie.year)  # 2010

Validating output yourself

Even when you only get a JSON string, validate before trusting it. Pydantic turns malformed data into a catchable error instead of a silent bug downstream.

import json
from pydantic import ValidationError

try:
    data = Movie.model_validate_json(raw_json_string)
except ValidationError as e:
    # log, retry, or repair
    print(e)

Error handling and retries

Network calls fail. Wrap requests to handle rate limits and transient errors with exponential backoff.

import time
from openai import RateLimitError, APIError

def call_with_retry(**kwargs):
    for attempt in range(5):
        try:
            return client.chat.completions.create(**kwargs)
        except (RateLimitError, APIError):
            time.sleep(2 ** attempt)
    raise RuntimeError("LLM call failed after retries")

Putting it together

A production-grade call: load secrets from env, use the chat API with a low temperature, request schema-validated output, wrap in retry logic, and validate the parsed object before using it. This pattern is the reliable building block every agent in later lessons depends on.