Serve a model

urun serve deploys a model from the model catalog without you having to write a custom App. It resolves a catalog entry, creates a templated App pinned to that entry, and deploys it through the normal urun deploy pipeline.

Enumerate the catalog

urun serve catalog

Prints the resolvable matrix — every (model, variant, GPU) placement the catalog knows about, with its engine, weights size, and latency targets:

MODEL                VARIANT      GPU      ENGINE              TARGETS (ttft/tps)
qwen-coder           fp8          l4:1     vllm                600ms / 120
glm-5.2              UD-IQ2_M     l4:1     llamacpp            60000ms / 3
diffusiongemma-26b   Q4_K_M       l4:1     llamacpp-diffusion  4000ms / 40
qwen-coder-480b      fp8          b200:8   vllm                350ms / 450

Serve a model

urun serve <id>[:<variant>] [--gpu SPEC]

urun serve qwen-coder                     # default variant, first placement
urun serve qwen-coder:fp8                  # pin the variant
urun serve qwen-coder:fp8 --gpu l4:1       # pin variant + GPU placement
urun serve glm-5.2:UD-IQ2_M --gpu l4:1     # GGUF tiered offload on a single L4

What happens:

resolve catalog entry  →  map entry → serve config  →  create templated App
      →  urun deploy

1. Resolve — <id> with no variant resolves to the model's default variant; with no --gpu the variant's first placement is used. The resolved row carries everything needed to run the model: engine configuration, weights size, latency targets.
2. Create — a templated serve App is created, pinned to that exact entry. You don't author or maintain it.
3. Deploy — the templated App ships through the standard deploy pipeline.

Flags

After it's up

What you get is a live LLM session — the same session primitive every uRun App gets. The client attaches once, the model stays warm across turns, and token deltas stream back the moment they are generated. The published client path is @urun-sh/core: session.requestStream(...) streams a reply token-by-token, and session.complete(...) resolves with the full body when you don't need deltas. Payloads use the industry-standard Chat Completions / Responses shape (OpenAI-compatible), so existing tooling drops in unchanged.

import { App } from '@urun-sh/core'

const session = App('urun-serve', { baseUrl, orgId, jwt }).serve()

// Streaming — one string per token delta, ends on the terminal response
for await (const delta of session.requestStream({
  model: 'qwen-coder',
  messages: [{ role: 'user', content: 'explain MoE routing' }],
  stream: true,
})) {
  process.stdout.write(delta)
}

// Convenience — resolves with the full `chat.completion` body when you don't need deltas
const res = await session.complete({
  model: 'qwen-coder',
  messages: [{ role: 'user', content: 'write quicksort in python' }],
})

The client and the Python serving runtime share one contract, riding the session's own doc + stream primitives: the shared registry on doc key llm, plus a dedicated consumer-addressed response stream per call (llm-resp:<id>) carrying token deltas and a terminal body.

Next

• urun.model / urun.serve (SDK) — the Python equivalent
• Manage the deployed App — status, scale, retire