Use Sense for realtime work
Connect to the Sense Model API in a Session to assess inputs continuously
This page walks through using the Sense SDK to run the Sense Model API in a Session for realtime work. You will do the following:
- Get a uRun API key.
- Prepare the Sense SDK.
- Declare a standing question set, a cadence for assessing the questions, and an initial state.
- Prepare audio, video, or text inputs to assess.
- Open a Session and start streaming.
- Read the answers.
- React to decisions.
Sense is a decision model: you configure typed questions and streaming media, and Sense returns typed answers with probabilities and confidence. If you are new to the model, read the Sense overview first.
The Sense SDK registers your questions, connects and authenticates your Sessions, and assembles streamed answers into waves for your application to act on. The Sense SDK is available in Python and TypeScript.
This page walks through your first Session with the Sense SDK. For more information, refer to the Sense reference.
Prerequisites
- Python 3.10+.
- A uRun user account so you can make an API key.
- Node.js 22+.
- A uRun user account so you can make an API key.
Get an API key
Create an API key:
- In the console, go to Settings → API Keys and click + Create API key.
- Enter a Label for the key that will help you identify where it is used later.
- Select Create key.
- Copy the generated secret and store it somewhere safe. It is not retrievable later.
Prepare the Sense SDK
-
Install the SDK in your language of choice.
terminal pip install urun-senseterminal npm install @urun-sh/sense -
Set the
URUN_API_KEYenvironment variable for authentication:terminal export URUN_API_KEY="urun_sk_YOUR_KEY"
Declare questions
In a Session, Sense continuously evaluates a standing question set on a cadence measured in hertz (default and maximum 12.5). You declare the questions on a Sense client, and the client registers them when a Session opens.
-
Create the client. Sense evaluates your standing question set against the streaming inputs on a cadence measured in hertz; the default is
12.5, which is also the maximum — you cannot configure a faster cadence.Python from sense import Sense app = Sense()TypeScript import { Sense } from "@urun-sh/sense"; const app = new Sense(); -
Declare your standing set of questions. This example demonstrates all three types of questions:
choice,noul, andscore.Python route = app.choice( "route", instructions="Which team should handle this message?", criteria={"billing": "Charges, invoices, refunds", "technical": "Bugs, errors, setup"}, ) wants_refund = app.noul( "wants_refund", instructions="Is the person asking for a refund?", criteria={ "true": "the person is asking for their money back", "false": "the person is not asking for a refund", }, ) urgency = app.score( "urgency", instructions="How urgent is this message?", criteria=[ "Low — routine, no deadline", "Medium — needs attention today", "High — blocking, act now", ], )TypeScript const route = app.choice("route", { instructions: "Which team should handle this message?", criteria: { billing: "Charges, invoices, refunds", technical: "Bugs, errors, setup" }, }); const wantsRefund = app.noul("wants_refund", { instructions: "Is the person asking for a refund?", criteria: { true: "the person is asking for their money back", false: "the person is not asking for a refund", }, }); const urgency = app.score("urgency", { instructions: "How urgent is this message?", criteria: [ "Low — routine, no deadline", "Medium — needs attention today", "High — blocking, act now", ], }); -
Define an initial text
statethat provides context about your streaming inputs. Use it for information that is not in the streaming media itself, such as who is speaking. You will push it as the Session's first text input after you open the Session.Python STATE = "The person is on a live support call"TypeScript const STATE = "The person is on a live support call";
Prepare streaming inputs
Sense assesses three input modalities, each in a fixed format:
- Audio: raw PCM16, mono, at 24 kHz, in small chunks (we recommend approximately 40 ms).
- Video: a JPEG frame of at most 256 KB. Sense assesses the most recent frame; a frame persists until a new one replaces it.
- Text: plain strings.
session.attach(...) is the one input path: call it before session.run() (which takes no input arguments) and hand it the source — the SDK captures, converts, and pushes chunks in the right format for you. Both languages take one sources object. In TypeScript, a browser MediaStream rides the stream key — the mic is resampled to 24 kHz PCM16 and framed into ~40 ms chunks, and camera frames are sampled to JPEG at the Session cadence. In Python, attach takes audio=, video=, and text= sources. Each source is an iterable or a zero-arg callable that yields the next chunk (null/None ends it).
Open a Session and start streaming
-
Open a Session.
Python session = app.session()TypeScript const session = app.session();The SDK connects to the Sense Model API when you attach or run —
session.attach(...)starts the Session, dialing in the background — authenticates with yourURUN_API_KEY, registers the questions you declared earlier, and handles the Session lifecycle. -
Start streaming. Push your initial text state, attach your live source with
session.attach(...), then run the Session withrun(), which yields answers as questions are assessed.Python # Attach the initial state (the first item of the text source rides the # first turn) and the live source; the SDK captures, converts, and # pushes chunks in the right format. attach returns a handle — call # attached.detach() to stop sampling when you are done. attached = await session.attach(audio=mic_chunks, text=[STATE]) async for wave in session.run(): # code for consuming answers directly as described below ...TypeScript // One attach call: the browser MediaStream rides the sources object's // stream key, and the initial state (queued for the first turn) rides // the text source. attach returns a handle — call attached.detach() to // stop sampling when you are done. const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: true }); const attached = await session.attach({ stream, text: [STATE] }); for await (const wave of session.run()) { // code for consuming answers directly as described below }In a browser, do not embed your org API key. Your server keeps the key and answers one endpoint that mints a short-lived client secret; create the client with a function that fetches it:
TypeScript const app = new Sense({ apiKey: async () => (await fetch("/client-secret", { method: "POST" })).json().value, });See
mint_server.tsin the Sense SDK repo for the minting endpoint.The
textsource above sends your initial state — its first item rides the first turn. Strings attached after that are appended as a rolling text context that Sense assesses alongside other media.
Read the answers
Answers are streamed in decision waves. A wave is one cadence tick's complete answer set with one answer per declared question.
-
Read an answer by question name with
wave.result(...), then read its typed fields.Python async for wave in session.run(): route = wave.result("route") # choice — selected option, its probability, and confidence print(route.value, f"p {route.p:.2f}", f"confidence {route.confidence:.2f}") refund = wave.result("wants_refund") # noul — a boolean and the probability the answer is yes print(refund.value, f"p {refund.p:.2f}") urgency = wave.result("urgency") # score — weighted level, top level, its probability, and confidence print(urgency.value, urgency.top, f"p {urgency.p:.2f}", f"confidence {urgency.confidence:.2f}")TypeScript for await (const wave of session.run()) { const route = wave.result("route"); // choice — selected option, its probability, and confidence console.log(route.value, `p ${route.p.toFixed(2)}`, `confidence ${route.confidence.toFixed(2)}`); const refund = wave.result("wants_refund"); // noul — a boolean and the probability the answer is yes console.log(refund.value, `p ${refund.p.toFixed(2)}`); const urgency = wave.result("urgency"); // score — weighted level, top level, its probability, and confidence console.log(urgency.value, urgency.top, `p ${urgency.p.toFixed(2)}`, `confidence ${urgency.confidence.toFixed(2)}`); }Here's an example of one wave's answers, read through the fields above:
Example wave { "route": { "value": "billing", "p": 0.9, "confidence": 0.8 }, "wants_refund": { "value": true, "p": 0.97, "confidence": 0.96 }, "urgency": { "value": 1.45, "top": "Medium — needs attention today", "p": 0.55, "confidence": 0.325 } }
React to decisions
Iterating over waves is one way to consume answers. The other way, enabled by the SDK, is more ergonomic: you can register handlers with the SDK, and let Sense call them when something happens.
- A
whenhandler runs when a condition you define becomes true. - An
onWavehandler runs on every wave.
-
To create a handler that runs when a condition you define returns true, use
when— as a decorator in Python (@app.when(pred)) orapp.when(pred, handler)in TypeScript. Add hysteresis with.between(enter, exit)so that answers near boundaries don't flap. Note thatentermust be greater thanexit.Python # The @app.when decorator registers the function below it as the # handler for the condition. @app.when(wants_refund.p.between(0.8, 0.6)) def escalate(_): # escalate a refund request in a ticketing system ...In this example, the
whencondition is true once the probability of yes reaches0.8and resets to false only if the probability of yes falls below0.6.You can combine logic as and/or/not with
&/|/~. For example:Python @app.when(wants_refund.p.between(0.8, 0.6) & (urgency >= 1)) def escalate_if_urgent(_): # escalate a refund request in a ticketing system ...TypeScript import { type Context } from "@urun-sh/sense"; // Create a function for your logic to trigger function escalate(_event: Context) { // escalate a refund request in a ticketing system } // Define the logic for triggering your function app.when( wantsRefund.p.between(0.8, 0.6), escalate, );In this example, the
whencondition is true once the probability of yes reaches0.8and resets to false only if the probability of yes falls below0.6.You can combine logic as and/or/not with
.and(...)/.or(...)/.not(). For example:TypeScript app.when( wantsRefund.p.between(0.8, 0.6).and(urgency.gte(1)), escalate, ); -
Optional. Handlers receive an event context they can parse and act on.
To extract info from the latest set of answers, use
event.latest():Python def summary(event): wave = event.latest() u = wave.result("urgency") print( f"refund request, urgency {u.value:.2f} (wave {wave.wave_id})" )TypeScript import { type Context } from "@urun-sh/sense"; function summary(event: Context) { const wave = event.latest()!; const u = wave.result("urgency"); console.log( `refund request, urgency ${u.value.toFixed(2)} (wave ${wave.waveId})`, ); }To start tracked background work, use
event.run. You can cancel the background work by name withevent.cancel.Python def draft_response(event): # Start background work tracked as "draft" event.run("draft", lambda ctx: generate_reply(event.latest())) def barge_in(event): # Cancel the background work and the current evaluation turn event.cancel("draft")TypeScript import { type Context } from "@urun-sh/sense"; function draftResponse(event: Context) { // Start background work tracked as "draft" const wave = event.latest()!; event.run("draft", () => generateReply(wave)); } function bargeIn(event: Context) { // Cancel the background work event.cancel("draft"); }These are just a few examples of how you can act on event context. For details, refer to the Sense reference.
-
To create a handler that runs on every wave, use
onWave— as a decorator in Python (@app.on_wave, bare or with options likecoalesce=True) orapp.onWave(handler, { coalesce })in TypeScript. This is useful for logging, metrics, or UI. Settingcoalescecollapses consecutive waves whose decisions didn't change.Python @app.on_wave(coalesce=True) def observe(event): wave = event.latest() print(wave.wave_id, wave.result("route").value)TypeScript app.onWave((event) => { const wave = event.latest()!; console.log(wave.waveId, wave.result("route").value); }, { coalesce: true });
Put it all together
Refer to the Sense SDK repo for a working end-to-end example:
Next steps
Visit the Sense reference for more options and information.
Was this page helpful?