docs

Use Sense for realtime work

Connect to the Sense Model API in a Session to assess inputs continuously

This page walks through using the Sense SDK to run the Sense Model API in a Session for realtime work. You will do the following:

  1. Get a uRun API key.
  2. Prepare the Sense SDK.
  3. Declare a standing question set, a cadence for assessing the questions, and an initial state.
  4. Prepare audio, video, or text inputs to assess.
  5. Open a Session and start streaming.
  6. Read the answers.
  7. React to decisions.

Sense is a decision model: you configure typed questions and streaming media, and Sense returns typed answers with probabilities and confidence. If you are new to the model, read the Sense overview first.

The Sense SDK registers your questions, connects and authenticates your Sessions, and assembles streamed answers into waves for your application to act on. The Sense SDK is available in Python and TypeScript.

This page walks through your first Session with the Sense SDK. For more information, refer to the Sense reference.

Prerequisites

Get an API key

Create an API key:

  1. In the console, go to Settings → API Keys and click + Create API key.
  2. Enter a Label for the key that will help you identify where it is used later.
  3. Select Create key.
  4. Copy the generated secret and store it somewhere safe. It is not retrievable later.

Prepare the Sense SDK

  1. Install the SDK in your language of choice.

    terminal
    pip install urun-sense
    terminal
    npm install @urun-sh/sense
  2. Set the URUN_API_KEY environment variable for authentication:

    terminal
    export URUN_API_KEY="urun_sk_YOUR_KEY"

Declare questions

In a Session, Sense continuously evaluates a standing question set on a cadence measured in hertz (default and maximum 12.5). You declare the questions on a Sense client, and the client registers them when a Session opens.

  1. Create the client. Sense evaluates your standing question set against the streaming inputs on a cadence measured in hertz; the default is 12.5, which is also the maximum — you cannot configure a faster cadence.

    Python
    from sense import Sense
    
    app = Sense()
    TypeScript
    import { Sense } from "@urun-sh/sense";
    
    const app = new Sense();
  2. Declare your standing set of questions. This example demonstrates all three types of questions: choice, noul, and score.

    Python
    route = app.choice(
        "route",
        instructions="Which team should handle this message?",
        criteria={"billing": "Charges, invoices, refunds", "technical": "Bugs, errors, setup"},
    )
    wants_refund = app.noul(
        "wants_refund",
        instructions="Is the person asking for a refund?",
        criteria={
            "true": "the person is asking for their money back",
            "false": "the person is not asking for a refund",
        },
    )
    urgency = app.score(
        "urgency",
        instructions="How urgent is this message?",
        criteria=[
            "Low — routine, no deadline",
            "Medium — needs attention today",
            "High — blocking, act now",
        ],
    )
    TypeScript
    const route = app.choice("route", {
      instructions: "Which team should handle this message?",
      criteria: { billing: "Charges, invoices, refunds", technical: "Bugs, errors, setup" },
    });
    const wantsRefund = app.noul("wants_refund", {
      instructions: "Is the person asking for a refund?",
      criteria: {
        true: "the person is asking for their money back",
        false: "the person is not asking for a refund",
      },
    });
    const urgency = app.score("urgency", {
      instructions: "How urgent is this message?",
      criteria: [
        "Low — routine, no deadline",
        "Medium — needs attention today",
        "High — blocking, act now",
      ],
    });
  3. Define an initial text state that provides context about your streaming inputs. Use it for information that is not in the streaming media itself, such as who is speaking. You will push it as the Session's first text input after you open the Session.

    Python
    STATE = "The person is on a live support call"
    TypeScript
    const STATE = "The person is on a live support call";

Prepare streaming inputs

Sense assesses three input modalities, each in a fixed format:

  • Audio: raw PCM16, mono, at 24 kHz, in small chunks (we recommend approximately 40 ms).
  • Video: a JPEG frame of at most 256 KB. Sense assesses the most recent frame; a frame persists until a new one replaces it.
  • Text: plain strings.

session.attach(...) is the one input path: call it before session.run() (which takes no input arguments) and hand it the source — the SDK captures, converts, and pushes chunks in the right format for you. Both languages take one sources object. In TypeScript, a browser MediaStream rides the stream key — the mic is resampled to 24 kHz PCM16 and framed into ~40 ms chunks, and camera frames are sampled to JPEG at the Session cadence. In Python, attach takes audio=, video=, and text= sources. Each source is an iterable or a zero-arg callable that yields the next chunk (null/None ends it).

Open a Session and start streaming

  1. Open a Session.

    Python
    session = app.session()
    TypeScript
    const session = app.session();

    The SDK connects to the Sense Model API when you attach or run — session.attach(...) starts the Session, dialing in the background — authenticates with your URUN_API_KEY, registers the questions you declared earlier, and handles the Session lifecycle.

  2. Start streaming. Push your initial text state, attach your live source with session.attach(...), then run the Session with run(), which yields answers as questions are assessed.

    Python
    # Attach the initial state (the first item of the text source rides the
    # first turn) and the live source; the SDK captures, converts, and
    # pushes chunks in the right format. attach returns a handle — call
    # attached.detach() to stop sampling when you are done.
    attached = await session.attach(audio=mic_chunks, text=[STATE])
    
    async for wave in session.run():
        # code for consuming answers directly as described below
        ...
    TypeScript
    // One attach call: the browser MediaStream rides the sources object's
    // stream key, and the initial state (queued for the first turn) rides
    // the text source. attach returns a handle — call attached.detach() to
    // stop sampling when you are done.
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: true });
    const attached = await session.attach({ stream, text: [STATE] });
    
    for await (const wave of session.run()) {
      // code for consuming answers directly as described below
    }

    In a browser, do not embed your org API key. Your server keeps the key and answers one endpoint that mints a short-lived client secret; create the client with a function that fetches it:

    TypeScript
    const app = new Sense({
      apiKey: async () => (await fetch("/client-secret", { method: "POST" })).json().value,
    });

    See mint_server.ts in the Sense SDK repo for the minting endpoint.

    The text source above sends your initial state — its first item rides the first turn. Strings attached after that are appended as a rolling text context that Sense assesses alongside other media.

Read the answers

Answers are streamed in decision waves. A wave is one cadence tick's complete answer set with one answer per declared question.

  1. Read an answer by question name with wave.result(...), then read its typed fields.

    Python
    async for wave in session.run():
        route = wave.result("route") # choice — selected option, its probability, and confidence
        print(route.value, f"p {route.p:.2f}", f"confidence {route.confidence:.2f}")
    
        refund = wave.result("wants_refund") # noul — a boolean and the probability the answer is yes
        print(refund.value, f"p {refund.p:.2f}")
    
        urgency = wave.result("urgency") # score — weighted level, top level, its probability, and confidence
        print(urgency.value, urgency.top, f"p {urgency.p:.2f}", f"confidence {urgency.confidence:.2f}")
    TypeScript
    for await (const wave of session.run()) {
      const route = wave.result("route"); // choice — selected option, its probability, and confidence
      console.log(route.value, `p ${route.p.toFixed(2)}`, `confidence ${route.confidence.toFixed(2)}`);
    
      const refund = wave.result("wants_refund"); // noul — a boolean and the probability the answer is yes
      console.log(refund.value, `p ${refund.p.toFixed(2)}`);
    
      const urgency = wave.result("urgency"); // score — weighted level, top level, its probability, and confidence
      console.log(urgency.value, urgency.top, `p ${urgency.p.toFixed(2)}`, `confidence ${urgency.confidence.toFixed(2)}`);
    }

    Here's an example of one wave's answers, read through the fields above:

    Example wave
    {
      "route": { "value": "billing", "p": 0.9, "confidence": 0.8 },
      "wants_refund": { "value": true, "p": 0.97, "confidence": 0.96 },
      "urgency": { "value": 1.45, "top": "Medium — needs attention today", "p": 0.55, "confidence": 0.325 }
    }

React to decisions

Iterating over waves is one way to consume answers. The other way, enabled by the SDK, is more ergonomic: you can register handlers with the SDK, and let Sense call them when something happens.

  • A when handler runs when a condition you define becomes true.
  • An onWave handler runs on every wave.
  1. To create a handler that runs when a condition you define returns true, use when — as a decorator in Python (@app.when(pred)) or app.when(pred, handler) in TypeScript. Add hysteresis with .between(enter, exit) so that answers near boundaries don't flap. Note that enter must be greater than exit.

    Python
    # The @app.when decorator registers the function below it as the
    # handler for the condition.
    @app.when(wants_refund.p.between(0.8, 0.6))
    def escalate(_):
        # escalate a refund request in a ticketing system
        ...

    In this example, the when condition is true once the probability of yes reaches 0.8 and resets to false only if the probability of yes falls below 0.6.

    You can combine logic as and/or/not with &/|/~. For example:

    Python
    @app.when(wants_refund.p.between(0.8, 0.6) & (urgency >= 1))
    def escalate_if_urgent(_):
        # escalate a refund request in a ticketing system
        ...
    TypeScript
    import { type Context } from "@urun-sh/sense";
    
    // Create a function for your logic to trigger
    function escalate(_event: Context) {
      // escalate a refund request in a ticketing system
    }
    
    // Define the logic for triggering your function
    app.when(
      wantsRefund.p.between(0.8, 0.6),
      escalate,
    );

    In this example, the when condition is true once the probability of yes reaches 0.8 and resets to false only if the probability of yes falls below 0.6.

    You can combine logic as and/or/not with .and(...)/.or(...)/.not(). For example:

    TypeScript
    app.when(
      wantsRefund.p.between(0.8, 0.6).and(urgency.gte(1)),
      escalate,
    );
  2. Optional. Handlers receive an event context they can parse and act on.

    To extract info from the latest set of answers, use event.latest():

    Python
    def summary(event):
        wave = event.latest()
        u = wave.result("urgency")
        print(
            f"refund request, urgency {u.value:.2f} (wave {wave.wave_id})"
        )
    TypeScript
    import { type Context } from "@urun-sh/sense";
    
    function summary(event: Context) {
      const wave = event.latest()!;
      const u = wave.result("urgency");
      console.log(
        `refund request, urgency ${u.value.toFixed(2)} (wave ${wave.waveId})`,
      );
    }

    To start tracked background work, use event.run. You can cancel the background work by name with event.cancel.

    Python
    def draft_response(event):
        # Start background work tracked as "draft"
        event.run("draft", lambda ctx: generate_reply(event.latest()))
    
    def barge_in(event):
        # Cancel the background work and the current evaluation turn
        event.cancel("draft")
    TypeScript
    import { type Context } from "@urun-sh/sense";
    
    function draftResponse(event: Context) {
      // Start background work tracked as "draft"
      const wave = event.latest()!;
      event.run("draft", () => generateReply(wave));
    }
    
    function bargeIn(event: Context) {
      // Cancel the background work
      event.cancel("draft");
    }

    These are just a few examples of how you can act on event context. For details, refer to the Sense reference.

  3. To create a handler that runs on every wave, use onWave — as a decorator in Python (@app.on_wave, bare or with options like coalesce=True) or app.onWave(handler, { coalesce }) in TypeScript. This is useful for logging, metrics, or UI. Setting coalesce collapses consecutive waves whose decisions didn't change.

    Python
    @app.on_wave(coalesce=True)
    def observe(event):
        wave = event.latest()
        print(wave.wave_id, wave.result("route").value)
    TypeScript
    app.onWave((event) => {
      const wave = event.latest()!;
      console.log(wave.waveId, wave.result("route").value);
    }, { coalesce: true });

Put it all together

Refer to the Sense SDK repo for a working end-to-end example:

Next steps

Visit the Sense reference for more options and information.

Was this page helpful?

0/2000

On this page