docs

Sense reference

Reference for Sense, uRun's perception model

This page is the reference for the Sense model. For what Sense is and why you would use it, see the Sense overview.

You can connect to the Sense Model API through a request for async work or a Session for realtime work. How you call the model depends on your connection type.

  • In a request, you send one JSON body with the content and questions to assess, and get back one JSON object with the answers.
  • In a Session, you use the Sense SDK: you declare a standing set of questions, a cadence, and an initial state, then stream multimedia inputs which Sense evaluates continuously. The SDK yields the answers as waves — one complete answer set per evaluation tick.
Select your connection type

Connection

To use the Sense Model API in a request/response pattern for batch or async work, call it with your API key through the URL uRun exposes for TypeSafe Jev:

  • TypeSafe Jev - https://inference.urun.sh/v1/systemone — takes typed state, questions, and optional media in; emits answers as a JSON object.

Inputs

FieldRequiredTypeDescription
stateyesstring or JSONThe content to evaluate.
medianoarrayAudio, image, and video items attached to the state.
questionsyesobjectWhat you want to assess, keyed by the question name you choose.
  • The state is the text your questions judge. It can be a string, a JSON object, or a JSON array.

    {
      "state": "My card was charged twice for the same order.", 
      "questions": {
        "refund_requested": {
          "type": "noul",
          "instructions": "Is the person asking for a refund?"
        }
      }
    }
    {
      "state": { 
        "ticket": "I was charged twice for order A-104.", 
        "order": { "id": "A-104", "amount_usd": 49 }, 
        "refund_policy": "Duplicate charges are eligible for a refund."
      }, 
      "questions": {
        "refund_requested": {
          "type": "noul",
          "instructions": "Does `ticket` request a refund, and does `refund_policy` allow one for `order`?"
        }
      }
    }
    {
      "state": [ 
        { "from": "person", "text": "I was charged twice for order A-104." }, 
        { "from": "support", "text": "We're checking the charges now." } 
      ], 
      "questions": {
        "refund_requested": {
          "type": "noul",
          "instructions": "Does `[0].text` request a refund?"
        }
      }
    }
  • Attach audio, images, and video with media, which takes an array of up to 32 items totaling up to 20 MB combined when decoded.

    Each item in the array is an object with the following properties:

    PropertyRequiredTypeDescription
    typeyesstringaudio, image, or video.
    datayesstringBase64-encoded bytes, or a data URL (data:<mime>;base64,<bytes>).
    mimeonly when the payload doesn't identify itselfstringaudio/wav, audio/x-wav, audio/wave, audio/mpeg, audio/flac, audio/ogg, image/png, image/jpeg, image/webp, video/mp4, video/webm, or video/quicktime. Omit when data is a data URL (its mime is used) or the bytes are a recognized container (sniffed).
    start_s, end_snonumberWindow the media — for example the last 8 seconds of a longer clip.
    {
      "state": "A recording of a support call, with a screenshot of the order page.",
      "media": [ 
        { "type": "audio", "data": "<base64-encoded audio>", "mime": "audio/wav" }, 
        { "type": "image", "data": "<base64-encoded image>", "mime": "image/png" }, 
        { "type": "video", "data": "<base64-encoded video>", "mime": "video/mp4" } 
      ], 
      "questions": {
        "refund_requested": {
          "type": "noul",
          "instructions": "Is the person asking for a refund?"
        }
      }
    }

    state is still required when you send media. Use it for context that is not in the media itself, such as who is speaking.

Question types

A question can be a choice, score, or noul. Questions are defined as objects and different types take different properties.

PropertyApplies toRequiredDescription
typeAll typesNo — defaults to choiceThe question kind: choice, noul, or score.
instructionsAll typesYesThe question itself, in plain words. Describe the judgment to make.
criteriachoice, scoreYes — optional for noulThe answers to choose among. For noul, optional { "true": "...", "false": "..." } descriptions.
  • A choice question asks "which option fits best?".

    criteria is a map of up to 255 option IDs to descriptions.

    "questions": {
      "route": {
        "type": "choice", 
        "instructions": "Which team should handle this message?", 
        "criteria": { 
          "billing": "Payment or refund issues", 
          "technical": "Bugs or integration problems", 
          "sales": "Pricing or account questions"
        }
      }
    }
  • A noul question asks "is this true?"

    criteria is optional and describes what true and false mean. Give every option a description:

    "questions": {
      "finished_speaking": {
        "type": "noul", 
        "instructions": "Has the person finished their turn?", 
        "criteria": { 
          "true": "the person yielded the turn", 
          "false": "the person is only pausing"
        }
      }
    }
  • A score question asks "where does this fall on the defined spectrum?"

    criteria is an ordered array of 2 to 10 level descriptions, low end first. Each entry is a plain description string; the level's index is its id.

    "questions": {
      "tone": {
        "type": "score", 
        "instructions": "How does the person speaking sound right now?",
        "criteria": [ 
          "well structured thoughts",
          "pausing a lot",
          "repeating themselves",
          "using strong language"
        ]
      }
    }

    Describe situations in your levels, not degrees. Defining "angry" as "using strong language" is more specific and therefore better than defining "angry" as "very upset".

Outputs

Different question types return different fields.

Question typeReturned fields
choicechoice (the selected option), probabilities (per option), confidence
noulnoul (probability of yes)
scorescore (weighted level), legend, probabilities (per level), confidence

Option probabilities are softmax-normalized across the question's options (or levels) and sum to 1 up to floating-point rounding.

Each request with questions returns one JSON object with answers for all the questions. The response also carries usage, the token counts for the call.

"answers": {
  "route": {
    "type": "choice",
    "choice": "billing",
    "probabilities": { "billing": 0.9, "technical": 0.1 },
    "confidence": 0.8
  },
  "wants_refund": {
    "type": "noul",
    "noul": 0.97
  },
  "urgency": {
    "type": "score",
    "score": 0.85,
    "legend": {
      "0": "Low",
      "1": "Medium",
      "2": "High"
    },
    "probabilities": { "0": 0.35, "1": 0.45, "2": 0.2 },
    "confidence": 0.175
  }
}

Confidence

choice and score answers return a confidence from 0 to 1.

  • For choice, confidence measures how concentrated the answer's probability mass is:
    • 1 means all probability is on one option.
    • 0 means the probabilities are spread evenly across all options.
  • For score, confidence measures how tightly the probability mass clusters around the most-likely level, counting nearer levels as less of a spread than distant ones:
    • 1 means all probability is on one level.
    • 0 means the probabilities are spread evenly across all levels or split between far-apart levels.

In both cases, confidence is a property of the distribution you got back — not a probability that the answer is correct.

Use confidence in your application code to gate how much you trust a decision:

  • High — act on it automatically.
  • Middle — confirm with a person or gather more state before acting.
  • Low — do not act on it.

Limits

Calls that exceed a limit are rejected with an error that notes the limit.

Field(s)Limit
state plus all questions, combined524,288 characters
Options per choice question2 to 255
Levels per score question2 to 10
media items32
media bytes, decoded20 MB
Context window32,768 tokens

Complete example

Encode media files and define your questions

terminal
AUDIO_B64=$(base64 < clip.wav | tr -d '\n')

cat > request.json <<EOF
{
  "state": "Audio of a person speaking.",
  "media": [
    { "type": "audio", "data": "$AUDIO_B64", "mime": "audio/wav" }
  ],
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Payments, refunds",
        "technical": "Bugs, outages",
        "sales": "Pricing, upgrades"
      }
    },
    "refund_warranted": {
      "type": "noul",
      "instructions": "Does the duplicate charge qualify for a refund?",
      "criteria": {
        "true": "the duplicate charge qualifies for a refund under the policy",
        "false": "it does not qualify"
      }
    },
    "urgency": {
      "type": "score",
      "instructions": "How urgent is this request?",
      "criteria": [
        "can wait",
        "should be handled today",
        "the person is blocked"
      ]
    }
  }
}
EOF

Send the content and questions to the Sense Model API.

terminal
curl https://inference.urun.sh/v1/systemone \
  -H "Authorization: Bearer urun_sk_YOUR_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

The answer:

response
{
  "model": "sense-latest",
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {
        "billing": 0.991048276424408,
        "technical": 0.005195490550249815,
        "sales": 0.0037562334910035133
      },
      "confidence": 0.9865724139443727
    },
    "refund_warranted": {
      "type": "noul",
      "noul": 0.981
    },
    "urgency": {
      "type": "score",
      "score": 1.5,
      "legend": { "0": "can wait", "1": "should be handled today", "2": "the person is blocked" },
      "probabilities": { "0": 0.1, "1": 0.3, "2": 0.6 },
      "confidence": 0.52
    }
  }
}

Was this page helpful?

0/2000

On this page