Sense - uRun's System One model
Typed, calibrated decisions from audio, video, and images without translating to text first
Sense is a System One model for voice and video agents and was inspired by the Jev model. It runs on continuously streaming media and answers questions about the input many times per second: for example, whether the person is still speaking or done, whether an interruption is a new question or just an "mmmhmm", and what the person's emotional state is.
Multimedia inputs
Sense can assess the following natively.
- Text
- Audio
- Images
- Video
This means you don't need a speech-to-text model in front of Sense to analyze audio. Sense hears the audio itself. Questions like "has the person finished speaking?" and "is the person interrupting with a new question or voicing agreement?" are answered from the sound.
Every question is answered against the same input, at the same time, up to 12.5 times per second.
Typed questions and answers
Every question has a type (choice, noul, or score), and every answer is a typed value.
- A
choicequestion has named options. For example, was that interruption abackchannelor abarge in? The answer is one of the options you declared. - A
noulquestion is a yes/no probability. For example, does the agent need to retrieve additional information to answer the question? The answer is a number from0to1. Near1is a confident yes, near0is a confident no, near0.5the model is unsure. - A
scorequestion considers a spectrum you describe. For example, how severe is this bug report? The answer is a position on the spectrum you defined, for exampleCosmetic - no impact on functionality,Broken - a workaround exists,Blocking - no workaround for affected users.
Every type of question returns probabilities. This means your code can gate on how sure Sense is.
| Question | Purpose | Answer |
|---|---|---|
choice | Which option fits best? | The selected option, a probability for every option, and a confidence score. |
noul | Is this true? | The probability the answer is yes, from 0 to 1. |
score | Where on the spectrum? | The weighted level (an answer can land between levels), a legend mapping levels back to your descriptions, per-level probabilities, and a confidence score. |
Connection types
You can connect to the Sense Model API through a request for async work or a Session for realtime work.
For async work in a request, uRun exposes an API compatible with TypeSafe Jev:
- TypeSafe Jev
For streaming work in a Session, you use the Sense SDK, which is available in the following languages:
- Python
- TypeScript
The SDK connects to an OpenAI Realtime-compatible API and handles authentication, question registration, and answer assembly for you.
Use cases
Sense was designed for making voice agents feel natural. The model has been trained on a variety of questions for powering realtime turn-taking. You define your own questions for making realtime interactions feel natural or for your business logic.
For example, here are some of the turn-taking questions powering our demos of Sense:
| Question | Type | Instructions | Levels / Options |
|---|---|---|---|
eou | score | Has the current utterance ended (the speaker yielded the turn), as opposed to merely pausing? | no chance, very unlikely, unlikely, uncertain, likely, very likely, certain |
interruption | choice | While the assistant is speaking, what is this incoming audio? | none, backchannel, barge_in |
backchannel_now | choice | Should the assistant emit a backchannel now, and which one? | none, mhm, go_on, right |
emotion | choice | The speaker's dominant emotion? | neutral, curious, frustrated, pleased, urgent |
start_drafting | noul | Should the assistant begin drafting its reply on the utterance so far, before the person has finished speaking? | n/a |
cancel_draft | noul | Should the assistant retract the reply it is drafting (the person kept talking or took the thought back)? | n/a |
Next steps
For a zero-code way to try Sense for realtime work in a Session, check out the Sense playground in the uRun console.
Then, explore the docs for more information
Was this page helpful?