docs

Sense - uRun's System One model

Typed, calibrated decisions from audio, video, and images without translating to text first

Sense is a System One model for voice and video agents and was inspired by the Jev model. It runs on continuously streaming media and answers questions about the input many times per second: for example, whether the person is still speaking or done, whether an interruption is a new question or just an "mmmhmm", and what the person's emotional state is.

Multimedia inputs

Sense can assess the following natively.

  • Text
  • Audio
  • Images
  • Video

This means you don't need a speech-to-text model in front of Sense to analyze audio. Sense hears the audio itself. Questions like "has the person finished speaking?" and "is the person interrupting with a new question or voicing agreement?" are answered from the sound.

Every question is answered against the same input, at the same time, up to 12.5 times per second.

Typed questions and answers

Every question has a type (choice, noul, or score), and every answer is a typed value.

  • A choice question has named options. For example, was that interruption a backchannel or a barge in? The answer is one of the options you declared.
  • A noul question is a yes/no probability. For example, does the agent need to retrieve additional information to answer the question? The answer is a number from 0 to 1. Near 1 is a confident yes, near 0 is a confident no, near 0.5 the model is unsure.
  • A score question considers a spectrum you describe. For example, how severe is this bug report? The answer is a position on the spectrum you defined, for example Cosmetic - no impact on functionality, Broken - a workaround exists, Blocking - no workaround for affected users.

Every type of question returns probabilities. This means your code can gate on how sure Sense is.

QuestionPurposeAnswer
choiceWhich option fits best?The selected option, a probability for every option, and a confidence score.
noulIs this true?The probability the answer is yes, from 0 to 1.
scoreWhere on the spectrum?The weighted level (an answer can land between levels), a legend mapping levels back to your descriptions, per-level probabilities, and a confidence score.

Connection types

You can connect to the Sense Model API through a request for async work or a Session for realtime work.

For async work in a request, uRun exposes an API compatible with TypeSafe Jev:

  • TypeSafe Jev

For streaming work in a Session, you use the Sense SDK, which is available in the following languages:

  • Python
  • TypeScript

The SDK connects to an OpenAI Realtime-compatible API and handles authentication, question registration, and answer assembly for you.

Use cases

Sense was designed for making voice agents feel natural. The model has been trained on a variety of questions for powering realtime turn-taking. You define your own questions for making realtime interactions feel natural or for your business logic.

For example, here are some of the turn-taking questions powering our demos of Sense:

QuestionTypeInstructionsLevels / Options
eouscoreHas the current utterance ended (the speaker yielded the turn), as opposed to merely pausing?no chance, very unlikely, unlikely, uncertain, likely, very likely, certain
interruptionchoiceWhile the assistant is speaking, what is this incoming audio?none, backchannel, barge_in
backchannel_nowchoiceShould the assistant emit a backchannel now, and which one?none, mhm, go_on, right
emotionchoiceThe speaker's dominant emotion?neutral, curious, frustrated, pleased, urgent
start_draftingnoulShould the assistant begin drafting its reply on the utterance so far, before the person has finished speaking?n/a
cancel_draftnoulShould the assistant retract the reply it is drafting (the person kept talking or took the thought back)?n/a

Next steps

For a zero-code way to try Sense for realtime work in a Session, check out the Sense playground in the uRun console.

Then, explore the docs for more information

Was this page helpful?

0/2000

On this page