audio/speech

A speech file describes people talking: a memo, an interview, a call, or a lecture. Use it for any recording where the words are the point. The writer measures almost everything in the file. The model only writes the summary and describes the voices.

envelope the same fields in every typemeasured computed from the file's bytes; the same value every timeobserved what a model saw or heard; never a recommendationmixed some parts measured and some observed; the description says whichderived computed from other fields; the writer regenerates it and never edits itinstant · t one point in timespan · s, e a start time and an end timespans inside applies to the whole recording and contains a list of timed itemst, or t and e an instant, or a span if it has an end timeNo mark: the field has no time of its own and applies to the whole recording.

New Recording 14.m4a.analyzemediaone JSON object next to the original

From top to bottom: the envelope, a paragraph about the conversation, the levels, the voices, the words in four views, and a flat timeline.

The envelopealways present, in this order

format

"analyzemedia"

Always this string.

version

"0.1"

Within a major version, changes only add fields.

type

"audio/speech"

The writer declares the type. Readers don't second-guess it.

generatedmeasured

{ at, by, models }

When the writer made the file, which writer, and which models it used. Nothing per item.

sourcethe file, plus its tags and capture time when it carries them

name, bytes, sha256, quickHashmeasured

The hashes bind the sidecar to the file. A renamed file still matches. A different file doesn't.

container, duration, audiomeasured

audio { codec, channels, sampleRate, bitRate }

What the container reports.

tagsmeasured

{ title, artist, album, albumArtist, track, disc, year, genre, composer, comment, artwork }

The container's own tags, present only when the file has them. artwork holds { mime, bytes, width, height }, not the image bytes.

recordedmeasured

{ utc, local, offset, zoneSource }

Present when the file carries a capture time. Audio has no location, so there is no time of day.

summaryobserved

string

One paragraph about what the conversation covers and what happens in it.

signalwhole-file levels; every value is a fact of the samples

loudnessmeasured

{ integrated, truePeak, range }

LUFS, dBTP, and LU, as broadcast meters report them.

noiseFloor · dynamicRangemeasured

dBFS · dB

noiseFloor is the level of the quietest sustained stretch. dynamicRange is the peak minus the noise floor.

clipping[]measuredspan · s, e

[ { s, e } ]

Spans at full scale. The list is empty when the recording is clean.

waveformmeasuredspans inside

{ step, peaks [] }

The peak level per step seconds, as fractions of full scale. This is enough to draw an overview without decoding the file.

people[]observed

[ { label, description } ]

The voices, lettered in order of first speaking. description holds what the sound gives: a name if someone says one, a role if someone states one, and otherwise the voice itself in a few words.

speechalways present in a speech file; four views of the same words

Use paragraphs for reading, words for alignment, fillers and silences for editing, and speakers to know who said what.

languagemeasured

BCP 47

For example, en or es-419. One language per file.

speakers[]mixed

[ { id, person } ]

id is the voice's index in the words, and it is measured. person is the letter of the voice in people[]. Every voice is heard, and none is seen.

paragraphs[]measuredspan · s, e

[ { s, e, speaker, sentences [ { s, e, text } ] } ]

The reading view. Each paragraph has one speaker and holds punctuated sentences with their own spans.

words[]measuredspan · s, e

[ { w, s, e, confidence, speaker } ]

Every word with its span, with punctuation attached and a confidence from 0 to 1. This is the alignment view.

fillers[]measuredinstant · t

[ { w, t } ]

The ums and uhs. They also appear in the words, and this list repeats them so that an editor can cut them.

silences[]measuredspan · s, e

[ { s, e } ]

Spans below the noise floor that last 0.5 s or longer.

loudnessmeasured

{ integrated, truePeak, range }

The loudness of the track. These are the same numbers as signal.loudness.

A speech file never has a music section. If a jingle plays, the words over it are still speech, and the file doesn't describe the jingle. signal.waveform shows where the sound is loud, but nothing says what the sound is.

events[]derivedt, or t and e

[ { t, e, kind, text } ]

Paragraphs as speech, silences as silence, and pauses as pause. A writer regenerates this list and never edits it.

A seven-minute memo on the timeline

This is the example from the tables: two people planning a kitchen. The waveform row draws signal.waveform.peaks as bars. The paragraphs alternate between the two voices. The silences fall where the waveform goes quiet.

Everything on this timeline is measured. The only observed fields in a speech file are summary, the description of each voice, and which voice belongs to which person.

A speech file answers the question "who said what, when, and how clean is the recording". A model reads summary and the paragraphs. An editor cuts on the fillers, the silences, and the waveform. A speech file never holds music, even if a jingle plays.

Open questions

loudness twice signal.loudness and speech.loudness hold the same three numbers. One of them can go, and signal is the natural home.
words[] and paragraphs[] Two views of the same words is by design, but the words are the bulk of the file. If the readers are language models, paragraphs with sentence spans might be enough. The words could then stay in the caption-matching pipeline rather than in the document.
Nothing else Every other field already holds one measured fact.

Rules you can rely on

Time Times are seconds from the start, with two decimals. Spans have s and e. Instants have t. The format doesn't use timecodes.
Identity Ids start at 1 in time order. People and songs are letters in order of first appearance.
Absent versus empty A missing section means that the writer didn't analyze it. An empty section means that the writer analyzed it and found nothing.
Measured versus observed The grade is a property of the field, stated for every field. No field holds an opinion about quality, importance, or intent.
Vocabularies Each vocabulary is a closed list that includes other. Read an unknown value as other.
Types are declared A file has the type that it declares. The format defines no test to tell types apart.