The format

This page maps the format: the five fields that every file starts with, the source block by medium, the grades, and the sections that each type carries. The type pages define every field.

The envelope

Every file starts with these five fields, in this order. Everything after them belongs to the type.

field	shape	meaning
`format`	`"analyzemedia"`	Always this string.
`version`	`"0.1"`	The major and minor version. Within a released major version, changes only add fields.
`type`	`video/footage · video/edit · audio/speech · audio/music · image/scene · image/graphic`	Which of the six types this file is.
`generated`	`{ at, by, models }`	When the writer made the file, which writer, and which models it used (a map of role to model ID). Nothing per item. If the analysis used a proxy, `generated.proxy` says so, and `source` still describes the original.
`source`	`{ name, bytes, sha256, quickHash, container, duration, video, audio, image, tags, camera, exposure, recorded, location }`	The file and the recording, by medium. Every field is measured except `location.venue`, which is a model's choice of the place that the picture shows.

`source`, by medium

field	shape	when
`name, bytes, sha256, quickHash`	`string, integer, hex, hex`	Every medium.
`container, duration`	`string, seconds`	Video and audio.
`video`	`{ codec, width, height, fps, rotation, hdr, colorTransfer, colorPrimaries, pixelFormat, bitRate }`	Video only.
`audio`	`{ codec, channels, sampleRate, bitRate } \| null`	Video and audio. The value is null for a silent video.
`image`	`{ format, width, height, orientation, colorSpace, bitDepth, hdr, alpha }`	Image only. Width and height apply after orientation.
`tags`	`{ title, artist, album, albumArtist, track, disc, year, genre, composer, comment, artwork }`	Audio, from the container's own tags, when the file has them.
`camera`	`{ make, model, lens, software }`	Video and image, when the file names a camera.
`exposure`	`{ iso, shutter, aperture, focalLength, focalLength35, exposureBias, flash, whiteBalance }`	Image, when the file carries exposure data.
`recorded`	`{ utc, local, offset, zoneSource, timeOfDay { bucket, sunrise, sunset, goldenHourStart, minutesToSunset, sunUp } }`	When the file carries a capture time. The time of day requires a location.
`location`	`{ lat, lon, altitude, accuracy, address { … }, nearby [ { name, type, distance } ], venue { name, reason } \| null }`	When the file carries a position.

Grades

Every field on every type page has one of four grades. A field that has a time also states its time shape: an instant, a span, or a list of spans inside a whole-file block.

Measured A fact of the file's bytes. The value is the same every time.
Observed A model's description of the picture or the sound. The model describes what is there, not what to do with it.
Derived Recomputed from the rest of the file. A writer regenerates a derived field and never edits it.
Mixed The field has measured and observed parts. The type page states which parts are which.

Sections by type

A filled circle means that the section is present when the writer analyzes it. A blank cell means that the type never has the section.

section	footage	edit	speech	music	scene	graphic	what it holds
always
`format, version, type, generated, source`	●	●	●	●	●	●	The envelope, the same in every type.
`summary`	●	●	●		●	●	The whole piece in one paragraph.
`events[]`	●	●	●	●			One time-ordered list of everything, derived. Images have no time.
the picture
`camera, picture, form`	●				●		What the camera did, the color and tone, and the shot as a cinematographer names it. Footage and scene only. A scene has no camera motion.
`scenes[]`		●					An edit's only structural unit. Each scene is one situation, with its cuts, layout, and speed.
`captions, overlays[]`		●					What the editor put on the picture.
`kind`					●	●	What sort of image this is: a photograph or a painting, a screen or a chart.
`composition`					●		How a still image is built: a paragraph, the devices, and the subject.
`parts[]`						●	A graphic's parts in reading order, with boxes.
who and what
`people[]`	●	●	●		●		The subjects. In speech audio, the voices.
`objects[], text[]`	●				●	●	Things worth naming, and the text that appears in the world.
`actions[], moments[], environment[]`	●						What happens in a take, where it changes, and what comes and goes around it.
`surroundings`					●		Everything that is not the subject, in a paragraph.
sound
`speech`	●	●	●				The transcript: words, sentences, paragraphs, speakers, and silences.
`music`	●	●		●			Which recording plays, which part of it, and the tempo, key, and beats.
`signal`			●	●			Whole-file levels: loudness, noise floor, clipping, and a waveform overview.
`sfx[]`		●					An edit's sound effects.

Rules you can rely on

Time Times are seconds from the start, as numbers with two decimals. Spans have s and e. Instants have t. The format doesn't use frame numbers or timecodes. An image has no time at any level.
Position Positions are fractions of width and height from the top left: x, y for a point and x, y, w, h for a box. The format doesn't use pixels.
Identity Ids are integers that start at 1, in time order. People are letters (A, B, and so on) in order of first appearance. Songs are also letters.
Order Every array is sorted by start time, by instant, by id, or by reading order in a graphic.
Absent versus empty A missing section means that the writer didn't analyze it. An empty array or null means that the writer analyzed it and found nothing. A file that contains only the envelope is valid.
Measured versus observed The grade is a property of the field, and the type pages state it for every field. No field holds an opinion about quality, importance, or intent.
Types are declared A file has the type that it declares. The file doesn't explain the choice, and the format defines no test to tell types apart.
Vocabularies Each type page lists its closed vocabularies. Each vocabulary includes other. Read an unknown value as other.
Unknown fields Unknown fields are allowed at any level. Tools that rewrite the file preserve them.