What a sidecar is

A sidecar is the JSON file that the AnalyzeMedia API returns for a video, an audio file, or an image. It sits next to the original and describes what the file is, when and where it was made, and what it contains. The format is open: the API is one writer, and any program that has the original file can write one too. The format defines six types that share one envelope.

In one page

Name The original's full name plus .analyzemedia, for example IMG_1731.MOV.analyzemedia. The sidecar sits next to the original.
Contents One JSON object in UTF-8, with no comments. The extension is always .analyzemedia, not .json.
Bound to the original source.sha256 holds the SHA-256 of the original's bytes. source.quickHash holds a fast fingerprint of the size, the first MiB, and the last MiB. Compare these hashes before you trust a sidecar.
Who writes it The API, or any program that has the original file. The generated block names the writer and the models it used.
Which type The writer declares the type in type: video/footage, video/edit, audio/speech, audio/music, image/scene, or image/graphic. Readers don't second-guess the declared type.
Grades Every field has a grade: measured (a fact of the bytes), observed (a model's description of what is there), derived (recomputed from the rest), or mixed. The type pages state the grade of every field.

Rules you can rely on

Time Times are seconds from the start, as numbers with two decimals. Spans have s and e. Instants have t. The format doesn't use frame numbers or timecodes. An image has no time at any level.
Position Positions are fractions of width and height from the top left: x, y for a point and x, y, w, h for a box. The format doesn't use pixels.
Identity Ids are integers that start at 1, in time order. People are letters (A, B, and so on) in order of first appearance. Songs are also letters.
Order Every array is sorted by start time, by instant, by id, or by reading order in a graphic.
Absent versus empty A missing section means that the writer didn't analyze it. An empty array or null means that the writer analyzed it and found nothing. A file that contains only the envelope is valid.
Measured versus observed The grade is a property of the field, and the type pages state it for every field. No field holds an opinion about quality, importance, or intent.
Types are declared A file has the type that it declares. The file doesn't explain the choice, and the format defines no test to tell types apart.
Vocabularies Each type page lists its closed vocabularies. Each vocabulary includes other. Read an unknown value as other.
Unknown fields Unknown fields are allowed at any level. Tools that rewrite the file preserve them.

The six types

type	for example	what it describes
`video/footage`	`IMG_1731.MOV.analyzemedia`	One continuous take, as recorded: what was in front of the lens and what the camera did.
`video/edit`	`launch-final.mp4.analyzemedia`	A finished piece and how it is built: scenes, cuts, what is on the frame, and the sound.
`audio/speech`	`New Recording 14.m4a.analyzemedia`	People talking: the transcript, the voices, the silences, and the levels.
`audio/music`	`03 Low Tide.flac.analyzemedia`	A track or a mix: which recording it is, which part plays, tempo, key, beats, and energy.
`image/scene`	`DSC_0412.HEIC.analyzemedia`	A view of something: form, composition, and people and objects with boxes. It has no time.
`image/graphic`	`checkout-flow.png.analyzemedia`	Something made of marks: a screen, a document, or a chart. Its parts have boxes. It has no viewpoint.

Where to go next

The format The envelope, source by medium, the grades, and which sections each type carries.
Writing a sidecar What a writer must do and what it can leave out.
Reading a sidecar How to match a sidecar to its file, what you can derive from it, and how to render it for a model.
Schemas One JSON schema per type, to validate against.
How the result maps to the format What the API returns, section by section.
Route reference Every route of the API.

Version 0.1 is the first version, and it isn't released yet. Until the release, anything can change. After a major version is released, later versions within it only add fields.