The six types

Every sidecar starts with the same fields: the type, the writer and its models, and the facts about the original file. The fields after that depend on the type. You declare the type when you start an analysis, and the API never infers it from the content. Two types cover video, two cover audio, and two cover images.

The content doesn't override the declared type. A photo of a screen is a scene, a screenshot of a photo is a graphic, and a jingle under a voice is speech. If you omit the type, the API uses video/edit for video files, audio/speech for audio files, and image/scene for images.

video/footage

One continuous take as recorded: a phone clip, a take from a shoot, or a screen recording. The file includes:

• camera movement, measured: support, pans, tilts, and zooms, each with its time
• color and tone, and the shot size, angle, and lighting
• the people, labeled A, B, C in order of appearance
• the actions with their time spans, the moments where they change, and the environment around the subjects
• the speech, with speakers and times
• the music
• the recording time and location, if the file contains them

Every field of video/footage

video/edit

An edited video: a short, an ad, a talk, or a trailer. Describes how the video is built, not how its footage was shot. The file includes:

• the scenes, one situation each, with the cuts inside them
• the layout of each scene: full frame, split screen, or picture in picture
• speed changes, freeze frames, and slow motion
• the captions and their style, and the overlays with their bounding boxes
• the pacing: the number of cuts and the average shot length
• the people, the speech, the music, and the sound effects

Every field of video/edit

audio/speech

A recording of people talking: a voice memo, an interview, a podcast, or a meeting. The file includes:

• the signal: loudness, noise floor, dynamic range, and clipping
• the silences
• every word with its time, grouped into sentences and paragraphs
• the speakers, with a description of each voice
• a summary of what was said

Every field of audio/speech

audio/music

A track, a mix, or a live recording of music. The file includes:

• the signal, as for speech
• the identified recording, when recognition finds a match
• each piece in the file with its time span, tempo, key, and beats
• the energy over time

Every field of audio/music

image/scene

A photograph, a video frame, or a painting. An image has no time. The file includes:

• the camera, lens, exposure, capture time, and location, from the file's metadata
• color, tone, and sharpness, measured
• the kind of image and its composition
• the people, with a bounding box, gaze direction, and expression for each
• the objects and any visible text, with bounding boxes
• a paragraph about the surroundings

Every field of image/scene

image/graphic

An image made of marks rather than captured by a camera: a screenshot, a poster, a slide, a chart, or a document. The file includes:

• the kind of graphic and its language
• the aspect ratio, orientation, and palette
• every part in reading order, with its kind, bounding box, and text

Every field of image/graphic