Segmentation

Segmentation turns a video into time intervals. You describe the task in plain language, and Story0 returns intervals in seconds, each with a label, the evidence behind it, and a review flag where a person should look. It costs $0.10 per minute of video in either mode.

Partition or extract

Use partition when you need to know the state of the whole video, and extract when you only care about the parts that match.

Create a segmentation

This extract request finds jogging and keeps at most five moments of at least two seconds each:

const result = await story0.segments.createAndWait(
  {
    input: { asset_id: "asset_01m6gwgx34hq7d2m9ycv5k8b1e" },
    mode: "extract",
    instruction: "Moments where a person is jogging",
    constraints: { min_seconds: 2, max_segments: 5 },
    billing: { max_charge: { currency: "USD", amount: "0.25" } },
    metadata: { story_id: "park-0412" },
  },
  { idempotencyKey: "park-0412:jogging:v1" },
);

result = client.segments.create_and_wait(
    input={"asset_id": "asset_01m6gwgx34hq7d2m9ycv5k8b1e"},
    mode="extract",
    instruction="Moments where a person is jogging",
    constraints={"min_seconds": 2, "max_segments": 5},
    billing={"max_charge": {"currency": "USD", "amount": "0.25"}},
    metadata={"story_id": "park-0412"},
    idempotency_key="park-0412:jogging:v1",
)

curl https://api.story0.com/v1/segments \
  -H "Authorization: Bearer $STORY0_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: park-0412:jogging:v1" \
  -d '{
    "input": { "asset_id": "asset_01m6gwgx34hq7d2m9ycv5k8b1e" },
    "mode": "extract",
    "instruction": "Moments where a person is jogging",
    "constraints": { "min_seconds": 2, "max_segments": 5 },
    "billing": { "max_charge": { "currency": "USD", "amount": "0.25" } },
    "metadata": { "story_id": "park-0412" }
  }'

createAndWait returns the segmentation once it exists. segments.create (and the raw POST) returns the job immediately with 202 Accepted, for pipelines that wait through webhooks instead. For a complete partition example, see the Quickstart.

Request fields

Send an Idempotency-Key header with every request; the SDKs add one automatically. See Idempotency and retries.

Labels

Each label is an object with a key and a description:

{ "key": "dog_visible", "description": "At least one dog is visible, even partly out of frame." }

• Keys start with a lowercase letter and continue with lowercase letters, digits or underscores, up to 40 characters in all (^[a-z][a-z0-9_]{0,39}$). They appear in results, so keep them stable.
• Descriptions are 1 to 500 characters. They tell Story0 what the label means; this is where your decision rules go.
• Keys must be unique, and none may equal the uncertainty label.
• For partition, the labels should be mutually exclusive, and together with the uncertainty label they should cover every possibility. If two things can be true at once, such as "dog visible" and "person visible", run two partitions.

The uncertainty label

Rather than forcing a guess, Story0 uses the uncertainty label where the frames don't show enough to choose: the subject is obscured, blurred, cut off or too dark, or the shot cuts away. Uncertain intervals always have review.required: true and never carry evidence. The default label is uncertain; rename it with uncertainty.label (same key format) if that word means something else in your taxonomy.

The temporal contract

Partition

• Intervals are in time order and cover the whole range: the first starts at range.start, the last ends at range.end, and each one starts exactly where the previous one ended. There are no gaps and no overlaps.
• Each interval has exactly one label, either one of yours or the uncertainty label. Neighboring intervals always differ, because consecutive moments with the same label are merged.
• Intervals are start-inclusive and end-exclusive, [start, end). A moment at exactly 4.5 seconds belongs to the interval that starts at 4.5.
• Brief events are kept. A label seen in a single sampled frame gets its own interval, one frame interval long; nothing is smoothed away. Anything shorter than the frame interval can fall between samples and be missed, so raise sampling.fps if brief events matter.

Extract

• Moments are in time order. The gaps between them are moments that didn't match or couldn't be decided.
• Every moment has label: "match" and a short title, taken from its first piece of evidence (or "Matching moment" when there's none).
• A single uncertain frame between two matching frames doesn't split a moment. It's bridged, and the moment is flagged partly_uncertain with review.required: true.
• No matches is a successful result with an empty segments list, not an error.

Boundaries and resolution

Story0 samples frames at an even rate and judges each one. A boundary falls halfway between two neighboring samples whose labels differ, so boundaries sit on a grid as fine as the sampling: boundary_resolution_seconds, which equals coverage.frame_interval_seconds. At the default 2 frames per second that's 0.5 seconds. Read a boundary at 4.0 as "the change happened between the frames at 3.75 and 4.25", not as a frame-exact cut.

boundaries.snap accepts only none today. Boundaries are never moved to shot cuts, because a subject can enter or leave in the middle of an uninterrupted shot.

Sampling

By default the sampling rate depends on the length of the range you analyze:

Set sampling.fps between 0.1 and 4 to choose your own rate. Higher rates catch briefer events and give finer boundaries; lower rates are enough for long, slow footage. The rate doesn't change the price.

Note: Story0 inspects at most 600 frames per request. If sampling.fps multiplied by the length of the range would exceed that, the rate is lowered to fit, so the whole range is always covered; coverage.frame_interval_seconds reports the interval actually used. To look densely at part of a long video, combine a higher rate with a range.

coverage.sampled_frames and coverage.frame_interval_seconds report what was actually inspected. config.fps echoes the rate you asked for, or null for the default.

Analyze part of a video

"range": { "start": 120, "end": 180 }

• start defaults to 0 and end to the end of the video. When you give both, end must be greater than start.
• An end past the end of the video is treated as the end of the video.
• A range with less than a quarter of a second of video in it fails the job with range_outside_media (not charged).
• A range shorter than half a frame interval contains no sampled frame, so it comes back as a single uncertain interval.
• Intervals and evidence use the video's own timeline. An interval at 130–135 means 130 seconds from the start of the video, not of the range.
• The result's range shows what was analyzed, and the default sampling rate depends on the range's length rather than the video's.
• You're billed for the length of the range, not the whole video (still at least 30 seconds).

Shape extracted moments

constraints adjusts the moments extract returns:

min_seconds can't be greater than max_seconds. Story0 applies padding first, then splitting, then the count limit, so split pieces count individually toward max_segments. A few details to plan for:

• Padding never merges moments, so two short moments close together can overlap once padded.
• Near the start or end of the range, padding is cut off at the edge, so a padded moment can end up shorter than min_seconds.
• When max_segments drops moments, the remaining ones keep their IDs, so you might see moment_1 followed by moment_3.
• padded_to_min_seconds and split_at_max_seconds are informational. On their own they don't set review.required.

Read the result

coverage.timeline: "complete" is a structural promise: the whole range was sampled, and in partition mode the intervals tile it exactly. It doesn't mean every frame of the video was inspected, and it says nothing about whether every label is right.

Evidence

Each evidence item is a sampled frame that supports its interval's label:

{ "id": "ev_2", "kind": "seen", "start": 4.25, "end": 4.5, "description": "Golden retriever walking on a path in the park." }

• start is the time of the frame and end is where that frame's share of the timeline ends. Seek a player to start to show a reviewer what Story0 saw.
• kind is seen: a visual observation. description is a short note about the frame.
• partition intervals carry up to 2 items and extract moments up to 3. Uncertain intervals have none, and a definite interval can also have none.
• Evidence supports a label at that moment. It isn't proof that the label holds for every frame of the interval: what happens between samples is inferred.
• Evidence IDs (ev_1, ev_2, …) are unique within a result.

Review flags

Every interval and moment has review: { "required": …, "reasons": [ … ] }:

review.required: false means the analysis found no reason to flag the interval. It is not a clearance for licensing, publication or safety.

Empty, uncertain and failed

A file that isn't readable video fails with media_unreadable when its media facts are read, before any analysis or charge; it never turns into an all-uncertain timeline. Within a readable video, frames that can't be decoded are reported as uncertain time, never as a negative label: in partition they become an uncertain interval flagged for review, and in extract a gap counted in coverage.uncertain_seconds.

Reuse

If you send the same segmentation request for the same media again, Story0 can answer from your account's earlier analysis instead of analyzing the video again. "The same" means:

• the same media, recognized by its SHA-256 hash, so an identical file uploaded again counts; and
• the same request: mode, instruction (ignoring leading and trailing spaces), labels in the same order, uncertainty label (defaults count, so omitting it equals sending uncertain), and sampling.fps, range and constraints as you sent them.

metadata and billing.max_charge don't matter. A reused result has provenance.analysis_reused: true and the same segments, and it's charged only the $0.02-per-minute segment line instead of the full $0.10. Change anything about the request and Story0 analyzes the video again at the full price. Reuse stays within your account, and test-mode requests never use it.

Write good instructions and labels

Story0 judges each sampled frame on what it shows and then joins those judgments into intervals. Write for that.

• Describe what's visible. "At least one dog is visible" works. "The dog is barking" depends on sound, which Story0 doesn't use.
• Say what counts and what doesn't. "Count a dog that is partly out of frame. An audible bark alone does not count." Edge cases written into a description beat a guess.
• Keep partition labels exclusive. dog_visible and no_dog_visible split every moment cleanly. "Dog" and "person" don't, because both can be on screen; run two partitions instead.
• Prefer states to events. "A player is celebrating" can be seen in one frame; "a goal is scored" needs the moments before and after. For events, use extract with an instruction about what the frames show, such as "the ball crossing the goal line".
• Ask about what a viewer would notice. Frames are inspected at a reduced size, so small on-screen text, distant faces and fine detail may not be legible.
• Change descriptions, keep keys. Keys live in your data and dashboards. Refine behavior by editing a description, and version your idempotency keys when you do.
• Check before you scale. Run a handful of representative videos, look at the uncertain intervals and the evidence, and tighten the descriptions where results surprise you.

Edge cases

• One label throughout: a single interval covering the whole range.
• Uncertain throughout: a single interval with the uncertainty label and review.required: true.
• Very short videos: billed as 30 seconds. At 2 frames per second, a 3-second clip gives 6 frames.
• A range that runs past the end: end is clamped to the video's duration. Keep start inside the video.
• A tiny range: a range shorter than half a frame interval contains no sample, so it comes back as one uncertain interval.

Record reviewer feedback (preview)

When a person checks a result, you can record their verdict with POST /v1/feedback. Feedback is append-only and never changes the result it refers to.

await story0.feedback.create({
  result_id: "seg_01m6gwj2c15y8n3q7w2e9t4h6b",
  segment_id: "interval_3",
  kind: "relabel",
  label: "dog_visible",
  start: 15,
  end: 15.5,
  actor: "editor@example.com",
  comment: "The dog's tail is still in frame at the left edge.",
});

client.feedback.create(
    result_id="seg_01m6gwj2c15y8n3q7w2e9t4h6b",
    segment_id="interval_3",
    kind="relabel",
    label="dog_visible",
    start=15,
    end=15.5,
    actor="editor@example.com",
    comment="The dog's tail is still in frame at the left edge.",
)

kind is correct, incorrect, relabel (which needs the reviewer's label) or note. Point it at a segment_id from the result, optionally with start and end in seconds, a comment of up to 2,000 characters and an actor in your own terms. For screenings, use category instead of segment_id. Without an Idempotency-Key, a repeated call records the feedback twice; with one, the repeat returns the first record.