inference.minimax.h3.chili.ref2vid.v0

View markdown

The inference.minimax.h3.chili.ref2vid.v0 job generates a 24 fps video with synchronized stereo audio using uploaded references. Images guide subjects and style, videos guide motion, and audio guides voices and music.

Upload the references as job inputs and list their filenames in references, in the order you want them used. If references is omitted, all uploaded inputs are used in upload order. Include at least one image or video; audio cannot be used on its own.

{
    "type": "inference.minimax.h3.chili.ref2vid.v0",
    "config": {
        "prompt": "the subject walks through a sunlit park, birds chirping",
        "references": ["face.jpg", "profile.jpg"],
        "duration": 6
    }
}

Prompting

The prompt is used exactly as written: nothing rewrites or expands it before it reaches the model, which was trained on descriptive captions. Describe the shot rather than making a request: the subject, setting, time of day, weather, lighting, camera move and motion. For example, “show me what christmas would be like in Tokyo” tends to produce a generic winter scene, while “Shibuya crossing in Tokyo on Christmas Eve at night, light snow falling, crowds with umbrellas, a giant illuminated Christmas tree, neon signs reflecting in the wet street, slow cinematic dolly shot” produces the intended scene. The references already fix who or what appears, so use the prompt to describe the setting, action, motion and camera move you want for them. This job type runs a distilled 8-evaluation recipe, so it follows the prompt less closely than full-step generation; vague prompts steer it even less, so a detailed description matters more here.

You can upload up to 15 references, subject to these limits:

Reference	Maximum count	Supported formats	Maximum file size	Duration
Image	9	JPEG, PNG, WEBP, HEIC, HEIF	30 MB each	—
Video	3	MP4 or MOV with H.264/H.265	50 MB each	2–15 seconds each; 15 seconds combined
Audio	3	WAV or MP3	15 MB each	2–15 seconds each; 15 seconds combined

Images and video frames must be 256–5760 pixels per side, with a width-to-height ratio between 0.4 and 2.5. Video frame rates must be 23.976–60 fps. The combined duration limits apply separately to video references and audio references.

Structured prompt format

MiniMax trains H3 on a structured prompt, not on free text. Sending that structure yourself is the most reliable way to control the soundtrack: on a 24-prompt bank rendered on this tier, 10 of 24 free-text prompts came back with an unrequested music bed, and 0 of 24 did when the same prompts were sent in this format (same seeds, same duration). The format adds three things free text usually leaves to the model’s own judgement: an explicit soundscape, an explicit music decision, and an explicit statement of whether anyone speaks.

Rules that apply to every section below:

A line is spoken only when it is written as dialogue: she (S1) says softly: <d>[English] Don't tell me you forgot again.</d>. (S1) numbers the speaker, <d>…</d> encloses the spoken words, and the language tag is always [English] (the model voices other languages inside the tag as written). Put dialogue last in the shot: prose after a spoken line tends to be voiced too.
When nobody speaks, say so: She does not speak., He does not speak. or, for a scene with no single subject, No spoken dialogue. Without this sentence the model may invent speech.
overall_soundscape: is one sentence of diegetic sound, i.e. what the scene itself produces. For a person: Soft room tone, her quiet breathing and the faint rustle of fabric as she moves. For a scene: Natural ambient sound matching the scene and its visible actions.
non_diegetic_music: is N/A unless you want a score, in which case describe it in one sentence, e.g. Calm ambient piano plays throughout.
Separate sections with a blank line. The whole prompt, format included, must stay within the 7000-character limit.
Write the visual description exactly as the Prompting section above recommends; the format wraps it, it does not replace it.

Beyond the soundtrack, the same format carries the rest of what the model was trained to read:

Write the description as a timeline, not a caption: what is visible and audible first, then the actions, reactions and state changes in the order they happen. Every detail should be something that can be seen or heard; requests like “make it cinematic” steer little.
Give camera movement as type, amplitude and speed: “the camera pushes in with small amplitude at slow speed”, “trucks right slowly”, “holds a locked-off shot”.
For more than one shot, keep [Shot 1] without a timestamp and start each later shot with a strictly increasing cut time inside the duration: [Shot 2] At 00:05.000, the camera cuts to a close-up of steam rising from the sliced bread.
Keep a speaker’s id across shots. For narration that is not spoken on screen, use the exact phrase “says in an off-screen voiceover” and add that the on-screen character’s lips remain closed. When a line crosses a cut, mark both parts with <scenetrans> and say the audio continues across the cut; when the clip ends mid-sentence, mark the line with <cutoff>.
When you do want a score, describe it as instrumentation, tempo and dynamics (“sparse piano at a slow tempo, joined by low strings that build and fade”) rather than mood words.
Text that must appear on screen is spelled out verbatim in the prompt.
An unspecified clip gets natural ambience; write overall_soundscape: N/A only when you want silence.

Reference-to-video uses the full six-section form. Uploaded images are <Picture 1>, <Picture 2>, … and uploaded videos <Video 1>, <Video 2>, … in upload order; the subject they define is <Subject 1>. Every uploaded reference should be named in subject_definitions: and retention_analysis:. This example has two images and one video:

subject_definitions:
<Subject 1> is the subject in <Video 1>, keeping its appearance and the setting, lighting and framing shown there.
<Video 1> is the reference video that provides the scene, framing and the motion; its audio is not reused.
<Picture 1> is an appearance reference for the subject described below.
<Picture 2> is an appearance reference for the subject described below.

summary:
[reference generation] The target video shows <Subject 1> as described below. The subject walks through a park and waves at the camera. No spoken dialogue.

retention_analysis:
<Subject 1> (appears in [Shot 1]): fully_preserved - its identity and appearance are retained; everything else follows the description.
<Video 1>: reference - its scene, framing and motion guide the target video; the audio signal is not copied.

detailed_description:
The target video is a single continuous shot.
[Shot 1] <Subject 1>, from <Video 1>, the subject walks through a park and waves at the camera. No spoken dialogue.

overall_soundscape:
Natural ambient sound matching the scene and its visible actions.

non_diegetic_music:
N/A

With images only, drop the <Video 1> lines and describe <Subject 1> as the subject shown in <Picture 1>. Uploaded audio is not named in the prompt; it is used as the soundtrack reference directly.

The task type in summary: tells the model how the references are used: [reference generation] when they guide appearance, motion or style; [video continuation] when the clip should pick up where a reference video ends; [audio reuse] when an uploaded track is copied, [audio reference] when only its style, timbre or beat is imitated. Combine them with + when more than one applies.

prompt is required and must contain 3–7000 characters. Describe both the visual scene and the sounds you want to hear.

duration accepts whole seconds from 4 to 15 and defaults to 6. The actual video may be slightly longer than requested: 4 seconds produces about 4.46 seconds, and 15 seconds produces about 15.08 seconds.

resolution supports only 768P, which is also the default. Exact output dimensions depend on the aspect ratio. seed accepts an integer from 0 to 2147483647 for reproducible generation; omit it to use a random seed.

aspect_ratio defaults to adaptive, which uses the supported ratio closest to the first image or video reference. To choose the output shape explicitly, set it to 16:9, 4:3, 1:1, 3:4, 9:16, or 21:9.

Schema

{
  "type": "object",
  "required": [
    "type",
    "config"
  ],
  "additionalProperties": false,
  "properties": {
    "type": {
      "type": "string",
      "enum": [
        "inference.minimax.h3.base.ref2vid.v0",
        "inference.minimax.h3.fast.ref2vid.v0",
        "inference.minimax.h3.base.ref2vid.v1",
        "inference.minimax.h3.fast.ref2vid.v1",
        "inference.minimax.h3.turbo.ref2vid.v0",
        "inference.minimax.h3.hyper.ref2vid.v0",
        "inference.minimax.h3.chili.ref2vid.v0"
      ]
    },
    "config": {
      "type": "object",
      "required": [
        "prompt"
      ],
      "additionalProperties": false,
      "properties": {
        "references": {
          "type": "array",
          "minItems": 1,
          "maxItems": 15,
          "items": {
            "type": "string",
            "format": "filename",
            "example": "subject.png",
            "description": "Filename of a reference file."
          },
          "description": "Reference files to feature in the video, in the order the model should read them. The role of each reference is derived from its content type: images, videos, and audio act as subject/style, motion, and voice/music references respectively. Up to 9 images (JPEG, PNG, WEBP, HEIC, or HEIF; 30MB each; 256-5760px per side; width/height ratio 0.4-2.5), 3 videos (MP4 or MOV with H.264/H.265; 50MB each; 2-15s each and 15s combined; same pixel and ratio bounds; 23.976-60 fps), and 3 audio files (WAV or MP3; 15MB each; 2-15s each and 15s combined); audio never on its own. Optional: with the field absent, every uploaded input is a reference, in upload order."
        },
        "prompt": {
          "type": "string",
          "minLength": 3,
          "maxLength": 7000,
          "default": "a dancing cat under moonlight",
          "description": "Description of the desired video and audio."
        },
        "duration": {
          "type": "integer",
          "minimum": 4,
          "maximum": 15,
          "default": 6,
          "description": "Duration of the video in seconds. Rounded up to the model's frame grid (4 s returns 4.46 s, 15 s returns 15.08 s)."
        },
        "aspect_ratio": {
          "type": "string",
          "enum": [
            "adaptive",
            "16:9",
            "4:3",
            "1:1",
            "3:4",
            "9:16",
            "21:9"
          ],
          "default": "adaptive",
          "description": "Controls the approximate aspect ratio of the resulting video. With \"adaptive\" (the default), the aspect ratio derives from the first visual reference (snapped to the nearest supported ratio); an explicit ratio overrides it."
        },
        "resolution": {
          "type": "string",
          "enum": [
            "768P"
          ],
          "default": "768P",
          "description": "Only 768P is supported: the canvas targets a 768px short edge with a 768x1344 pixel area cap and 32px rounding; exact width and height follow the aspect ratio of the first visual reference."
        },
        "seed": {
          "type": "integer",
          "minimum": 0,
          "maximum": 2147483647,
          "description": "Seed for reproducible generation. Random when omitted."
        }
      }
    }
  }
}