HappyHorse 1.0 Prompting Guide: 12 Techniques for Joint Audio + Video

HappyHorse 1.0 launched on April 9, 2026 as the first open-source AI video model to top the Artificial Analysis Video Arena leaderboard — Elo 1333 in Text-to-Video and Elo 1392 in Image-to-Video, both #1 globally under blind human preference voting. Built by Alibaba's Taotian Future Life Lab and led by Zhang Di (former technical architect of Kling AI), it is a 15-billion-parameter unified Transformer that generates joint video and audio from a text or image prompt in a single forward pass.

Most AI video users have had the same experience: the more specific the prompt gets, the more the model quietly sands off the details. HappyHorse 1.0 behaves differently — dense prompts often work better here than simplified ones, but only when the structure is doing real work. This guide is a technique-by-technique walkthrough of the patterns that show up repeatedly in production use, condensed into twelve copy-paste examples grounded in publicly available API documentation and community testing.

HappyHorse 1.0 is more tolerant of detailed prompts than most mainstream video models. That tolerance is not a license to write longer — it is a license to keep more concrete detail. The prompt should tell the model what exists in the scene, what moves, how the camera behaves, how light behaves, what the audio bed should sound like, and what the scene should feel like in time.

The structural order that holds together best across use cases is: scene and timing first, subject second, action and motion third, camera language fourth, lighting and texture fifth, audio intent sixth. Early tokens set the rendering "mode" — talking head, product spot, ambient B-roll, multi-shot scene — and later tokens flesh out the detail. Free-form sentences that interleave these layers tend to drift; ordered prompts hold their shape across reruns.

HappyHorse 1.0 has an explicit photorealistic rendering mode, and the most reliable way to engage it is to use the word "photorealistic" — or adjacent phrases like "shot like a 35mm film photograph", "documentary style", "iPhone-style handheld" — in the prompt. Detailed camera specs (specific lens, sensor, ISO) are interpreted loosely; treat them as look-and-feel cues rather than physical simulation. The bigger lever is naming imperfections: pores, fine lines, fabric wear, available light, slight motion blur. Anti-cues like "no glamorization" and "no heavy retouching" push the model away from the generic AI portrait look.

HappyHorse 1.0 follows explicit camera-language tokens. The reliable cookbook of moves: push-in, pull-back, dolly, tracking shot, orbit, handheld follow, locked-off, slow pan, tilt up, tilt down, crane up, crane down. Combine a movement with an angle (low-angle, eye-level, overhead, Dutch) and a shot size (wide, medium, close-up, extreme close-up) to nail the cinematographic register. For trajectories, specify both endpoints — "from frame-right to frame-left" — instead of leaving the start and end positions for the model to infer.

Dialogue is HappyHorse 1.0's signature capability gap from earlier video models. The lip-sync pathway is engaged by placing the literal line in quotes — the model treats quoted text as the dialogue track and aligns mouth shapes to the phonemes inside the same denoising step. For best results: name the language explicitly, place the line in quotes, and add a constraint clause for the audio bed ("no music, ambient room tone only").

The single most useful add-on phrase is "EXACT, verbatim, no extra characters" applied to the literal line. Without it, the model occasionally paraphrases or appends a short interjection; with it, the rendered audio matches the prompt exactly. Use Pro mode for clean dialogue.

HappyHorse 1.0 ships native lip-sync support in seven languages: English, Mandarin, Cantonese, Japanese, Korean, German, and French. The prompting rules are identical across languages: place the literal line in quotes, name the language explicitly, and let the model align mouth shapes to the phonemes of that language. For non-Latin scripts (Japanese, Mandarin, Cantonese, Korean), write the line natively rather than romanizing it — the lip-sync pathway is trained on actual phonemes, not on transliterated approximations.

For mixed-language scenes (one character speaking English, another speaking Mandarin), list each line separately with its speaker and language. The model handles the language pairing automatically as long as the elements are unambiguous.

For people in scenes, describe scale, body framing, gaze, and object interactions. Generic phrases like "a person doing X" tend to drift on body proportion and limb articulation. Concrete phrases — "full body visible, feet included," "child-sized relative to the table," "looking down at the open book, not at the camera," "hands naturally gripping the handlebars" — pin down the geometry.

This is the difference between "two friends laughing" rendering as a stiff promo shot versus rendering as a believable in-the-moment clip. Add an explicit gaze direction whenever you do not want the subject looking at the camera; the default tendency is camera-aware framing.

HappyHorse 1.0 generates audio by default — if you do not name the audio bed, the model picks one. The most underused part of HappyHorse 1.0 prompting is the audio intent line. State the audio explicitly: voice-over (in quotes), dialogue (in quotes), foley (named sounds — wave bed, traffic, footsteps, fabric rustle, ceramic tap), ambient bed, and music (or "no music"). When you want a specific sound to land at a specific frame, attach it to the visual event ("a soft ceramic tap as the dropper cap is lifted").

In the fal API, the `happyhorse_elements` field lets you define named assets — each with a name, a description, and 2–4 input images — and reference them in the prompt as `@element_name`. Maximum three elements per task. This is HappyHorse 1.0's explicit composition layer: it lets you pin a specific dog, a specific person, or a specific product into the scene with no naming ambiguity.

The pattern that works: define the elements first (block at the top of the prompt), then write the scene prompt referencing each element by its `@name`. The model treats each element as a distinct asset rather than blending them into one composite reference.

When you pass multiple input images to the reference-to-video endpoint, the recommendation is to reference each input by index and description ("Image 1: product photo… Image 2: style reference… Image 3: target scene…") and describe how they interact ("place the bag from Image 1 into the scene of Image 3, match the lighting of Image 2"). Be explicit about which elements move where.

This indexing convention is what unlocks the "insert this product/person into that scene" workflow without re-generating the whole frame. It also lets you combine three or four references in one call — for example, a person from image 1 wearing garments from images 2, 3, and 4 — with the model treating each input as a distinct asset rather than blending them into one composite reference.

HappyHorse 1.0 supports targeted edits through the video-edit endpoint without explicit masks, but the prompt has to be tight to avoid drift. The phrase-of-record is: "change only X" + "keep everything else the same" + repeat the preserve list. For genuinely surgical edits — time-of-day swaps, garment color changes, object removals — also explicitly say not to alter motion, framing, camera move, or surrounding objects, and state whether the audio bed should be preserved or regenerated.

The restated preserve list is what differentiates a clean first-pass edit from a near-miss that requires three retries. Repetition is intentional: the model treats both the change instruction and the preserve list as constraints, and listing the preserve elements twice raises the weight on each one.

Long prompts can work well on HappyHorse 1.0, but debugging is easier when you start with a clean base prompt and refine with small, single-change follow-ups. The recommended pattern: ship a clean first prompt at Std mode, then iterate with phrases like "make the lighting warmer," "remove the music," "tighten the framing." Use references like "same scene as before" or "the subject" to leverage context — but re-specify any critical detail if it begins to drift.

This is the inverse of the design-tool reflex, where each pass adds more constraints. With HappyHorse 1.0, each refinement pass should remove the noise from the prior pass and change at most one or two things. Multi-region simultaneous edits (three or more independent changes in a single call) typically need two or three iterations for a clean result.

HappyHorse 1.0 was trained by a team operating in both Chinese and English, which gives the model unusually strong world knowledge for culturally-specific scenes — Beijing hutong courtyards, Cantonese street markets, Japanese izakaya interiors, Korean bookshops. The implication for prompting: write the scene in whichever language better captures the cultural specificity. For East Asian scenes, native-language prompts often produce more physically accurate detail than translated equivalents.

For any well-documented event, place, or cultural moment, you can prompt with situational cues — a date, a place, a notable scene — and let the model infer the visual context without spelling it out. This works for pre-2026 references; for very recent post-launch events, provide a reference image instead of relying on world knowledge.

For sequence work — short narrative scenes, brand spots with a beginning-middle-end, storyboard-to-shot conversions — the fal text-to-video endpoint exposes a multi-shot mode. Up to five shots, each up to twelve seconds, individually prompted, generated in one call with character identity and visual style preserved across shots. The pattern that works: describe the protagonist once at the top, then list each shot as a numbered beat with its own framing, action, lighting, and audio. State the consistency requirements explicitly — face, hair, wardrobe — so the model preserves them across all shots.

HappyHorse 1.0 exposes two modes — Std (the DMD-2 distilled 8-step student) and Pro (an extended denoising schedule) — and supports clip lengths from 3 to 15 seconds in five aspect ratios (16:9, 9:16, 1:1, 4:3, 3:4) at 720p or 1080p. Audio is on by default and adds about 50% to credit usage. Std is good enough for short-form, ideation, and any clip that goes through downstream review. Pro is the right default for dialogue scenes, brand hero shots, multi-shot sequences, and image-to-video animations where fidelity is the bottleneck.

If you take one thing from this guide, take the template. It follows the recommended structural order and works as a copy-paste starting point for almost every use case in this article:

Intended use → Scene and timing → Subject (with scale, pose, gaze) → Action and motion → Camera language (movement, angle, shot size) → Lighting and texture (direction, quality) → Literal in-prompt dialogue in quotes (EXACT, verbatim) → Audio intent (voice-over, foley, ambient bed, music or no music) → Resolution, aspect ratio, duration.

Start at Std mode, 720p, 16:9 (or 9:16 for short-form), audio on. Run two generations to calibrate the prompt, then move to Pro mode and 1080p for the final asset. For refinements, edit the existing clip with a natural-language instruction rather than regenerating from scratch — the latter is the single biggest source of drift in production work.