Most open text-to-video models still think in single takes. You write one prompt, you get one continuous shot. ShotPlan, a new framework from TeleAI (China Telecom's Institute of Artificial Intelligence) and the Harbin Institute of Technology, changes that. It generates a multi-shot cinematic sequence from a single text prompt with per-shot captions, placing hard cuts at exact timestamps, adding cross-fade transitions, and steering camera moves inside individual shots. The team reports a transition deviation of just 0.64 frames, meaning cuts land almost exactly where you asked for them.
The models and training code are already public under an Apache-2.0 license, so this is not a paper-only teaser. If you make short cinematic clips, product spots, or narrative sequences, ShotPlan is worth understanding now, because it targets the exact problem that makes AI video hard to edit: control over where one shot ends and the next begins.
What ShotPlan actually does
A standard text-to-video model produces a single unbroken clip. To build a scene with three shots, most creators generate three separate videos and stitch them in an editor, then fight to keep the character, lighting, and setting consistent across the seams. ShotPlan folds that whole process into one generation pass. You give it a prompt plus captions for each shot, and it returns a single video that already contains the cuts and transitions in the right places.
It supports three kinds of shot boundary. Hard cuts drop instantly to a new shot at a chosen frame. Soft transitions cross-fade between shots with a controllable fade-in start and end. And temporally localized camera movement lets a move like a circle, zoom, dolly, or truck happen inside a specific shot rather than smearing across the whole clip. ShotPlan is built on Wan2.1-T2V-14B, the open text-to-video model, so it inherits that model's visual quality while adding the shot-level structure on top.
How the planning tokens work
The core idea is a set of learnable planning tokens. Rather than bolting on a separate shot-detection or editing stage, ShotPlan injects these tokens into the diffusion process so the model knows, during generation, where each shot begins and ends and what transition connects them. Once the plan has done its job, the tokens are removed before the final output is produced, which means they add no extra cost at inference time.
To make the timing precise, the authors introduce Fractional Temporal RoPE, or FRoPE. Standard rotary position embeddings assign whole-number positions to tokens. FRoPE assigns each token a fractional latent-time coordinate, which lets the model reason about sub-frame timing inside the compressed latent space (the video autoencoder compresses time by a factor of four, with a temporal stride of four). That fractional precision is what drives the 0.64-frame transition accuracy. The reference configuration generates clips at 832 by 480 resolution and 81 frames, and the models are published on Hugging Face so you can inspect the setup directly.
How ShotPlan compares to your current options
The practical question for a creator is not which research benchmark wins, but which workflow gets you a clean multi-shot clip fastest. Here is how ShotPlan stacks up against the two approaches most people use today.
| Approach | Shot boundaries | Cross-shot consistency | Editing effort | Cost / access |
|---|---|---|---|---|
| ShotPlan | Hard cuts and cross-fades placed by frame in one pass | Handled inside a single generation | Write per-shot captions, generate once | Open weights, Apache-2.0, runs on Wan2.1 |
| Single-prompt text-to-video | None; one continuous shot | Not applicable (one shot only) | Manual cutting to fake multiple shots | Varies by model |
| Generate clips separately, edit together | You set them in an editor | Hard; character and setting drift between clips | High; multiple generations plus timeline work | Multiple render passes |
ShotPlan benchmarks itself against dedicated multi-shot research systems including CineTrans, EchoShot, HoloCine, and MultiShotMaster, reporting scores on text alignment, character consistency, scene consistency, and narrative coherence. Those academic baselines are the right yardstick for researchers, but for a working creator the more useful comparison is the one above: ShotPlan replaces a multi-clip, multi-render editing grind with a single structured prompt.
The models and dataset you can download
ShotPlan ships as real, downloadable assets rather than a demo behind a form. The checkpoints are on Hugging Face, including a Wan2.2-based high-noise variant alongside the Wan2.1 build. The training data is a curated set of 13K multi-shot clips, released as a public dataset, though that dataset carries research-only terms separate from the code license. The code itself is Apache-2.0, inherited from DiffSynth-Studio, the diffusion toolkit ShotPlan vendors and builds on.
That licensing split matters if you plan to use ShotPlan commercially. The model weights and inference code are permissively licensed, but the training dataset is not, so you can run the model to make clips while keeping the dataset itself out of any commercial pipeline. The paper is listed as coming soon, so the exact reproduction numbers may firm up once it lands.
How to put ShotPlan into a workflow
If you already run open video models locally or on a rented GPU, ShotPlan slots into that setup with a few concrete steps.
- Get the base ready. ShotPlan runs on top of Wan2.1-T2V-14B through DiffSynth-Studio, so install those first from the ShotPlan repository and pull the matching checkpoint from Hugging Face.
- Write a global prompt plus per-shot captions. Describe the overall scene once, then write a short caption for each shot. This is where you decide how many shots the sequence has and what happens in each.
- Specify the transitions. Choose a hard cut or a cross-fade between each pair of shots, and set the frame where the boundary should fall. For a cross-fade, set the fade-in start and end.
- Add camera moves where you want them. Attach a circle, zoom, dolly, or truck to a specific shot rather than the whole clip, so the motion reads as deliberate cinematography.
- Generate once and review the seams. Because the cuts are placed during generation, check that the transitions land where you intended, then iterate on captions rather than re-editing in a timeline.
What this enables: a creator can go from a written shot list to a single cinematic clip with correct cuts, without generating separate clips and stitching them. That collapses the slowest part of AI video work, keeping characters and settings consistent across shots, into one prompt-driven step.
Limits worth knowing before you rely on it
ShotPlan is early. The reference output is 832 by 480 at 81 frames, which is fine for social and previz but short of a finished 4K deliverable, and you will likely upscale afterward. The reported benchmark scores (text alignment 0.26, character consistency 0.46, scene consistency 0.37, narrative 0.88) show real strength on narrative structure but leave room on fine-grained consistency, so expect to regenerate shots that drift. And with the paper still forthcoming, the full method details and reproducibility claims are not yet locked. Treat it as a fast-moving open project you can already test, not a finished product.
Frequently asked questions
What is ShotPlan?
ShotPlan is an open framework that generates multi-shot cinematic video from a single text prompt with per-shot captions. It places hard cuts and cross-fades at chosen frames and adds camera moves inside individual shots, built on the Wan2.1-T2V-14B model.
Is ShotPlan free to use?
The code and model weights are released under Apache-2.0, so they are free to use, including commercially. The training dataset is released separately under research-only terms, so the dataset itself should not go into a commercial pipeline even though the model can.
What hardware do I need to run it?
ShotPlan runs on top of Wan2.1-T2V-14B through DiffSynth-Studio, so it needs a GPU capable of running a 14B-parameter video diffusion model. Most creators will use a high-memory GPU locally or a rented cloud instance rather than a laptop.
How is this different from generating clips and editing them together?
Editing separate clips forces you to keep characters and settings consistent across renders and then cut them in a timeline. ShotPlan produces the whole multi-shot sequence in one generation with the cuts already placed, so consistency is handled inside the model and the editing step largely disappears.
What transitions and camera moves does it support?
It supports hard cuts at exact timestamps and soft cross-fade transitions with a controllable fade-in start and end. For camera work it supports temporally localized moves including circle, zoom, dolly, and truck, applied to a specific shot rather than the entire clip.
Is there a paper I can read?
The paper is listed as coming soon on the project page. The code, model checkpoints, and dataset are already public on GitHub and Hugging Face, so you can test the system before the paper is released.
