millee.md
← All guides
Content Updated June 15, 2026

Claude × Remotion: Edit Your Videos by Talking

Turn raw footage into a finished short with silent gaps cut, captions added, motion graphics and sound design, just by telling Claude what you want in plain English.

Turn raw footage into a fully-edited short — silent gaps cut, captions added, motion graphics, and sound design — just by talking to Claude in plain English.

💡 How it works: Claude Code (inside the Claude desktop app) does the editing in Remotion, a code-based video tool. You drop in a clip, tell Claude what you want, wait a couple of minutes, and you get a finished video.

✅ What you’ll need

1. Install the basics

Start a new chat in Claude Code, and paste the following prompt:

Set up a Remotion video project for vertical short-form reels (1080×1920 @ 30fps) with the following:

1. Initialize a Remotion project (npm create video@latest, blank template).
2. Install ffmpeg if it's not already on this machine (brew install ffmpeg on Mac).
3. Clone and build whisper.cpp into a local bin/ folder, and download the base.en model.
4. Create scripts/transcribe.mjs — a Node script that takes an mp4 path, runs whisper.cpp on the audio, and outputs a JSON file of word-level timestamps in the shape { text, startMs, endMs }.
5. Create a public/recordings/ folder where I'll drop raw MP4s.
6. Create src/Captions.tsx — a Remotion component that reads the transcript JSON and displays the active word at one of three Y positions depending on framing context (gap overlay / screen-recording zoom / full-frame).
7. Create src/Root.tsx with a single Composition that mounts the raw video full-frame, overlays gap scenes on top, and renders captions over everything.

Don't write any actual reels yet — just the framework. Use TypeScript. Print a final summary of every command I need to run, and confirm everything builds.

2. Install the Remotion skill into Claude Code

  1. Go to Customize → Skills → Add Skill (the + icon) → Create Skill → Upload Skill

The Upload Skill panel in Claude Code

  1. Download the skill from this Google Drive link
  2. Drag/upload the .md file into the “Upload skill” box shown above

3. Set up your Sound Pack & Music Pack

Sound effects and background music live in your project’s public/assets/ folder.

  1. Create these two folders (or just ask Claude to):
    • public/assets/sound pack/
    • public/assets/music pack/
  2. Drop your sound effects (.mp3) into sound pack, and your music tracks into music pack.

The sound pack folder structure

The music pack folder structure

Where to get them for free:

💡 Name your files clearly (e.g. woosh.mp3, ding.mp3, riser.mp3) — it helps Claude pick the right sound for each moment.

🎯 How to use it — 3 use cases

1. Adding captions

  1. Drop your raw clip into public/recordings/.

  2. Tell Claude:

    “Transcribe this and add TikTok-style captions, 2 to 3 words at a time.”

  3. Claude transcribes your audio and lays word-by-word captions across the video, and will fix any misheard names if you tell it the correct spelling.

Word-by-word captions rendered over the footage

2. Creating motion graphics

  1. While filming, crop/move your footage down to create a black square in the part of the frame where you want a graphic to appear — that’s the marker.

A black square marker left in the top half of the frame

  1. Drop the clip into public/recordings/ and tell Claude what each moment should show:

    “In the black box while I talk about X, animate a Y.”

  2. Claude finds the black boxes automatically and builds clean motion graphics inside them, matched to what you’re saying.

The finished motion graphic filling the marker box

3. Using the Sound Pack & Music Pack

  1. Make sure your sounds/music are in the folders from step 3.

  2. Tell Claude:

    “Add [track name] as background music, low under my voice, and place sound effects from the sound pack wherever they fit.”

  3. Claude sets the music low enough that your voice stays prominent, and drops sound effects (whooshes, dings, etc.) right on the animation beats. Ask it to swap or adjust the volume of any sound by ear.

🚀 That’s it. Drop in raw footage, tell Claude how you want it to look, and let it edit. Anything you want changed afterward, just ask in plain English.

← All guides