How I Built a Manifest-Driven Video Engine with Remotion and ElevenLabs

I kept hitting the same wall.

Text posts were grand. Audio had a clean pipeline. Images were sorted. But short launch videos were still a mess: screen recordings, ad-hoc edits, random music picks, and no way to reproduce the result next week.

I'm not gonna lie to you, I burned way too many hours pretending this could be solved with "just one more manual edit." It couldn't.

So I built a dedicated video-engine repo and treated video like code.

1. Define scenes as data.
2. Generate or reuse audio.
3. Render deterministically.
4. Review against quality targets.

Inspiration tweet: Thariq's original post on X.

Why a Separate Engine

I tested a few approaches. Template-only was too rigid. Conversational scene building sounded cool but was painful to replay exactly.

Manifest-driven won because it gave me control and replayability.

project context -> manifest.json -> audio generation -> Remotion render -> review report

The Core Stack

None of this is exotic on its own. The value is how the pieces fit together.

• Remotion for frame-driven React rendering
• TransitionSeries for scene transitions
• Zod for manifest validation
• ElevenLabs Sound Effects API for SFX
• ElevenLabs Music API for background tracks
• parseMedia for duration checks

The strict rule in this engine: no CSS timeline animations. Motion comes from useCurrentFrame(), spring(), and interpolate() so renders stay deterministic and debuggable.

Manifest-First, Not Component-First

The manifest is the source of truth. Components are just the renderer.

{
  "meta": { "title": "My Demo", "fps": 30, "width": 1920, "height": 1080 },
  "theme": { "bgColor": "#0a0a0a", "accentColor": "#D57456" },
  "music": { "prompt": "warm uplifting lo-fi", "durationMs": 35000 },
  "scenes": [
    { "id": "intro", "type": "split-intro", "durationMs": 3200 },
    { "id": "focus", "type": "camera-sequence", "durationMs": 5200 },
    { "id": "proof", "type": "animated-card", "durationMs": 5000 }
  ]
}

This gave me three quick wins:

1. Scene review got easier because timing and transitions were explicit.
2. Regeneration became cheap: edit JSON, rerender, compare.
3. Validation caught bad scene types and missing assets before long renders.

Video Demo: Reference vs Recreation

1) Reference Video (what we are recreating)

• Original post: Thariq reference post on X

2) Recreated Version (final output)

• Final render version: twitter-insights-v11.mp4

What the Quality Gap Actually Looked Like

The first renders looked rough. Flat motion. Clipped text. Contrast issues on light cards. Springs that felt like they were stuck in mud.

So I did a frame-level comparison against the reference videos and forced myself to set hard targets instead of guessing.

Then I tackled fixes in phases: clipping and contrast first, spring tuning second, transition and scene-type polish after that. Less glamorous, way more effective.

Audio-First Was the Right Call

I originally planned visuals first. That was a mistake, full stop.

Switching to audio-first made timing sane again:

1. Parse manifest for SFX/music needs.
2. Generate or reuse cached audio via prompt hash.
3. Measure real durations.
4. Update manifest timing.
5. Render with known assets.

That removed a lot of the "why is this transition late?" debugging spiral.

The Review Loop That Made It Usable

The turning point was building a single review command after render.

Before this, I was basically squinting at frames and arguing with myself. After this, I got a concrete report:

• Audio profile and mood checks
• Visual structure checks
• Comparison scorecard vs reference
• PASS/WARN/FAIL with next actions

python3 scripts/review-render.py out/video.mp4 \
  --reference data/reference-profiles/v1-insights.json \
  --output out/reviews/video-review.md

What I Learned (The Hard Way)

1. Determinism beats convenience for production content.
2. A manifest is not overhead, it's your safety rail.
3. Quality criteria need numbers, not vibes.
4. Audio timing quietly breaks everything if you ignore it.
5. Review automation matters before adding more scene types.

Also: if animation feels dead, check spring damping before rewriting half your scene library. Ask me how I know.

What Comes Next

• Better templates by video archetype
• Cleaner social format variants (1:1, 9:16)
• Tighter /content-video integration in the full content pipeline
• Stronger visual regression checks

It remains to be seen how far I can push full automation without losing quality, but this is already a massive step up from manual editing roulette.

References

• Remotion
• Remotion AI docs
• Remotion Transitions
• ElevenLabs API docs
• Thariq reference post on X