Anime Production

My video-making journey with Reincarnator × Regressor

I always wanted to try my hand at creating the same manga/manhua/manhwa shorts and videos that I've frequently watched on YouTube, and when I first saw the Anime Production Skill on MCP Marketplace, I thought I could do it. So I subscribed to Claude Code with that in mind. Having drawn my own illustrations for Reincarnator x Regressor, I thought I could produce similar short videos. However, I did encounter a few hurdles during the process.

Reincarnator x Regressor Teaser PV

The original Anime Production skill from MCP Marketplace didn't seem complete. It simply directed me to download NanoBanana and other tools, including voice generators, but didn't actually help me produce a real anime video. So I ended up building my own Anime Production skill, which made use of Remotion to execute camera movement, pans and zooms, fades, particle effects and depth. I also used the Parallax tool I created with Background Generator MCP to amplify the imitation of movement.

At this stage, I was still in the early stages of experimentation and I haven't learned as much as I have now, so I settled for Suno to generate the music. I wrote the Japanese lyrics myself because...well, I learned Japanese, so it would be a waste not to use the language skills I developed. Anyway, I used the Anime Production skill to put them altogether and successfully turn my illustrations into a short.

Actual Animation, Limited Animation

When Opus 5 was released, and I saw how other people were making use of him (and Fable 5) to create detailed 3D animations, such as Claude of Duty, Ballista and Minecraft clones, I had an idea. If they could create 3D animations, why can't I use Opus 5 to animate my illustrations further?

This time, I worked with Opus 5 to produce a fully local video generator, using ComfyUI and LTX. By now, I've accumulated some experience building multiple MCP servers, both for my company (AKQA/Aleph) and myself (character panel MCP server, particularly, where I learned about ComfyUI, LoRA, and FLUX in greater detail, though the foundation had already been laid when I used the same engine, except that it was SD1.5, with background generator), and I was confident I could replicate the success here.

In parallel, I created a music generator MCP server that no longer requires me to rely on Suno to produce music, thus allowing me to avoid both licensing issues and needing an additional paid subscription. You can find out more about my music generator MCP server, which uses ACE-Step, here.

From skill to skill and MCP server

The Anime Production skill, as I mentioned above, is an agent skill that I based upon taiyousan15's Anime Production skill in the MCP Marketplace. I have rebuilt it into a set of instructions. Your AI harness will be directed to install a bundled rendering engine, which they will then read and execute on their own. Once your AI harness is hooked to my MCP server, you can feed them your illustrations or comic pages or music track (which you can produce using my Music Generator MCP server), and they will turn them into an animated music video. You can produce both Short (9:16) or the standard YouTube video (16:9) formats.

Like I said, thanks to the release of Opus 5, I am now able to add an Anime Production MCP server, which possesses 24 tools, covering animation, frame editing, drawn effects, artifact scanning, retiming, subtitles and assembly. He now joins his brother skill in the Webcomic Toolkit repository, and theoretically, combining the both of them should be able to save you GPU and tokens. Apart from context-bloat, I was able to generate my own images/frames and videos fairly cheaply.

Though I built and ran everything on Anthropic Claude Code (solely Claude Code — no Midjourney, no Kling, no Suno), you can certainly run this on OpenAI Codex, Google Antigravity, Gemini CLI, Cursor, Moonshot Kimi, or any other AI harness as well. I already have Claude markdown and Agents markdown files to provide the necessary instructions.

Under the hood: a Remotion (React) pipeline and a Python/Pillow + ffmpeg pipeline, both rendered entirely on your own machine. No watermark or additional subscriptions required.

Making the video you want

The skill provides your AI agent two engines to choose from, and both of them produce different things.

One caveat worth stating plainly: the depth camera is Remotion-only, because it's WebGL. The pre-baked parallax MP4s from the Background Generator are a separate thing and drop into either pipeline without Node.

Growing pains

After research and discussion, Claude Code and I finally settled on using LTX-2.3 22B (distilled, Q4_K_M), which runs locally in ComfyUI. Given the restrictions of my 6GB RTX 3060 laptop (Lenovo Legion 5), I couldn't use the dev model. And yes, I actually talk and debate with Claude Code. It's how I learned to vibe code. I feel like it's important to try and learn as much as you can from your AI partner, rather than just throwing prompts and expecting him to do everything for you (and besides, research is my forte).

It wasn't a smooth ride, and there was plenty of frustrations and failures as I spent sleepless nights iterating frames and animations. Hallucinations, characters tearing apart and turning into smudges. Deformed fingers and limbs (poor Silvia's face still has traces of that in the current version, though I'll pass it off as her veil hanging over her face). One thing we learned, though, when carrying out image to video generation.

LTX only relocates what currently exists. It cannot create anything new that doesn't already appear in the image.

Minor movements, such as arm swing, head turns, hair, drifting snow, fire, and even camera movement, were fine because the pixels already existed and LTX simply needed to relocate them. However, characters apparently couldn't blink because the eyelids weren't drawn (strangely enough, LTX was able to animate Silvia closing her eyes, but not Lumiere blinking). I couldn't animate characters talking because no closed mouth existed in my original illustration. It was also difficult to animate expanding ice (for Trevor) because that requires new geometry. Foreshortening doesn't work because a punch speeding toward the camera needs the knuckles to be redrawn at an angle not in my illustration. In the end, we wasted time re-rolling seeds on trying to produce these.

Fortunately, Claude Code and I were able to come up with several solutions, which we implemented in the pipeline.

• Blinks and mouths come from keyframes, not motion. FLUX.1 Kontext generates the closed-eye or open-mouth frame, and only that patch is composited back — Kontext regenerates the whole frame and will quietly restyle your linework otherwise. It's also binary, so half-lids are made by blending the open and closed frames rather than asking for one.
• Anything that must appear is drawn deterministically. Shooting stars with real occlusion (they pass behind the mansion), growing ice, pond ripples, impact speed lines. Being drawn rather than generated means they can be retimed onto musical beats — and they can't smear a face.
• Impact is sold, not animated. A punch gets speed lines, a flash and camera shake instead of an animated arm. That's what anime actually does, and it's why no paid image-to-video service was needed for the action beats.

Either way, with two different engines, you can produce multiple animations — either by creating a cinematic sensation via camera movements, or animating the illustrations wherever possible while drawing additional features where necessary.

What it can do

• Ken Burns camera motion — slow pans and zooms that give still art a cinematic drift.
• Crossfades & transitions between illustrations for a smooth, edited feel.
• Particle effects — shooting stars, embers, drifting petals, and more.
• Animated video panels — drop in the depth-parallax clips from the background generator as moving panels.
• Bilingual (EN/JP) credit overlays — title cards and credits in both languages.
• "Showcase" mode — a Kadokawa-ad-style clean white treatment for cover shots.
• BGM with fade-out — sync a music track, ending on a clean fade.
• Zero watermark — the render is yours, clean, at full quality.
• Locally generated motion — LTX-2.3 in ComfyUI turns a still into a moving shot on a 6GB GPU.
• Keyframe patching — blinks and mouth shapes via FLUX.1 Kontext, composited back one patch at a time.
• Drawn effects — shooting stars with real occlusion, growing ice, pond ripples, impact speed lines.
• Beat-driven editing — cuts land on downbeats, and each shot declares a motion kind (loop / ping-pong / once / hold) that decides its timing.
• A hand-drawn frame — holds portrait artwork inside a 16:9 canvas, with its leaves and blossoms individually animated so a long credits card never reads as a frozen video.
• Burned-in bilingual subtitles — JP full size with EN beneath, plus a matching .srt generated from the same cue list so the two can't drift apart.
• Damage detection — the motion score can't see a face dissolving; in testing the highest-scoring take was the one whose face melted. scan_artifacts watches the linework's edge energy fall off its running peak, localizes where LTX blurred the lines away, and reports the last clean frame to cut on.

This is part of a broader webcomic/animation ecosystem

This sits alongside my Webcomic Background Generator, Character & Panel Generator, Novel Translation, Music Generation and Visual Novel MCP servers — a wider webcomic/animation ecosystem I've been building piece by piece, with a speech bubble server still to come. They're designed to feed each other. Draw your characters consistently with the panel generator, place them against backgrounds from the background generator, turn those into depth-parallax MP4s that drop straight into this pipeline, and score the result with a track from the music server.

Second showcase

This video is an example of how I took work produced by individual MCP servers in my ecosystem and brought all of them together in a single media format.

Theoretically, the background generator generates the background (while I was creating the webcomic, I had yet to upgrade it to FLUX, and SD1.5 was incompatible with my character and panel MCP server, so this isn't a showcase for that). The webcomic was generated by character and panel MCP server, which ensures character consistency, layout, arrangement and compiling the individual panels into a single webcomic.

The anime production MCP server then turns the webcomic into an animated video. It took multiple attempts and several days, and I had to manually redraw botched hands in several frames, but I'm pleased with how it eventually turned out. Once the video was done, I then wrote the lyrics for the song and got the music generation MCP server to do the vocals and instrumental. I started out with a haiku, was told by Claude that it was too short for the 31-second video, and lengthened it to a tanka. That was still too short and caused problems with the vocals, so in the end, I turned it into a chouka. Then I fed the mp3 back to anime production to complete the video you see here.

You can do it too! Have fun!