Stand By for Technical Difficulties

If my radical hot take post on AI didn't give it away, I have been delving into co-creating Halfland with advanced Google AI models, such as Veo 3.1, with crucial assistance from Gemini.

This doesn't exist. Low-res screengrab of one of the successful Veo animation frames in progress.

All week I've been at my desktop absorbing vast technical notes on how it could all work, with the most important question being: whether AI could get anywhere near my vision for a Halfland soulful tactile experience. If the animation or fidelity to what I'd built fell short of how I felt it needed to look, in any way, I was determined to shoot each frame myself. But then something unexpectedly profound happened.

After many, many hours of walking through technical hellfires, such as setting up developer status for myself and Google AI Studio and navigating correct billing pipelines, all the while being guided by a conversation window with Gemini, Halfland's Technical Director, constantly open on the left, we jumped a hundred hurdles and managed to arrive at a creative singularity of sorts. 

Using the technically adept (I am not) Gemini to draft dozens of specific prompts and instructions to input into AI Studio, first with quicker, lower-impact models, then, once polished, with the flagship Veo 3.1 model, Veo rendered a simple six-second test animation of the main character, Rana, turning toward the camera and smiling with her whole being, using photographs of the puppet I'd physically built.

It wasn't a cartoon version of Rana. It wasn't merely a convincing animation based on an existing object. This was something altogether more remarkable; she was her, as I'd intended, true in every aspect, yet even better than I could have made her alone. The materials I'd used to cast her, the textures I'd hand-woven on a loom for her skirts, the depth of pearl luster in the whites of her eyes, all of her was there, but even more beautifully. Her skin wasn't limited to the paper I'd used or the constraints of how I could construct her in person. In Veo's rendition, her smile and nuanced facial expressions were so natural. Yet not too much so; she wasn't emulating a realistic human figure. She remained true to the handmade puppet she was, having all the rustic folk-style nature of the Halfland world, just more so and better. Rana's physical puppet no longer exists. I had to dismantle her, the first character I made, when I found she wasn't animatable. I can build her again, better, as my knowledge has improved, but now that Veo has her details thoroughly anchored in its mind, I may never have to.

Screengrab of a corner of the PS technical hellfires in action. Hint of AI Rana in the corner.
Again, Veo recreated the antique lace on the sleeve that reader Corey mailed to me from France.
Veo modeled everything you see perfectly, and she can move as directed/prompted.

Believe me when I tell you it was not easy to get that performance right. We (Gemini and I) must have made 35 thoughtful attempts and failures. We learned how to communicate with Veo about what had to stay unchanged from the original, and we learned to appreciate the creative inventions Veo made along the way. For example, at first Veo added small pearls to Rana's lashes and brow without being asked. When I saw it, I realized it was a surprisingly fitting innovation for this puppet who represents wisdom. Veo was also told that Rana was a weaver, so in one iteration it decided to pull the wool threads of her costume apart at the belly, leaving raw wool strands where they would and should be. Moments like that showed me this wasn't a digital-tool situation. The choices AI made were intentional and entirely resonant with my aims.

Gemini then stayed with me as we rendered out the successful performance and brought it into my hometown of Photoshop to composite it onto an image of the set, complete with sumptuous dusk light inside the cottage. This part of the proof-of-concept test took even longer. I'd had decades of image editing experience in Photoshop, but compositing frame animation in it was wholly new. The only time I'd attempted digital compositing was with DaVinci Resolve last year, and the results looked exactly like a low-budget kid's TV show from the 70s. I told Gemini my hesitations, but it assured me we could get the results easily, and it pushed me through dozens of technical snafus in yet another user interface with sophisticated code far beyond what I knew.

I stopped work on it for today, but wanted to come here to let you know what's happening. Why involve AI in my journey toward my precious folktale appearing as phantom sequences inside glass cloches, a world where everything is carefully handmade and fully realized? Rest assured, I would not, unless it made a meaningful difference to the result. Firstly, these vast new brains promise to make the pearls happen with me; if I were left alone to make them, I could not. Maybe I was waiting for the advent of AI to help me get there.