Figure 1: The relationship between a Live2D character's face direction and its parameters. When the character looks up and to the right, the displayed values are X=10.99 and Y=9.00.
Following our previous post about a "touchable AI chat experience," you might be wondering how we display and control Live2D characters with AI in an iOS app.
How does Live2D animation work?
Unlike a video, which consists of complete images shown one frame at a time, Live2D animation starts with 2D artwork divided into parts—such as strands of hair, pupils, lips, and the head, hands, and feet. Parameters control how each part moves, allowing the character to move smoothly like an animated figure. Depending on the product, there are several ways to implement this. Two examples are Live2D Cubism, commonly used by VTubers, and Spine, which is used in some mobile games.
By controlling parameters, we can set values that make a character perform an action. In Figure 1, for example, the horizontal and vertical face-angle parameters are (10, 10) when the character looks toward the upper-right corner of the screen. Since the values map directly to the positions of model parts, smooth movement usually relies on a sequence of values along a curve, such as a Bézier curve, to prevent jitter between frames. In Live2D Cubism, parameter values can come from real-time tracking (for example, VTube Studio, which VTubers commonly use), pre-recorded motions, or expression files. Using an AI or ML model to generate or combine motion parameters and control a Live2D character is therefore a natural and practical idea. There are already projects such as the AI VTuber Neuro-sama and the desktop Open LLM Vtuber framework.
With the concept in place, surely implementing it in amiverse was as simple as borrowing some ideas and documentation, putting a polished VTuber avatar in the app, and setting a few parameters, right? The gap between that fantasy and reality became clear as soon as we started building...
The less glamorous engineering challenges
On the engineering side, we had to figure out how to integrate the native Live2D Cubism SDK framework into a modern Swift-based iOS workflow, adjust the Metal rendering pipeline to avoid glitches and rendering failures, and wrap code that mixes Objective-C, C, and C++. The list goes on.
In my limited experience, there probably aren't many teams building this kind of app with native iOS, Swift, and a C++ framework. That meant the hardest part was often the lack of prior examples to learn from. Even AI tools couldn't easily find complete, working, and sensible examples. We had to go back to first principles: research, collect information, experiment, analyze, and test again and again.
Once the controls were working, the real challenge was giving the character a soul. Our current text-tag approach asks an AI model to analyze the meaning and emotion of each conversation, then maps the result to predefined Live2D controls. These can play a motion file, apply an expression with a transition, or perform more conventional model transformations such as translating, scaling, and rotating. Behind every successful blink and motion in amiverse are countless app crashes in Xcode.
So... why build this?
After all that engineering, our goal is to create a more polished and lively chat experience. When you tap the screen, the character looks at you and reacts. When you talk about something that's bothering you at school or work, the character responds with an expressive look. We want amiverse to be more than a chat tool: a personal space that can respond with warmth and emotion. We believe that's the greatest value amiverse can offer.
We're exploring more ways for users to interact with characters, such as custom animations driven by facial tracking and real-time generation of motion curves. We're also continuing to improve how accurately AI expresses itself through animation. As a team smaller than a nanoteam—an "attoteam," perhaps—we'd love to hear from real users. If a touchable AI chat experience or VTuber-style AI chat sounds interesting, try downloading amiverse from the App Store. If you have any feedback, reach us on Instagram or Threads.
Thank you!