With the rise of VTubers and other virtual characters, the desire to enter the virtual world has never been greater. In this article I will introduce mocopi: an easy and convenient system to enter the virual world. As a dancer, I was excited to see a cute anime girl replicating my movements.
What is mocopi?
Mocopi is a motion tracking system released by Sony in December 2022.
The system consists of:
- 6 lightweight sensors
- 5 bands and a clip to secure the sensors to the body
- A smartphone app
Since the system is small and light, it is easy to transport and set-up. With just 6 sensors, (on the head, left wrist, right wrist, hip, left ankle and right ankle) the whole body can be tracked in 3D using motion sensing and AI technology.
(1) Sensors
(2) Joint alignment
(3) Joint position
How does mocopi work?
The positions of the base joints (where the sensors are worn) are determined by integrating acceleration data and using an AI model to account for positional errors. The positions of intermediate joints with no sensors are predicted using AI models which have been trained on various human movements. Below is an image showing the skeleton used in mocopi.
What can mocopi do?
The primary feature of mocopi is mapping the motion of a user onto an avatar in real-time. There are two default avatars available in the app but your own avatar (VRM format) can be imported to the app. Videos can be filmed within the app, recording the avatar’s motion. Optional features include: displaying the avatar mirrored, having the camera’s position fixed and having the avatar’s hip placement fixed. The background is plain and the colour can be chosen by the user for aesthetic or editing purposes. Additionally, the avatar can lip sync according to the sound spoken into the mic. Several SDKs are available for download on the official website such as the motion Receiver Plugin which imports the motion data in real time to Unity or MotionBuilder and the BVH Sender which can send motion data (BVH format) to a local network via UDP.
Setup
A smartphone of at least android 11 or iOS 15.7.1 is required. Once the app mocopi is downloaded, the sensors can be paired to the mobile device via bluetooth (pairing is only required the first time). The sensors are attached to the body using straps and a clip. The user can input their height then callibration takes place by standing upright.
Performance
Below is a video of the application working in real time. The video on the right shows the video recorded within the app and the video on the left shows the corresponding movement of the user wearing the sensors. One of the default avatars: RAYNOS-chan was used.
The video below shows RAYNOS-chan’s mouth moving when words were spoken in real space.
Summary
Mocopi provides an easy and efficient way of entering the virtual world compared to previous motion tracking systems. The application worked close to real-time and kept up with fast and complex movements despite having little equipment. The use-cases for mocopi are endless. Examples I can think of at the top of my head are: creating videos for recreational purposes, turning into a virtual character, analysing motion data for athletes, saving movement data for synchronisation purposes, and tracking movement of construction workers. Personally, I would like to use mocopi to record dances I have learned using the mirrored option so that I can re-learn them later on.
References:
- Sony mocopi Official Homepage: https://www.sony.jp/mocopi/
- Sony official support page: https://www.sony.net/Products/mocopi-dev/jp/
コメントを残す