Super Bionic Bash is the most recent development in the games catalog at Limbitless Solutions, all of which provide engaging ways to train Limbitless' Bionic Kids in the use of their new bionic arms. Super Bionic Bash is a 4-player party game that cycles through board game rounds and minigames. These various minigames use electromyography and an inertial measurement unit (IMU) to provide unique input to the game.
This game is an Epic MegaGrants recipient.
As I joined the project while it was still very early in its development, I was tasked with building out much of the game's core infrastructure. The goal was to make the project open to other developers to contribute to. Other programmers or even entire other teams should easily be able to go in and use pre-made building blocks to construct a new minigame.
To further this goal, I created a C++ minigame gamemode base class that inherits from AGameModeBase and handles much of the common functionality between minigames. For example, the minigame base handles player spawning, keeps track of minigame duration, has a points/player ranking system, and supports splitscreen and a pre-game "instructions" mode.
This class provides an easy-to-use and highly customizable layer of abstraction for a lot of helpful tools for minigame creation. All of this behavior is fully customizable in the Unreal Editor, making it easy for designers to tweak and customize their levels without even touching code. Designers or other programmers can also extend the Minigame Base class in either blueprints or C++ to define the rules of their minigame and customize even further.
The base class provides a helpful set of public functions and multicast delegates to allow other classes to hook into existing systems and modify them when needed. One way this is currently used a lot in the project is through event-driven minigame UI. This UI simply acts as an observer and reacts to game state when it happens, which is both performant because it's not needlessly updating UI every frame and it's good game architecture because it removes the need for the core game logic to know about UI.
This is an example of one such delegate that is being used to drive UI in Blueprints for designer use.
The Minigame Base class works closely with the Minigame Player base class. This minigame player class is set up to interface nicely with the Minigame Base class and also comes equipped with a Flex Device component. This component represents the physical flex device that is used by our players, providing an easy way to access flex and motion sensor data in minigames. This component abstracts away all of the BLE and signal processing magic from the designer in a nice, simple package.
For example, here's the flex component (called "flex") in action to drive a shooting range minigame where the player can aim on screen using the orientation of the flex device.
As mentioned before, the flex controllers we use at Limbitless are equipped with motion sensors. These sensors measure the proper acceleration and the angular velocity of the device. However, no prior game at the organization has used these before and there was very little documentation or previous code to make use of the motion sensors. My supervisors tasked me to simply "make motion controls work" and left everything else up to me.
With the problem being rather vague, the first thing I did was do some research. I looked into how motion sensors worked and the physics behind them to get a better understanding of what the sensor readings meant. For example, the accelerometer measures proper acceleration, which is acceleration from an inertial observer. This means that, when the device is at rest on a table, it will read acceleration due to the normal force of the table pushing up!
I also researched a lot about how a gyroscope works, its limitations, and how sensor fusion can be used to sense the environment with more accuracy than one sensor on its own.
I also did a deep dive into the mathematics related to how computers describe rotations. I delved into everything from euler angles, rotation matrices, and quaternions. Ultimately, I decided that quaternions would be the best option to describe the orientation of the motion sensor. Quaternions solved a lot of problems that plagued euler angles, seemed easier to work with than rotation matrices, and were already common in the Unreal ecosystem.
Quaternion math was always a topic that I thought was unapproachable and scary. I've always heard them described as something that's helpful but too esoteric for mere mortals. I soon found out that they really aren't bad, though. I learned that quaternions are essentially just 4 numbers that describe an angle and an axis. They can be composed together via right multiplication and can be used to rotate vectors using a process called conjugation.
My math exploration led me to deriving a formula which calculates the power of a quaternion, effectively scaling the rotation described by it. This turns out to be very similar to slerp, or spherical interpolation, but it was still fun to derive and implement a similar solution.
With a full understanding of the physics and math behind rotations, I was now fully equipped to implement motion controls.
The first step is to process the raw data received by the IMU, which are unsigned 8 bit integers. This requires some casting and scaling to get the data into the right format, which is gforce for acceleration and degrees per second for angular velocity. Some sign changes are also required to change from the IMU's right-handed coordinate system to Unreal's left-handed coordinate system.
Then, the angular velocity is integrated to approximate the angular displacement travelled in the last frame. This is applied to the current rotation of the device. This method of calculating orientation has a lot of error, however, and desyncs after just a few seconds of rotation.
This is where sensor fusion saves the day. The orientation is also calculated from the gravity vector as read by the accelerometer, which is reliable over the long-term but not the short-term. Applying a small amount of this rotation will ensure the calculate orientation stays accurate over long periods of time while preventing the accelerometer from overpowering gyroscope in short-term bursts of rotation.
There are still some limitations with this system. Since the device is not equipped with a magnetometer, it will drift along the global Z-axis over time. Orientation along the Z-axis cannot be measured from accelerometer's gravity reading like is possible with the X- and Y-axes. However, integrating the angular velocity read from the gyroscope is good enough for short-term use. For long-term accuracy, a magnetometer would be used to find the device's bearings from Earth's magnetic field.
One solution to this problem is to have the player constantly reset by having them hold the device in a certain orientation (e.g. have them hold it straight in front of them and call that the new "zero"). Many games that rely on motion controls do this, but often do this in ways that are well-hidden and don't seem obvious to the player as a recalibration.