About...

Whacky Wizards

AI unity integration and Machine Learning Models

For one of my recent projects, I challenged myself to integrate two pretrained AI models into a small Unity game demo one for language interpretation and one for image recognition. I chose Vosk (speech to text) and Google’s MediaPipe (image recognition). Neither model has actively supported Unity plugins that are easy to access, so I used libraries created by alphacephei and homuler to bring them into Unity.

I wanted to make a quick demo showcasing these libraries, and Unity was the natural choice given my existing skills. The setup process for both libraries was a bit tricky; I had to dig through a lot of provided code to figure out the correct way to feed inputs into each model. After some trial, error, and code adjustments, both models worked exactly as I wanted allowing me to move on to building the actual game demo around them.

projectrole

Designer/programmer

what did i do?

Coding

Research

Implementation

How it all works

In the demo, the game features two spells: Shield and Fireball. You cast them by performing a specific hand gesture while speaking the spell’s name. Vosk handles speech recognition turning your spoken words into text while MediaPipe detects which gesture you performed. My code then checks both inputs against the list of available spells, and if the gesture and name match, the spell is cast.

A lot of my own code runs behind the scenes to validate spells and handle the game logic. The core idea was to create a kind of wizard simulator, where you duel other mages using your arsenal of spells. You need to cast at the right moment blocking incoming attacks while dealing damage to your opponent. The difficulty ramps up over time, until eventually you’re overwhelmed and lose.

Honestly, I didn’t expect MediaPipe and Vosk to be so accurate especially since I’d never used these libraries before. After some initial integration work and a lot of code reading, they worked flawlessly, letting me create an interesting and responsive game powered entirely by voice and gestures.

Get in Touch