PicotionAI: Playing Pictionary with mixed reality, camera access, and artificial intelligence

Today I want to tell you about a little but very cute project I’ve developed for Pico headsets. It’s called PicotionAI and it is a game that includes mixed reality, camera access, and artificial intelligence! As I have done in other similar cases, I’m telling you this story hoping that it can inspire you to do cool projects of your own. Let me tell you everything!

Pico(tionAI)

Let’s start from the “Pico” part of the name of the game: as you can guess, it is called this way because it is made to run on Pico headsets, to be exact on the Pico 4 Enterprise and Pico 4 Ultra Enterprise.

The game came to life when last year I was advocating for having camera access on MR headsets, especially the Meta Quest. A Pico representative commented on my LinkedIn post saying that actually, Pico allowed for camera access on its enterprise headsets in specific approved cases. He also invited people interested in using computer vision on MR devices for B2B uses to reach out to them. We started talking and in the end, I managed to get a Pico 4 Enterprise and play a bit with camera access. Even more, in the end, I had the opportunity to make a little demo involving camera access for the launch of the Pico 4 Ultra enterprise…

(Pico)tionAI

Let’s now speak about how the idea of the Pictionary game came out. I was so excited to have camera access that I immediately started playing out with it. I did a lot of experiments… even one that had to be run in the bathroom! Unluckily Pico people commented on this last one by saying “We can not bring people to the toilet to try out our headsets” (Well, actually I agree it would be weird if while you are using the urinal at a VR event venue, someone would approach you and ask “Would you like to have some fun with me and my Pico?”). For one of these demos, I thought that it could have been cool to use passthrough camera access, and possibly AI, to make a game. But I did not want to make something lame like a speaking AI agent or just let you take photos of the surroundings. I wanted to do something more original.

Suddenly a light bulb lit up in my head: I remembered the fun nights with my friends playing Pictionary. The fun was especially because I suck at drawing and they had every time a hard time guessing what I was representing. So I thought: my friends could not guess what I was drawing, but can the AI do that? The game would have been about the user drawing with pen and paper, like in the original game, and the AI friend trying to guess what he/she is drawing. Every round should have been short (around 1-2 minutes), and at its finish, the more words the AI has guessed, the more points the user has scored. The interface would have been in mixed reality, with the system showing the 3D model of the object to draw. In the end, there should have been a classification to trigger gamification mechanics and entice people to play back to be on the first spot.

I started working on the project with the people at VRROOM Studio (that helped me with the 3D models because if I suck at drawing, I suck even more at 3D modeling) and after some days of work (across airports and hotels during my trips), I managed to deliver the tech demo in time to be used in some Pico events.

Do you want to see what it looks like? Then watch me playing a match in the video below! And if you were wondering… unluckily the AI struggles at recognizing what I’m drawing exactly like my friends 🙁

Some lessons to share

Be original

Why am I sharing with you this project that I made some months ago? As I’ve said, because I want to inspire you. With Google promising camera access on Android XR, Meta close to releasing passthrough APIs, and Pico and Apple offering camera access for enterprise, I think it is important that you start considering this tool for your next mixed reality projects.

I think it’s also important to think about creative ways to use camera access, especially in combination with AI. So do not stick only with the ideas that you have seen a million times, try to think about something new. I think the hand-drawing analysis was a cool one I’ve found: it could be applied also in other ways: maybe in MR you can look at the handmade drawing of your children and give them life in 3D!

Understand the limitations of AI

I used OpenAI APIs to analyze the images taken from the headset. What I do in the application is grab a frame once in a while, send it to OpenAI, and get as a result what it sees drawn by the user. What I discovered is that depending on the proportion of the drawing, the thickness of the marker, and the lighting in the scene, the accuracy of the detection changed a lot. If the headset sees just the drawing and the lighting is good, the accuracy can be impressive, but if the drawing is small, the AI sometimes struggles even to understand easy drawings.

This has been a tech demo made in a few days of work (and just for fun, no money was involved), but if I had time and budget to make this game, I would for sure work to increase the accuracy of AI by using computer vision to segment the white piece of paper where the drawing is happening, normalize its lighting, and then eventually even use a custom AI model to understand hand drawing. For now, what I do, is that if the AI keeps failing to detect the drawing, I start giving it some hints about the word to guess (technically at Pictionary this would be considered cheating, but I’m Italian, and I have my own conception of rules…).

This is to say that just using camera frames with ChatGPT is not always giving you the results you hope for.

Trust the SDKs

Once I got the approval from Pico to use camera access in my application, it was rather easy to use it with the Pico SDK. I read the docs, analyzed the samples, and I was ready to go. I think Pico did a great job in this sense.

New technologies mean new opportunities

A few years ago, we at New Technology Walkers made one of the first games in mixed reality for a standalone headset. Last year, with PicotionAI, we were again one of the first to make a game (prototype) using camera access on a standalone headset. In the first case, we presented the game on stage at an HTC event, while this time it was showcased in a few Pico events.

New technologies open new doors that you can use to build relationships with brands, create a new market, or any way to show off yourself and your team. Now the trend is a mixed reality, and almost no one has cracked the MR gaming formula yet, AI is at the top of its hype, and passthrough access is coming. In a moment when the VR market is in a transition moment, these new technologies represent a new opportunity to succeed. My suggestion is to try to exploit them.

As I’ve said before, if you have an idea for an enterprise use of camera access, maybe with AI, reach out to Pico and try to develop it on the Pico 4 Ultra Enterprise. If you have a consumer idea, wait for the other headsets that are coming with passthrough access and then test it.

Where to try it?

If you are curious to try PicotionAI, unluckily you can not find it on any store. But if you are at some event in Europe or the US where Pico is showcasing the Pico 4 Ultra, you can ask them if they have installed it as a demo. If you see a bloc notes in the demo room, then for sure the demo is running. Try it and let me know if you had fun with it!

(And as usual, I’m open for collaborations. So in case you need help to develop some experience with AI and MR, or you want to put some money into developing Picotionai as a full game, feel free to reach out to me!)

Related