Meta’s support website now describes a “headset camera” permission for Quest headsets, suggesting the Passthrough API is launching soon.
What Is Passthrough Camera Access?
While headsets like Quest 3 and Apple Vision Pro use cameras to let you see the real world, today only the system software gets raw access to these cameras. Third-party developers can use passthrough as a background, sure, but they don’t actually get access to it. They instead get higher-level data derived by the system, such as hand and body skeletal coordinates, a 3D mesh of your environment with bounding boxes for furniture, and limited object tracking capabilities. That means they can’t run their own computer vision models, which severely limits the augmentation capabilities of these headsets.
The exception is that on visionOS 2, Apple is now giving enterprise companies raw access to Vision Pro’s passthrough cameras for non-public internal apps, but this requires a special licence from Apple and is restricted to “in a business setting only”.
Meta announced the Passthrough API at Connect 2024 back in September, saying that it will “enable all kinds of cutting-edge MR experiences”.
Now, just over five months later, a Meta support page just went live describing a new “headset camera” permission which “allows an app to access the real-time passthrough camera feed from the front of your headset”.
The page was first spotted by VR enthusiast Luna, and gives three examples of how Quest apps could leverage the passthrough view:
• Object recognition. Developers can create apps that recognise and use specific objects within your real-environment. For example, a digital board game that incorporates physical game pieces or boards.
• Location recognition. Developers can create experiences that respond differently depending on where the camera feed shows you are located . For example, indoors or outdoors, at a famous landmark, or in a specific type of room.
• Other machine learning functionality. Developers are able to run custom machine learning models against data from the real-time camera feed. This could be used for retexturing/shading, games involving participants who are not wearing headsets, person/animal detection, or any number of custom industrial/training use-cases.
Android XR Will Let Apps Access The Passthrough Camera View, Google Confirms
Android XR will let apps access the passthrough camera view to, for example, run custom computer vision models, Google confirmed.
Meanwhile, Google confirmed last month that Android XR, its upcoming OS set to debut in Samsung’s standalone headset, will let apps access the passthrough camera view at launch.
