Sony contributes to advanced research on user centric augmented reality [video]

Sony is committed to be at the forefront of augmented reality (AR) research, which is all about creating live physical views that are enhanced by computer-generated sensory input. This week, Günter Alce and Klas Hermodsson from Sony’s mobile division are participating in the insideAR 2012 conference in Munich, Germany. Günter and Klas are part of the VENTURI project, which is all about constructing augmented realities in new ways, where the available information is presented in a ‘user’ rather than a ‘device’ centric way, which often is the case. This actually requires a change in how we think of and develop user interaction interfaces. Read on for more information VENTURI and other augmented reality research concepts!
The VENTURI project is a joint augmented reality project among eight* European research institutions and businesses, including Sony. One of the goals with this project is to make augmented reality a booming technology, and this is something we’re very excited about at Sony. To achieve this, it’s important to understand how the AR experiences can be improved.
The concept of augmented reality – the visualised enhancement of the real-world environment through computer-generated sensory input – has been around for awhile, but mostly as demonstrations with small mock-ups in controlled spaces. And we’re already augmenting reality today by using mobile devices with an internet connection in order to get additional info about our surroundings. We’re starting to add visual augmentation into some devices, but we are only tapping into a small part of the available potential. And today, many existing examples and concepts showing augmented reality have so far been very device centric.
The VENTURI project seeks to prevent the potential flooding of augmented reality information from a device, where a user may only see visual feedback in the form of data on a smartphone screen. The project instead aims to make the user the focal point of the AR experience, blending audio, haptic, and visual data all within the context of the user’s environment. The project will also look at how to gather and filter data into to present relevant AR data to the user. For example, for a visitor exploring a city while using AR, the VENTURI system could obtain sensor data (video of visual landmarks, current GPS location, walking speed, direction) and deliver relevant information about nearby points of interests, and while incorporating related reviews, photos, or comments from the user’s social media contacts. The image above shows how researchers within the VENTURI project have created a model to try out different AR concepts.
Sony contributions to VENTURI – UI & interaction design, and context interpretation
Sony’s contributions in VENTURI (which stands for “immersiVe ENhancemenT of User-woRld Interactions”) fall under two areas – 1) user interface and interaction design, and 2) context sensing and interpretation. When it comes to research about user interface and interaction design within AR, our main focus is to investigate users’ expectations and reactions toward content presented though audio and visual technologies (such as gaze tracking, speech and gestures). The users’ feedback, including perceived levels acceptance, intrusion, stimulation, and engagement from VENTURI mock-ups and prototypes, will then be used to design an end-user interface that fulfils the users’ need for interaction, and which can be used as emerging standards for future AR devices.
Sony’s second research area in VENTURI, driven by Håkan Jonsson, is context sensing and interpretation which investigates how to exploit all of the available on-board hardware (and virtual internet) sensors to provide a comprehensive understanding of user context, from multiple points of view. This research looks at different types of sensors, including audio, video, GPS, digital compass, accelerometers, and how they can be used together to provide accurate measures where traditional sensor location sensors fail. The information measured by all available sensors could be used to infer what the user is doing, and what could be further experienced through other senses.  For instance, the accelerator and compass could be exploited to provide an estimate of the relative position and orientation of a user and environment. Combined with geographical data, this component could also be exploited to estimate the user’s probably activity patterns, such as running or walking.
Keys to making augmented reality a success – seamless persistence and pervasiveness
Making AR viable for the future requires a user-appropriate, contextually aware AR system; one that is seamlessly integrated with core AR technologies and applications on a state-of-the-art mobile platform. VENTURI aims to build upon contextual awareness, so that the AR system knows, for example, whether the user is at home or on the move; what type of user and level of activity the user is engaged in; proximity to smart objects or other users. Currently, mobile platform evolution is not driven by user expectations, nor is it driven by AR core technologies or applications. This leaves the user isolated from information and creates a poor sense of immersion. VENTURI intends to optimise and extend current and next generation mobile platforms to ensure device-to-user continuity.
Other AR concepts and projects
Within Sony, there are a several projects and concepts being worked on within augmented reality research. One concept we worked on previously is a master thesis concept called the Augmented Reality Visor, which aims at providing functional, cognitively sound, and socially acceptable gesture-based interaction for full peripheral view AR. The concept incorporated general interaction, using shortcuts, finding one’s way, and interacting with faraway objects. A concept video for it was awarded second place at the ISMAR 2011 Conference.

Another interesting project we’re working on now is a tool called “Wizard of Oz”, which is an ongoing master thesis project to be used for high fidelity user studies. This tool makes it possible to simulate the functionality of an AR system to assess its usability. It is named after the well known methods of John F. Kelley (and obviously also the movie The Wizard of Oz, where the wizard is pushing buttons behind a curtain).
insideAR conference
Together with our VENTURI partners, we will attend the insideAR conference 2012 in Munich, Germany, which shows off the latest advances in augmented reality, including the newest industrial installations, the latest mobile app developments, wearable computing prototypes, and futuristic projects straight from R&D labs. At the conference, Günter Alce and Klas Hermodsson will investigate users’ expectations and reactions on new ways of presenting and interacting with mobile AR games. insideAR also includes free workshops for app developers in creating their own mobile AR apps. Check out the insideAR website for more information!
Do you have questions or comments about augmented reality? Share your feedback with us in the section below.
 Special thanks to Günter Alce and Klas Hermodsson for contributing to this article.
* The partners and institutions that are part of the VENTURI project include Fondazione Bruno Kessler, Fraunhofer Heinrich Hertz Institute, STMicroelectronics, metaio, Inria, eDiamSistemas, ST Ericsson, and Sony.