Microsoft Hololens, Augmented Reality as part of the Mixed Reality experience

Picture showing Microsoft HoloLens, used with permission from Microsoft

Augmented Reality or Mixed Reality?

Definition

Augmented Reality is a live direct or indirect view of a physical, real-world environment whose elements are “augmented” by computer-generated or extracted real-world sensory input such as sound, video, graphics or GPS data (Wikipedia). Therefore it’s a section of Mixed Reality (Image 1 “Reality-Virtuality-Continuum”).

A common example for an application of Augmented Reality would be the insertion of further information during the live broadcast of a soccer game. Offside lines or goal distances during a free kick are digitally shown on TV. The augmentation of every reality impression, together with all extra information, occurs live (in real time) and is able to respond to every human sense modality. However, the term “augmented reality” is often used in connection with a visually extended depiction and therefore functionally often reduced to extensions of only the visual perception of its users.

In everyday life, one’s auditory sense can also benefit of AR-functions. Voice announcements in public transports are most often automatized and computer-controlled. The software tracks down the recent location of the tramcar and based on this information, determines the next stop. As a result the passenger can hear a combination of different pieces of text and hence knows about the next stop and direction of the exit.

 

Requirements

In the following, typical requirements for a mobile AR solution based on image markers are listed.

Hardware requirements:

  • Sensors to capture real life surroundings and orient themselves in space (e.g. RGB Camera, GPS, Gyroscope)
  • Interfaces to register user inputs (e.g. Touchscreen, Microphone)
  • Computing power/ Processors to process and calculate data
  • Interfaces for the output of data and information (e.g. Display, Earphones, Vibration)


Software
requirements:
Technology for the purpose of …

  • Analysing camera pictures (Image analysis)
  • Analysing different sensor data (GPS, Acceleration Sensor, Gyrosope)
  • Linking existing data sources and resources (e.g. Database Interfaces etc.)
  • Managing user inputs and sequence control systems (z.B. Game-Engine)
  • Superimposing the camera input with virtual augmentations (Image Processing)

 

History

AR can be traced back to the 1960´s. Mobile Augmented Reality however, is relatively new, yet developing very quickly. The first AR-Browser-Apps named junaio, Layar and Wikitude have been developed between 2008 and 2010. With the appearance of the iPhone 3Gs, which had the necessary sensors and more computing power, mobile AR became interesting for mass market in 2009. Over the course of time, companies such as metaio, Total Immersion and Qualcomm established themselves as Software-Providers for Augmented Reality SDKs, which have been provided on a license model basis  to other developers. Due to the growth of these companies and software solutions respectively, other technology groups took them over. Therefore in October 2015, PTC bought QUALCOMM’s vuforia SDK for 65 mil USD. Shortly before that (05-2015), Apple acquired the company metaio and their technology portfolio. Almost contemporaneously, Microsoft announced new data headset, the HoloLenses. Forerunners to those are, among others, the data goggles by the company Vuzix and the Google Glass. Since mid-2015 two basic progresses in the technology sector for AR solutions have been developed. For one thing the development of already existing approaches of solutions for mobile “handheld” devices, such as Smartphone, Tablet and Phablet (mostly changes in software e.g. Apples ARKit or Googles ARCore, both becoming native AR-technologies of iOS and Android respectively) and for another the investigation and production of new AR-Output Devices (mostly smartglasses, data goggles or camera and sensor technology). Not only Microsoft HoloLens is trying to achieve those developments but also other companies, that run the projects “Meta” and “Magic Leap”.

Reality-Virtuality-Continuum

The R-V-Continuum (or Immersive Computing Spectrum) pictures the continuous transition of real environment (on the left) up to virtual environment (on the right) and gives every different section appropriate terms. Mixed Reality is the part of the R-V-Continuum that shows the range of the integral real and the whole virtual environment. Mixed Reality therefore comprises every section in which reality is mixed with virtuality (or vice versa). Augmented Reality is consequently part of Mixed Reality.

Reality-Virtuality-Continuum: From reality over augmented, mixed virtual reality to virtuality

Augmented Reality Applications and Examples

Where do Augmented Reality and Mixed Reality Solutions appear in every day life? How can Mixed Reality Headsets like the Microsoft HoloLens revolutionize the Human-Computer Interaction (HCI) in private but also professional environments in the future? In the following, some examples of AR/MR usecases and their advantages for various industries and business units will be pointed out.

Real Estate / Architecture / Interior Design

Augmented Reality architecture

Project for KWAG – AR App Leopold Carré – 2015

  • High Information rate due to interactive, animated depiction of planned or existing Real Estate as AR-Miniature Model
  • Intuitive access to the context of the building (neighbourhood development, existing objects, lighting and shadows, prominent properties of the object) due to the processing of information in the AR-Scene
  • Convincing presentation of construction project via pre-visualization of the buildings and their construction phases in life size in the foundation ditch
  • AR-Applications as tools for interactive planning of the interior design help the client’s imagination
  • and others…

Fairs / Exhibitions

Augmented Reality at trade shows, exhibitions and fairs

Project for OWA – AR App for exhibition booth – 2017

  • Maximum benefit from the exhibition area via virtual presentation of products while being shown in real size at the same time
  • Depiction of additional information on exhibits (Text, Images, Moving Images, 3D Animation) with linear or non-linear user guides via the AR-App
  • Cost savings due to reduced transport costs for exhibitors (products can be transported as virtual exhibits)
  • Uncomplicated clients approach via Augmented Reality trade-fair concepts (crowd-puller, Icebreaker, innovation feature) with smart devices or as kiosk system (digital signage)
  • and others…

Marketing / Augmented Reality Print

Augmented Reality in the Lamborghini Magazine

Project for Lamborghini. On behalf of TERRITORY – AR in Magazin – 2016

  • Increased information content of print via Augmented Reality Companion-Apps as an interactive addition to the printed format, also known as AR-Print or Interactive-Print (e.g. product catalogues, product packaging, flyers, mailshot, business cards, etc.)
  • Especially high-quality virtual processing of complex information (e.g. image brochures, annual reports, anniversary magazines, etc.)
  • Possibility to use Authoring-Systems to simplify production and administration of new contents
  • and others…

Finishing and Expansion of the product

Augmented Reality product extension

 Project for Columbus-Verlag – 4D Globus App – 2014

  • Smooth connection of timeless and high-quality product properties with innovative digital functions (e.g. Columbus-Verlag Globus-Weltkugel, 4D Globus App)
  • Successful enhancement of traditional board games via the clever combination of analogue and digital game mechanics (e.g. Ravensburger AR-Puzzle, Scotland Yard Master, etc.)
  • User-friendly combination of analogue and digital data stocks (e.g. the print of a magazine with digital media on the Internet, Web-to-Print Photobooks with video clips)
  • and others…

Training / Instructions / Education

Augmented Reality in training, instruction and education

Picture showing Microsoft HoloLens, used with permission from Microsoft

  • Simplified explanation of products, whose use and repair with the help of the corresponding AR-App (e.g. maintenance and service of electric devices, instructions for the construction of party games, etc.)
  • Optimized and individualized training of business people via AR Training Solution – independent from location, flexible concerning time and space, scalable
  • success in learning via interactive and positively emotional learning with AR Learning Applications as an innovative approach to communicate knowledge
  • and others…

Augmented Reality Medicine

Augmented Reality medicine

Picture showing Microsoft HoloLens, used with permission from Microsoft

  • Dentistry – Real time depiction of teeth reconstruction, while having freedom of acting due to free hands (AR-Goggles)
  • Surgery – Visual help via AR-Depiction leads to reduced error rates at surgeries
  • Training – detailed studies of the human body on a 3D, virtual basis (without threatening the life of a living patient)
  • and others…

Logistics / Warehouse

Augmented Reality in logistics and storage

Conzept of AR Vision-Picking – XPuls / www.tag-der-logistik.de

  • Optimized courses for warehouse and logistics workers, Vision-Picking makes paper or handheld-devices unnecessary
  • Important information on the product (weight, destination, characteristics, etc.) can be visualized for the supplier via Augmented Reality App in the context of the parcel
  • Clear depiction of the delivery address, Identification of receipt or individual delivery requests to the supplier
  • and others…

Productivity

Augmented Reality productivity

Picture showing Microsoft HoloLens, used with permission from Microsoft

  • Time and cost efficiency via Rapid-Prototyping with a virtual part overlapping on the real product (true to scale, easy to configure, short production phases, rapid iteration)
  • Decentralized cooperation , virtual collaboration as well as networking of locations and staff via Multi-User MR Applications (collaborative approach of AR and VR applications)
  • Translation-Apps with AR-Functions for a rapid and context related translation via picture recognition and smartphone
  • and others…