Use of AR technology

Use of AR technology
AR technology is one of the technologies that has rapidly become widespread in recent years. This time, I would like to introduce an article that introduced AR about the attempt at Pokemon Go that has spread worldwide. As an example, AR is currently used in many fields, so it will become much more convenient.

[Anyone can understand] AR (Augmented Reality) mechanism / technology
Update: 2017.10.04

>> Become a "required human resource" who will not lose his job even after 10 years

Click here for limited materials that will give you an advantage in changing jobs just by reading

Are you interested in AR (Augmented Reality) mechanisms and technologies?

There may be some people who are pinpointed, "Isn't it a technology in which a character pops out of the screen when viewed with a smartphone camera?"

However, I think many people have questions such as "I don't know how AR is made."
So, this time, I will explain what kind of mechanism AR is.

In addition to familiar cases such as the definition of AR and "Pokemon GO", we will introduce the back side of the mechanism from those that use moving images of the camera to those that use location information.

After reading this article, you will have a complete picture of how AR works and technology.

Contents of this article [Close]

1 What is AR (Augmented Reality)?
2 2 major information required for AR
2.1 Surrounding environment
2.2 Current position
3 There are two types of AR mechanisms
3.1 Vision-based type
3.2 Location-based type
4 Vision-based AR mechanism
4.1 Marker type
4.2 Markerless type
4.3 Advantages and disadvantages with and without markers
5 Location-based AR mechanism
5.1 GPS type
6 Comparison of 2 types of AR technology
6.1 Advantages and disadvantages of vision-based type
6.2 Advantages and disadvantages of location-based type
7 Summary
What is AR (Augmented Reality)?

Source: Photo AC

AR is an abbreviation for Augmented Reality, which means "augmented reality".

As the word "augmented reality" implies, AR is a technology that "expands the real world" and adds information and pictures to the real space.

While VR, which has been a hot topic in recent years, is immersive in the "virtual reality" created by various methods such as 3DCG and 360-degree cameras, AR is characterized by displaying and using different information in the real world. is.

A common example is to display GPS-based data and character CG on top of a video or photo taken by a smartphone camera.

One of the familiar cases related to AR is the Nintendo 3DS AR card.

Source: Nintendo Kids Space "Why do characters pop out of AR cards?"

The AR card is a set of 6 cards that comes with the purchase of the Nintendo 3DS series main unit, and you can play using the camera function by using the software called "AR Games" that is pre-installed on 3DS.

If you place the AR card anywhere and project the card in "AR Games", the CG will appear as if Mario and other characters were really in the scenery captured by the camera.

Source: "Pokémon GO" official website

"Pokemon GO" is also gaining worldwide popularity as a smartphone location-based game app equipped with AR mode.

The AR mode of "Pokemon GO" can be used in the scene of capturing Pokemon and in gym battles. You can switch the AR mode on and off by pressing the AR button at the top right of the screen.

When turned on, you can catch Pokemon against the background of real space, or form a party with 6 animals and perform a gym battle.

Source: Pokémon GO – Get Up and Go!

Another feature of the AR is that it adds information on the location of Pokestops and the appearance of Pokemon to the map of the real world.

Two major information required for AR

Source: Photo AC

AR is currently used in many applications for smartphones.

It is important to acquire two major information, "environment" and "current position", in order to superimpose images and characters on the real world and realize behavior linked to the current position with the AR application.

Surrounding environment
Alignment is indispensable when overlaying another video, image, CG, etc. in real space using AR.

Precise position recognition is required to superimpose images and information according to the current position on the real world without discomfort.

present location
With the AR app, it is possible to display information on the screen or change the operation of the app according to the current position of the device such as a smartphone.

Unfortunately, all services ended on January 22, 2014, and "Sekai Camera" provided by Tab, Inc. (* now renamed to tab, Inc.) is a representative example of an AR application that utilizes the current location. is.

"Sekai Camera" runs on iOS and Android, and users can post messages and images called air tags to the real space displayed through the camera installed in the terminal.

CEATEC2009 Sekai Camera

Its foresight is still highly regarded worldwide.

"(Omitted) The point is that no one really wants to search. And since searching searches for things that you know, it doesn't lead to the discovery of the unknown world. Such a connection with the unknown. The concept of Sekai Camera is to express it with tags, and the keyword is "tagging the world." The point is to tag everything in the world and connect related things. Tonchi Dot We created a new company called, and started development in 2007, and released it worldwide in 2009. There is a part where the iPhone just came out and it became easier to do, but 4 million downloads worldwide, at that time all over the world You got an investment offer from. "
--Source: Interview with Mr. Takahito Iguchi of Sekai Camera "The Japanese are the most dissatisfied with the Japanese" (Daily News Online)

When displaying the air tag, it is important to acquire the user's location information.

"Sekai Camera" uses the GPS built into the smartphone and an electronic compass to recognize the current position and the orientation of the camera on the smartphone, and the air tag attached to that direction is displayed on the screen.

Free career consultation online! We will not make any unreasonable solicitations

Tech Camp Engineer Career Change is a school that allows you to change jobs from inexperienced engineers and web designers. Tech camp allows you to learn programming online. We also provide "thorough support" and "learning environment to complete" online. The success rate for changing jobs remains at 98.7%, the same level as before Corona! If you have a career problem, why not get free online counseling (career consultation) from Tech Camp? You can make your own career plan. Online counseling is easy, so please use it.

There are two types of AR mechanisms

Source: Photo AC

There are two types of AR, a location-based type that displays information linked to location information such as GPS, and a vision-based type that uses technologies related to image recognition and space recognition.

These two types of technology do not interfere with each other and can be used together. By using it together, it is possible to create a higher quality AR application.

Vision-based type
Vision-based AR is also known as image recognition.

It is characterized by analyzing the environment in front of us and presenting information using technologies related to image recognition and spatial recognition.

A general example of a vision-based type is a marker-type AR that makes the terminal camera recognize an AR marker such as a square and displays information.

There is also a method called markerless type that recognizes the environment itself such as objects, buildings, and mountains that exist in real space without using a specific AR marker, and displays information.

The markerless type is also classified as a type of vision-based AR.

Location-based type
Location-based AR refers to a method of displaying information by associating it with location information acquired by GPS or the like.

In addition to GPS position information, the acceleration sensor measures the orientation of the terminal, and the magnetic sensor acquires the tilt to determine which position on the screen the information should be presented more precisely.

Hashtag for this article

#AR
Vision-based AR mechanism
Vision-based AR can be classified into two patterns, marker type and markerless type. We will introduce the mechanism for each.

Marker type
In marker-type AR, information is displayed by making the camera of the terminal recognize a figure with a fixed shape called a marker. The shape used for the marker is a square with a black border.

The marker is detected in the black frame, and the marker is identified by the pattern inside the frame. Color or white can be used for the pattern in the black frame.

When designing the marker, care must be taken not to make it a point-symmetrical figure.
If you use a point-symmetrical shape, you will not be able to recognize the orientation of the marker correctly.

ARToolKit is a typical software used for marker-type AR application development.

ARToolKit is a library for C language developed by Professor Hirokazu Kato of Nara Institute of Science and Technology, and the first demo was released at SIGGRAPH in 1999.

Source: ARToolKit.org

Since it is open source, it is also a feature of ARToolKit that derivative software such as NyARToolKit is appearing one after another.

When ARToolKit detects a marker of marker type AR, threshold processing is performed first. By performing threshold processing on the image obtained from the camera, it is converted to a black-and-white image and then black-and-white is inverted.

Then, delete the border, cut out the inner area, compare the cut out rectangle with the value of the pattern file (* the pattern of the inner area of ​​the frame is converted into data), and calculate the match rate to make the marker. It will be detected correctly.

In image composition, the outline and 4 corners of the marker acquired by the threshold processing are detected.

By matching the coordinate system of the marker calculated by contour extraction and the coordinate system of the image and characters to be displayed that have been decided in advance, the information is displayed on the marker in an accurately aligned state.

Also, if the image contains a certain amount of fine patterns and feature points, you can obtain the natural feature points of the plane by performing NFT (Natural Feature Tracking), and you can use your favorite images such as landscape photos and illustrations. It can be used as a marker. ARToolKit also supports NFT tracking.

Source: I tried ARToolKit's Natural Feature Tracking (NFT).

NFT treats any image as a marker by identifying feature points in the image.

On the other hand, NFT is sometimes classified as a markerless type in the sense that it does not use markers with a square black frame.

Source: Nintendo Kids Space "Why do characters pop out of AR cards?"

An example of AR that uses a marker type is the Nintendo 3DS AR card.

As a marker on the surface of the AR card, a box, Mario, Zelda, etc. are printed inside the black frame.

Markerless type
In markerless AR, corner points and feature points such as the corners of desks and windows in the real space currently being photographed are used instead of the feature points of marker AR, and additional information is displayed to match the terrain. ..

Therefore, the markerless type is real