CFan Academy of Sciences: Build a VR world that can walk freely

Many people mistakenly believe that VR must be three-dimensional, which is actually very wrong. Many VR game scenes made by artificial modeling, is indeed three-dimensional, and the player can also be “free” in which to move, but those through the VR camera live VR panoramic video, but the essence is still flat, the audience, of course, can not enter this two-dimensional world. Here we will talk about how to convert two-dimensional VR videos into three-dimensional scenes, and how the audience can move and visit them “freely”.

Why you can’t walk in VR panorama

When we put on glasses and enter the virtual space built by VR panoramic video (this article refers to VR video shot in real life), it is like being in a three-dimensional world, and we can freely look around the scenery up, down, left and right. But soon we will find that in this VR space “I” but difficult to move.

Why can we see freely but not walk freely? This is determined by the nature of VR panoramic video. When watching ordinary movies, no one has ever had the luxury of walking into the world of movies because we all know that movies are just a curtain, a two-dimensional plane, so of course we cannot walk in.

In fact, VR panoramic video is also essentially a two-dimensional flat “curtain”, but the difference is that it is not just a curtain, but is made of many pieces of curtains “stitched together” and surrounded by us, which is similar to the feeling we have when watching 360-degree ring screen movies ( Figure 1). So VR panoramic video produces a 360-degree immersion, but not a three-dimensional image.


Figure 1 Ring screen movie is an externalized VR panoramic video

VR photography has a special camera, usually a combination of 6 independent cameras (at least 2 cameras, more can be 9, 12, etc.), respectively, up, down, left, right, front and back in 6 directions at the same time to take pictures. What each camera captures is actually an ordinary video, and after “stitching” all the cameras together through post-production software, a highly immersive VR space is created (Figure 2 and Figure 3). As mentioned above, immersion is not three-dimensional, they are still essentially two-dimensional plane, so of course the audience can not enter it to walk, browse.


Figure 2 VR camera shooting from different angles of the scene at the same time


Figure 3 VR “stitches” 2D video from different angles into a whole

6DoF and spatial freedom

Before exploring how VR video can be reconstructed in 3D, let’s first understand a key concept – the degree of freedom. DoF is the abbreviation of Degree of Freedom, which is the number of variables in statistics, and the number of independent coordinates in the physical mechanics system. A three-dimensional space has a total of three right-angle axes, X, Y and Z, which are the three degrees of freedom. If an object in space can move without any constraint, it can do at least two kinds of motion, translation and rotation, along each axis. 2 times 3 equals 6, so the object has a total of 6 independent ways of motion in space, which is 6 degrees of freedom (6DoF), as illustrated in the following picture (Figure 4).


Figure 4 Three right-angle coordinate axes and two types of motion to build 6DoF (online motion picture example:

Using this concept to re-summarize what was mentioned before, that is, ordinary movies only have 1DoF, and the content of the movie screen we see is fixed no matter we move our position back and forth or left and right. VR panoramic video, on the other hand, has 3DoF, and when our eyes rotate along the X and Y axes (that is, up and down, left and right), we can see different picture contents.

360-degree light field camera realizes 6DoF space

In ordinary VR panoramic video, the viewer is given the ability to rotate the motion of 3DoF, so that the scene can be seen in different directions at a fixed point. However, ordinary VR panoramic video does not have the panning motion ability of 3DoF, so no matter how we walk, the image we see does not change, which is the main reason why we have no sense of movement in VR space.

Foreign manufacturers have introduced a 360-degree light field camera (Figure 5), which can produce 6Dof video with depth information.


Figure 5 Light field camera product diagram

Its principle is similar to geographic triangulation, because the light field camera is spherical, any point in space can be observed by multiple cameras, which creates many overlapping areas, thus maximizing the content of the recorded scene and obtaining the depth information of the subject through computing (Figure 6). When we watch the light field video, as we walk, this depth information is transformed into the corresponding image display, which gives us a real sense of movement.


Figure 6 The shooting principle of the light field camera

However, unlike the 3D modeling achieved through software in VR games, the light field camera is subject to the constraints of the physical lens, which records very limited three-dimensional information, when the audience’s movement beyond a certain range, because there is no more new information, it will lead to the picture no longer change, which means that the audience’s action is “blocked”, far from the VR The game is far from the “freedom”.

Position tracking and 6DoF 3D reconstruction

In the recent Oculus Connect5 conference, Oculus released the first fully independent operation of the 6DoF VR all-in-one Santa Quest, it can be tracked in thousands of square meters, with three-dimensional spatial modeling capabilities, VR and the real world as one.

Santa Quest uses Oculus Insight, their collective name for the technologies responsible for tracking and positioning, etc. Through the four ultra-wide angle lenses placed on the outside of the Santa Quest helmet, with special algorithms to track the user’s position in real time, in our 6DoF movement in VR space, real-time calculation and display of the corresponding scenario, so that we in In this virtual world we get a real sense of movement, so that we can walk in this VR world at will, browse different scenery (Figure 7).


Figure 7 Santa Quest builds a VR space that allows 6DoF motion

The important innovation of Santa Quest is not only the realization of 6DoF, but also its real-time 3D reconstruction of the scene, which can recognize the shape and even the material of various objects in the real scene, and then prevent realistic collisions by creating 3D point cloud maps that allow us to walk safely in the environment of VR (Figure 8).


Figure 8 Santa Quest for 3D reconstruction

The VR technology boom has passed, and now is a period of its smooth development. In the future it will not only bring us surprises, but more practical applications that will help life and deserve our attention at all times.

Leave a Comment