Repository

Control of Virtual Objects by the User in Virtual Environments

  Modern technologies in the computer science field have changed and improved significantly the possibilities to interact with the computer without using the keyboard or mouse. Instead, a user could interact with the computer by doing specific movements with the head e.g. moving up and down, turn to the left and right, or by performing specific hand gestures e.g. align fingers accordingly to execute a specific operation. For example, such methods are implemented among other things in health care for people who have a handicap. A second application field is the engineering sector, where an engineer is able to move in a virtual environment to inspect a building and interact with virtual objects. Also, an important application field is a military, where a pilot is wearing a HUD and is able to target the enemy before he fires the weapons. To accomplish these tasks dedicated hardware is required.
  In this project a 3D camera from Intel gets connected via the USB port to catch each movement of the user’s head. A move consists of multiple frames which will be forwarded to the pc for further processing. In this work, only head movements will be handled. To simulate each operation of a performed head movement, a virtual environment gets developed and some virtual objects positioned to simulate the feeling of real movements. For example, if the user turns to the left, right, up or down the camera will turn accordingly in the virtual environment. If the user moves with his head towards the camera this gets translated into going forwards or if he moves away this gets translated into going backward. Overall, the following head movements are handled: turning to the left, right, up, down, moving towards, and backward. The following image shows the capturing and processing process of the application.
  As mentioned above to catch the head movement of the user the Intel RealSense 3D camera gets used and connected to the computer via the USB port. Intel provides additionally the Intel RealSense SDK to provide operations for the software developers to interact with the camera. The virtual environment gets developed using the programming language C++ and the open-source computer graphics library OpenGL. OpenGL provides operations to draw onto the screen either in 2D or in 3D. The environment gets embedded in the graphical user interface (GUI) to display some information like current movement operation, statistics about the 3D camera, and so on. For this purpose, the Qt library gets used which is a freely available cross-platform library for non-commercial applications. The library consists not only of a UI framework but provides also methods for networking, databases, etc. In addition, Qt gets shipped with its own IDE called Qt Creator to provide an easier way to develop applications consisting of user interfaces. The overall low-level drawing process gets executed onto the GPU by using shaders. Shaders are getting programmed in a C-like shading language called GLSL and their purpose is to handle the vertices and to color the processed shape accordingly. There are two main shader types that are required for each drawing operation on the GPU. The first one is the vertex shader which handles vertex operations like placing the vertices at the calculated positions, vertex interpolation, etc. The second one is the fragment shader which handles the color operations and among other things the lightening process. The shaders are executed in the order in a pipeline whereas the vertex shader is the first shader and is located at the beginning of the pipeline. There are very important additional shaders like geometry shaders, tessellation shaders, etc. which will not be used in this work. If you want to learn more about OpenGL and shader programming, I can highly recommend the Red Book and this awesome online tutorial.
  Below there are screenshots of the application and the hardware used. The green frames in the GPU Pipeline depict the shaders programmed in this work. The C++ code is published on GitHub. I have also provided UML class diagrams to show how they relate to each other, I have provided below the UML class diagrams of the application.

Screenshots of the application

UML Class Diagrams

Download (greek language, credentials of the Hellenic Open University required)
bsc_thesis_2017.pdf