Monday, 4 May, 2020 UTC

Powerful 3D Experiences on Hololens using 3D Streaming Toolkit

Powerful 3D Experiences on Hololens using 3D Streaming Toolkit

Summary

Source
Hello Folks! I have been working in the field of VR and AR since the past 3 years and I recently started working on Hololens. Hololens, a smart glass manufactured by Microsoft has changed the way we view and interact with digital content. But there is a limit to what Hololens can show when it comes to high fidelity and high poly count content. So, Microsoft has come up with a solution to overcome this problem and it is called 3D Streaming Toolkit.
3D Streaming Toolkit performs computation on a server device which is mainly a windows machine that has a nvenc supported GPU and streams the frames to the hololens. So the entire computation is done on the server side and hololens merely becomes a viewer.
Source: Link
To explore more on 3D Streaming Toolkit, please refer to the official website:
3D Streaming Toolkit Documentation
https://medium.com/media/8a32568d0a96c5929ad42aa16f9a3bbf/href
In this story, I will brief the steps to use 3DStreamingToolKit libraries to stream heavy and dense content to Hololens and for achieving this I will be using a unity server application and Hololens directX client.
Further, for easy understanding of the audience and successfully have the 3D experience on Hololens, here is the list of things that I will be covering in this blog:-
  1. Prerequisites
  2. Download Project
  3. Install Dependencies
  4. Building the Server-Side Application
  5. Building the Hololens Client Application
  6. Launching Server and Client
To learn more about the basics of Hololens, my friend has written a story about it and can be found here
Prerequisites
  1. Latest Nvidia drivers with nvenc support
  2. Visual Studio 2017
  3. Windows 10 SDK 10.0.14393.795
  4. CUDA Toolkit 9.1
  5. Unity 2017.4.4f1 or greater
  6. Node js
Download Project
Please download the project from the below link using this command on git bash:-
git clone https://github.com/3DStreamingToolkit/3DStreamingToolkit.git
Link to the Github: https://github.com/3DStreamingToolkit/3DStreamingToolkit
Installing Dependencies
  • [Step 1] Open Windows PowerShell as administrator and set the execution policy to Unrestricted by running the command Set-ExecutionPolicy Unrestricted.
  • [Step 2] Download AzCopy from here.
  • [Step 3] Open command prompt and go the root folder i.e. 3DStreamingToolKit and run .\setup.cmd. This will download all the required libraries.
In the end, you will see the message Libraries retrieved and up to date.
Building the Server-Side Application
Building the libraries
  • [Step 1] Open 3DStreamingToolkit.sln in Visual Studio 2017.
  • [Step 2] Set the architecture to x64 and the mode to Release.
  • [Step 3] Click on Build Solution. This will build all the binaries and will copy them to the native unity project.

Trending AR VR Articles:

1. Five Augmented Reality Uses That Solve Real-Life Problems
2. Virtual Reality Headsets: What are the Options? Which is Right For You?
3. Augmented Reality Robotics
4. Infographic: The Future of Virtual Reality
Creating a Unity Package
  • [Step 1] Open the Unity-FractalStream project in the location 3DStreamingToolkit\Samples\Server\
  • [Step 2] Click on Assets in the menu bar and click on Export-Package and save it as a unity package.
Building Unity Server
  • [Step 1] Drag and drop the unity package that was created in the previous step into your unity project.
  • [Step 2] Add StereoCapableCameraHead prefab from the Prefabs folder to the scene.
  • [Step 3] Disable the existing MainCamera.
  • [Step 4] Set the position of the 3D model ahead by StereoCapableCameraHead by 3 i.e. z position = 3.
  • [Step 5] Select StereoCapableCameraHead. Drag and drop StereoCapableFractalCubes_PeerList from Assets/MutableState to Peer List field under Web RTC Server script under Inspector view of StereoCapableFractalCube prefab.
  • [Step 6] Expand StereoCapableFractalCube and select LeftEye. Under Inspector, click on Add Component and add Flip Y script. Under Mat RT field, add matRT material from Assets/Materials folder.
  • [Step 7] Select RightEye under StereoCapableCameraHead and repeat the same steps as above with Flip Y script.
  • [Step 8] Add MenuRoot prefab from the Prefabs folder to the scene.
  • [Step 9] Click on File and then select Build Settings.
  • [Step 10] Under Build Settings, add the current scene to the Scenes In Build. Set the platform to PC, Mac & Linux Standalone and set the architecture to x86_64 and Target Platform to Windows.
  • [Step 11] Click on Build
  • [Step 12] Go to the folder where the unity server application was built and open webrtcConfig.JSON
  • [Step 13] Set “port” to 3000 and set other settings accordingly. More details regarding the config file fields can be found here. For my sample, I am using a local network so I have set the “iceConfiguration” to none, “serverUri” to server system’s IP and “port” to 3000
Building the Hololens Client Application
  • [Step 1] Open StreamingDirectxClientHololens.sln under 3DStreamingToolkit\Samples\Client\DirectxUWP with Visual Studio 2017
  • [Step 2] Open the webrtcConfig.JSON file. Set the settings same as server-side webrtcConfig.JSON
  • [Step 3] Set the mode to Release and architecture to x86 and build and push it to the Hololens.
Launching Server and Client
Launching Signaling Server
  • [Step 1] Download Signalling Server -> git clone https://github.com/anastasiia-zolochevska/signaling-server.git
  • [Step 2] Open the command prompt and go to the signaling server root folder.
  • [Step 3] Run the commands. This will start the signaling server on port 3000
Command 1: npm install
Command 2: node ./server.js
Launching the Server
  • [Step 1] Open command prompt and cd to the folder where the unity server application was built and run the command:
<application_name.exe> -force-d3d11-no-singlethreaded
For my sample, the name of my application is 3dstk-server so my command is 3dstk-server.exe -force-d3d11-no-singlethreaded
  • [Step 2] Once launched, the screen would like this:
and signalling server output would look like this:
Launching Client
  • [Step 1] Launch the StreamingDirectXClientHololens on Hololens.
  • [Step 2] When successfully launched, the Hololens would be detected on the server window.
  • [Step 3] Press connect in server build. You will see the streamed content in Hololens.
That’s all about the basics of how we can have powerful 3D experiences on HoloLens using 3D Streaming Toolkit. If you made it till the end, thank you for sticking around for so long🙂. Please feel free to use the comments section to suggest modifications and/or additions.

Don’t forget to give us your 👏 !

https://medium.com/media/1e1f2ee7654748bb938735cbca6f0fd3/href
Powerful 3D Experiences on Hololens using 3D Streaming Toolkit was originally published in AR/VR Journey: Augmented & Virtual Reality Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.

... more @ arvrjourney

arvrjourney