VRVolVis

VR VolViz

This page describes various file formats, file conversion techniques and software that can be used to manipulate and render 3D volumes of data using volume-rendering techniques. Much of what is described here will easily work with single-scalar volumetric data, but challenges arise when there is a need for

• near-terabyte sized data,
• R,G&B volumetric (vector) data,
• rendering in real time for virtual reality (VR).

Recent Results

(Placed at the top for quick review)

Some recent development highlights:

• An OpenXR application that can render ANARI renderings in the HMD has been written
• Remote rendering of OpenXR applications to a Quest has been demonstrated
  • Using WiVRn
  • Tested with ANARI rendering, and it works!
  • (Still need a CUDA renderer on host with WiVRn server for reasonable rendering rate)
• At least one (perhaps two) ANARI backends now properly handle multiple overlapping volume rendering
  • The Visionaray (visionaray_cuda) ANARI backend has been tested — images below
  • The Barney ANARI backend has reportedly been fixed, but has not (yet) been confirmed — is next on the list of tests
• The red,green,blue test volumes have been replaced by actual 3D microscope data (skin) provided by the Mayo Clinic. This is a new dataset, but generally similar to the original (a bit larger).

Remote rendering

• ANARI's "remote" rendering backend feature has been tested, and presently works in some circumstances, but not yet for the volume rendering.
  • The author of the "remote" backend is currently refactoring the code, and my test application is being used as the Guinea Pig.
  • HOWEVER: As the ANARI SDK is not designed to load data remotely, this path is infeasible -- otherwise would need to ship entire dataset over network each time the server is run (which could take hours).
• WebXR: Began investigating the use of WebXR as a delivery means
  • HOWEVER: No browsers for Linux currently support WebXR
• WiVRn: Currently investigating the use of WiVRn as a tool to network OpenXR applications to phone-based headsets/HMDs, such as the Quest
  • Software has been installed on Quest-1
  • Have running WiVRN server on Linux Mint 21.3
    • • This did require an entire OS upgrade
      • Looking at how to build on larger-memory machines
  • Presently streams to local network
    • Need to test with ssh-tunneling
• CloudXR:
  • Am on the CloudXR testers list (communications with Greg Jones of NVIDIA)
    • Also met one of the project leads at SIGGRAPH (Arjun Dube)
  • ISSUE: CloudXR is presently just for OpenVR
    • OpenXR is in planning stages, I will investigate when it becomes available.
      • FYI: Jakob Bornecratz (formerly of Collabora, now NVIDIA) is working on this

Next on the list

• DONE: Finish the WiVRn build and test with OpenXR
• DONE: Develop an OpenXR-ANARI application
• Test the Barney ANARI backend for overlapping volume rendering
• XX: Continue working on testing/updating the "remote" rendering ANARI backend
  • NOTE: one concern is how to access data local to the remote machine for the volumes — otherwise the data would have to be sent through the socket connection which would be problematic for large datasets
    • And in general this is a breaking issue
    • However, for "regional" (i.e. limited) data, this remote rendering with ANARI could be acceptible
• POSTPONED: Target Date: August 7: Develop a WebVR interface suitable for viewing on Stand-alone & PC-VR systems
  • Shifted to use of WiVRn for streaming
  • Also todo: Explore the prospects of an ANARI-Web rendering interface
• Update the ANARI interface to ILLIXR, and add the Mayo Clinic volume rendering as an example application
• Work on new ANARI camera that can render a 360 image over the "remote" interface
• Explore multi-rendering for CAVE and multi-client usage
• Begin developing a user interface that allows the user to select "blobs" as nuclei, etc.
• PARTIAL: Work on an OpenXR interface
• Begin paper Introduction and Previous Work sections

Some pictures

• Toirt Samhliagh rendering of new data — red-channel only

• ANARI-VR rendering of new data — red-channel only

• ANARI-VR rendering of new data — three-channel

Data

Much of the experimental work described here is based on a volumetric dataset created by a 3D microscope, which produces real-color images stacked into the volume. That dataset is too large to provide for quick downloads, so an alternative source for example datasets is provided here (though most are the standard single-scalar type).

• Open SciVis Datasets

Software

Software that has been tested with this dataset (though often with some method of size reduction employed) include:

• ParaView
• ANARI
  • VisRTX ANARI backend
  • Visionaray ANARI backend
• VTKm & VTKm-graph
• ANARI Volume Viewer
• Barney mutli-threaded renderer
  • hayStack (Barney viewer)
  • Barney's "BANARI" ANARI backend
  • hayStack's "HANARI" ANARI backend
• PBRT code from Physically Based Rendering book
  • Ingo's reader for PBRT files

ParaView usage

hayStack usage

The hayStack application uses the multi-GPU Barney rendering library to display volumes with interactive controls of the opacity map. It is intended to be a simple application that serves as a proof-of-concept for the Barney renderer. There are a handful of command line options and runtime inputs to know: % ... where

• 4@ — ??
• -ndg — ??

Runtime keyboard inputs:

• ! — dump a screenshot
• C — output the camera coordinates to the terminal shell
• E — (perhaps) jump camera to edge of data
• T — dump the current transfer function as "hayMaker.xf"

Example output (RGB tests):

(Original example -- single channel)

Process

Python Data Manipulation Scripts

Step 1: Tiff to Tiff converter & extractor

This program converts the data from the original Tiff compression scheme to an LZW compression scheme, and at the same time can extract a subvolume and/or reduce the data samples of the volume.

Presently all parameters are hard-coded in the script:

• input_path
• reduce — sub-sample amount of the selected sub-volume
• startAt — beginning of the range to extract (along the X? axis)
• extracTo — end of the range to extract (along the X? axis)

Step 2: process_skin-<val>.py

This program reads the LZW-compressed Tiff file from step 1, and first extracts the R, G & B channels from the data. Using the RGB values, additional color attributes are calculated that can be used as scalar values that represent particular components of the full RGB color. Finally, the data is written to a VTK ".vti" 3D-Image file.

There are (presently) two versions of this file, the first ("process_skin.py") was hard-coded to the specific parameters of the early conversion tests. The second ("process_skin-e300.py") is being transitioned into one that can handle more "generic" (to a degree) inputs.

In the future, I will also be outputting "raw" numeric data for use with tools that only deal in the bare-bones data.

The current (hard-coded) parameters are:

• extract — the size of the original data (used to determine R,G,B spacing)
• gamma — the exponential curvature filter to apply to the data