VRVolVis

VR VolViz

This page describes various file formats, file conversion techniques and software that can be used to manipulate and render 3D volumes of data using volume-rendering techniques. Much of what is described here will easily work with single-scalar volumetric data, but challenges arise when there is a need for

• near-terabyte sized data,
• R,G&B volumetric (vector) data,
• rendering in real time for virtual reality (VR).

Recent Results (Spring 2025)

(Placed at the top for quick review)

IMMERSE Symposium Poster/Demo Abstract

• Title: Streaming Large data volume rendering to a low-cost HMD

This demonstration showcases an immersive visualization of a        
large-scale 3D microscopy volume dataset to a low-cost consumer HMD.        
The challenge of interactively rendering data on the order of 100's        
of gigabytes to an all-in-one HMD requires fast rendering, video        
compression and streaming, and an HMD interface to present the stream.        
To meet this challenge, several sub-systems are linked to provide        
the end-user experience.        

Systems include volume rendering on a high-end GPU, making use of        
the ANARI rendering standard.  The ANARI standard allows applications        
to choose from a collection of available rendering "backends" -- for        
this effort, a path-tracing renderer capable of multi-channel volume        
rendering to an omnidirectional camera is used ("Visionaray").        
The rendered image is then compressed and streamed using the FFmpeg        
library, which streams the video to a Janus WebRTC server.  Janus        
provides a server-based system where clients can connect and request        
access to various media sources.

The next requirement is for the low-cost HMD, such as a Quest headset
from Meta, to access and present the streamed data.  The nature of the
rendering process and image transmission ultimately leads to delays
in presenting the visuals to the user, which is a debilitating concern
for immersive visualization.  To mitigate this issue, the data is
rendered with a virtual omnidirectional camera, which is then
re-projected onto the inside of a sphere in the HMD.  This method
allows the user to quickly rotate their head without noticing any
lag as a result of presenting the rendering.  A browser-based WebXR
application connects to the Janus server and presents the video stream
to the user on the spherical surface.  It also provides a simple user
interface for the user to control the rendering.

Unsurprisingly, there are trade-offs for generating the images on one
system, compressing it streaming it -- the quality of the rendering
is generally negatively impacted.  The quality is reduced through the
addition of compression artifacts to keep network speeds high, but
furthermore, it is negatively affected by spreading the pixel
resolution over an entire sphere, even when the user can only look
at a portion of the sphere.  The other limited factor introduced by
spherical rendering is that while user head rotations are instantly
handled (reducing motion-to-photon latency), user head translation
movement will conversely increase the motion-to-photon response.

Our current demonstration shows a single channel view of data collected
though 3D microscopy.  A user interface in the HMD will demonstrate
some controls over the rendering parameter.

This framework will serve for testing additional features that are
important to the task of reviewing 3D microscopy data.  In particular,
the ability to annotate features within the data will provide
pathologists the means to collect specific data that can be analyzed.
We also will be adding a collaboration feature allowing users to
directly work together in the immersive space.  Beyond these additional
features, we will be exploring ways to mitigate the latency and
resolution issues, such as limiting the rendering and transmission
to a portion of the sphere.

WebXR application

A Multi-component application that renders on an RTX-GPU computer using an ANARI Backend (presently visionaray_cuda) using an omnidirectional camera. The rendered images are streamed using FFmpeg to compress and output as an RTP stream. A Janus server receives the RTP stream, and relays it out as a WebRTC communication stream that can be read by web browsers.

A Javascript web application using the Three.js library and the WebXR protocol receives the WebRTC stream and applies is as a video texture to the inside of a sphere with the viewer placed in the center.

Presently some very basic controls are available through the web interface: increasing and decreasing the position of the peak of opacity in the transfer function. These commands are relayed back to the rendering application through websockets (and a special combination HTTP/WebSocket server).

Currently, this all works within the router of my home, where I can more easily open socket ports. For testing purposes, a version is available at Video version on FreeVR.org. But this only plays a pre-recorded video, so the opacity buttons have no effect.

Next step is to build the infrastructure on a UIUC system where ports are more available. This process has begun on arches.cs.illinois.edu.

Some todo items

• Build on arches
• Improve the streaming quality
• Add more transfer function controls to the WebXR application
• Add the ability to add a simple annotation within the WebXR application

Recent Results (Spring 2025)

WebXR on Linux

• WebXR Linux Build instructions
• No updates this week

Streaming Volume Rendering to WebXR

• A/V Streaming to WebRTC
• Wrote Wiki documentation on the steps for streaming FFmpeg to Web
• Converted code from Vrui FFmpeg streaming to a separate "library"
  • (more of a proto-library)
• Integrated new FFmpeg-streaming "library" into ANARI-based renderer
• Tested ANARI-based rendering on remote (1000 miles) computer streamed to home

Papers

• Started collection ideas/citations for a Dual-Viewer CAVE paper
• Preliminary work on large-scale 3D volume rendering for VR paper

Other

• My RTX-4090 computer arrives today from NIST!

Plans for Spring 2025 Semester

Travel this semester

• Installation of new LED CAVE at NIST (late March/early April)
• IEEE VR 2025, plus visits to 5 other VR labs (March 4-14)
  • LRZ (Munich), HLRS (Stuttgart), U-Cologne, RWTH (Aachen), Inria Center (Rennes France)

Upcoming conference paper opportunities

• ISMAR (https://ieeeismar.org) – May
• LDAV (https://ldav.io) – June
• VRST (https://vrst.hosting.acm.org) – June
• IEEE VR – Sept/Oct

Tasks this semester

• PARTIAL: Investigate compiling Chromium for WebXR/OpenXR on Linux
  • I have created a Wiki page with build instructions
    • Not yet fully functioning
  • There is a Github page with a patch to do this -- [1]
  • I've already done a basic Chromium build, and will be testing the patch for OpenXR soon

• BEGUN: Retool the "Toirt Samhlaigh" volume visualization techniques
  • presently has the best quality volume visualization
  • will open up access to more HMD displays
  • can improve the streaming rendering

• Explore 360 rendering
  • use ANARI
  • may require adapting path-tracing rendering
  • can improve the streaming rendering

• Complete a tool that will allow visualization of 3D-microscope data to lower-cost HMDs, with collaboration
  • choose from the tested technologies
  • implement on an open machine (ie. w/o government port access restrictions)
  • complete paper

• (for NIST) Adapt existing immersive visualization software to operate with multiple head-tracked viewers
  • needed for the new LED CAVE
  • can serve as the basis for a paper

• (Perhaps) Investigate creation of an OpenXR runtime for CAVE-style VR
  • not necessarily beneficial to Mayo Clinic project
  • more something of interest to NIST (and other users of CAVE VR)
  • might be worth a systems paper

• (Perhaps) coursework development for UIUC VR curriculum
  • This semester, or summer?

Recent Results (Fall 2024)

(Placed at the top for quick review)

Upcoming Conferences

• Preparing for two upcoming conferences
  • CAAV — November 11-14: Presenting on ParaView
  • SC'24 — November 16-22: Bringing IQ-station to demo Immersive ParaView

Streaming Volume Rendering to WebXR

• Developing a prototype of a streaming platform from Toirt Samhlaigh volume visualizer to WebXR
  • Challenges (11/07/24):
    • Janus configuration and web interface has proven to be more difficult than anticipated
      • SRTP was failing to connect stream to Janus server — switched to RTP stream
      • Managed to get an audio stream to work
      • Have not yet been able to get video stream to work
    • Reconsidering whether this is the appropriate technology

  • Use the existing and working Toirt Samhlaigh immersive volume rendering tool
    • Built on the Vrui VR infrastructure
      • Existing collaboration feature
      • Prototyping export to RSTP stream using FFmpeg API
      • Will need to read head tracking data back from RSTP server
      • DONE: Prototype code that taps into the HMD rendering prior to warping
  • TESTING: use Janus RSTP to WebRTX gateway server
    • DONE: Built and configuring
    • IN-PROGRESS: Create a WebRTC app that reads the stream
  • WebXR browser interface to user
    • PROTOTYPING: Three.JS program
      • Read/Render 360 image stream from WebRTC
      • Provide head-tracking back to Toirt Samhlaigh via Janus

NOTE: the stripped boxes are all "in-progress" (prototyping) stage

Some recent development highlights:

• PARTIAL: Reverse Engineering the Volume Rendering algorithm in Toirt Samhlaigh
  • I've been delving into the Toirt Samhlaigh code that does the volume rendering
  • Adding comments as I understand it more
  • Have extracted the primary vertex and fragment shaders
  • Now building a small test program that distills the volume rendering
  • TODO: Learn how to transfer the volume rendering techniques to a new ANARI backend
• Preparing for ANARI Hackathon (next week: October 21-23)
  • Revamping some ANARI code in my test code
  • Rebuilding all the ANARI code that has a new release in preparation for the Hackathon
• Next Week: ANARI Hackathon (will consume most of my time)

• Exploring better volume rendering on WebXR/WebGL
  • Exploring some new volume rendering code, but not fully functional yet
• Exploring A-Frame networking
  • Networked A-Frame on github
  • Issues running with current version of npm
• Updated ILLIXR ANARI interface
  • includes writing a new example for the volume rendering (see image above)
• Reviewing volume rendering techniques from Toirt Samhlaigh
  • The volume rendering implementation of Toirt Samhlaigh is consistently better than other volume rendering backends, so I've begun looking into either:
    • Adding a streaming capability to Toirt Samhlaigh
    • Learning how to transfer the volume rendering techniques to a new ANARI backend

Planned near-term work

• Prep for ANARI Hackathon (October 21-23)
  • ANARI Hackathon Registration Form
  • Explore new volume rendering
  • Enhance ParaView interface
• Continue testing of A-Frame networking API
• Explore Trame combined with WebXR
• DONE: Create an ILLIXR volume rendering plugin
• Test on NCSA Illinois-Computes system
• Investigate AWS as possible web-host

Previous development highlights

• Exploration of Trame
  • open-source Python-based visualization web-server Kitware Trame
  • works with VTK
  • ported volume rendering example to Trame
  • compared ANARI with VTK volume rendering
    • ANARI is surprisingly slower
    • VTK does a reasonable job of multi-channel rendering
• Preliminary 360 sphere rendering experiments with WebXR
  • testing from recorded 360
  • need to develop a streaming methodology
• Updated ANARI port to ILLIXR
  • next: port the volume visualization to an ILLIXR plugin

• Lots of work building existing tools on newly installed OS
  • Time-consuming hassle to deal with incompatible versions of CUDA, etc.
• Successful build of new ILLIXR code
  • Next step: recreate the ANARI plugin for ILLIXR
• Testing new Visionaray multi-volume rendering
  • Performance has been improved by the Visionaray author
• Learned of a new OpenXR-remote rendering tool on the horizon
  • Electric-Maple

• Exploring transfer functions that better match Steven's expectations
  • Toirt Samhlaigh matches pretty well (see side-by-side above)
  • Barney can render the channels separately

    (currently looking at operations needed to merge)

    • Not yet working

• An OpenXR application that can render ANARI renderings in the HMD has been written
• Remote rendering of OpenXR applications to a Quest has been demonstrated
  • Using WiVRn
  • Tested with ANARI rendering, and it works!
  • (Still need a CUDA renderer on host with WiVRn server for reasonable rendering rate)
• At least one (perhaps two) ANARI backends now properly handle multiple overlapping volume rendering
  • The Visionaray (visionaray_cuda) ANARI backend has been tested — images below
  • The Barney ANARI backend has reportedly been fixed, but has not (yet) been confirmed — is next on the list of tests
• The red,green,blue test volumes have been replaced by actual 3D microscope data (skin) provided by the Mayo Clinic. This is a new dataset, but generally similar to the original (a bit larger).

Remote rendering

• ANARI's "remote" rendering backend feature has been tested, and presently works in some circumstances, but not yet for the volume rendering.
  • The author of the "remote" backend is currently refactoring the code, and my test application is being used as the Guinea Pig.
  • HOWEVER: As the ANARI SDK is not designed to load data remotely, this path is infeasible -- otherwise would need to ship entire dataset over network each time the server is run (which could take hours).
• WebXR: Began investigating the use of WebXR as a delivery means
  • HOWEVER: No browsers for Linux currently support WebXR
• WiVRn: Currently investigating the use of WiVRn as a tool to network OpenXR applications to phone-based headsets/HMDs, such as the Quest
  • Software has been installed on Quest-1
  • Have running WiVRN server on Linux Mint 21.3
    • • This did require an entire OS upgrade
      • Looking at how to build on larger-memory machines
  • Presently streams to local network
    • Need to test with ssh-tunneling
  • For security-concerned systems
    • Needs a new technique to establish network communications
      • The WiVRn authors seem open to adding a new technique
• Electric-Maple:
  • An in-process open-source alternative to 'WiVRn
  • Also based on the Monado code-base (like WiVRn)
  • Thus far only HMD interface implemented
  • Advantage (once it works) is that it could run on security-concerned systems
• CloudXR:
  • Am on the CloudXR testers list (communications with Greg Jones of NVIDIA)
    • Also met one of the project leads at SIGGRAPH (Arjun Dube)
  • ISSUE: CloudXR is presently just for OpenVR
    • OpenXR is in planning stages, I will investigate when it becomes available.
      • FYI: Jakob Bornecratz (formerly of Collabora, now NVIDIA) is working on this

Next on the list

• PARTIAL: Update the ANARI plugin for ILLIXR
• DONE: Finish the WiVRn build and test with OpenXR
• DONE: Develop an OpenXR-ANARI application
• Test the Barney ANARI backend for overlapping volume rendering
• POSTPONED: Target Date: August 7: Develop a WebVX interface suitable for viewing on Stand-alone & PC-VR systems
  • PARTIAL: Will be investigated starting in September
  • Shifted to use of WiVRn for streaming
  • BEGUN: Also todo: Explore the prospects of an ANARI-Web rendering interface
    • Testing with Trame
• PARTIAL:Update the ANARI interface to ILLIXR, and add the Mayo Clinic volume rendering as an example application
  • ANARI port completed, still need to add a volume rendering scene
• Work on new ANARI camera that can render a 360 image over the "remote" interface
• Explore multi-rendering for CAVE and multi-client usage
• Begin developing a user interface that allows the user to select "blobs" as nuclei, etc.
• PARTIAL: Work on an OpenXR interface
• BEGUN: Begin paper Introduction and Previous Work sections

Some pictures

• Toirt Samhliagh rendering of new data — red-channel only

• ANARI-VR rendering of new data — red-channel only

• ANARI-VR rendering of new data — three-channel

Data

Much of the experimental work described here is based on a volumetric dataset created by a 3D microscope, which produces real-color images stacked into the volume. That dataset is too large to provide for quick downloads, so an alternative source for example datasets is provided here (though most are the standard single-scalar type).

• Open SciVis Datasets

Software

Software that has been tested with this dataset (though often with some method of size reduction employed) include:

• ParaView
• ANARI
  • VisRTX ANARI backend
  • Visionaray ANARI backend
• VTKm & VTKm-graph
• ANARI Volume Viewer
• Barney mutli-threaded renderer
  • hayStack (Barney viewer)
  • Barney's "BANARI" ANARI backend
  • hayStack's "HANARI" ANARI backend
• PBRT code from Physically Based Rendering book
  • Ingo's reader for PBRT files

ParaView usage

hayStack usage

The hayStack application uses the multi-GPU Barney rendering library to display volumes with interactive controls of the opacity map. It is intended to be a simple application that serves as a proof-of-concept for the Barney renderer. There are a handful of command line options and runtime inputs to know: % ... where

• 4@ — ??
• -ndg — ??

Runtime keyboard inputs:

• ! — dump a screenshot
• C — output the camera coordinates to the terminal shell
• E — (perhaps) jump camera to edge of data
• T — dump the current transfer function as "hayMaker.xf"

Example output (RGB tests):

(Original example -- single channel)

Process

Python Data Manipulation Scripts

Step 1: Tiff to Tiff converter & extractor

This program converts the data from the original Tiff compression scheme to an LZW compression scheme, and at the same time can extract a subvolume and/or reduce the data samples of the volume.

Presently all parameters are hard-coded in the script:

• input_path
• reduce — sub-sample amount of the selected sub-volume
• startAt — beginning of the range to extract (along the X? axis)
• extractTo — end of the range to extract (along the X? axis)

Step 2: process_skin-<val>.py

This program reads the LZW-compressed Tiff file from step 1, and first extracts the R, G & B channels from the data. Using the RGB values, additional color attributes are calculated that can be used as scalar values that represent particular components of the full RGB color. Finally, the data is written to a VTK ".vti" 3D-Image file.

There are (presently) two versions of this file, the first ("process_skin.py") was hard-coded to the specific parameters of the early conversion tests. The second ("process_skin-e300.py") is being transitioned into one that can handle more "generic" (to a degree) inputs.

In the future, I will also be outputting "raw" numeric data for use with tools that only deal in the bare-bones data.

The current (hard-coded) parameters are:

• extract — the size of the original data (used to determine R,G,B spacing)
• gamma — the exponential curvature filter to apply to the data