ML2VR

Matlab is a mathematical visualization and simulation tool. Mathworks describes it thusly:

"MATLAB® is a high-level language and interactive environment for numerical computation, visualization, and programming. Using MATLAB, you can analyze data, develop algorithms, and create models and applications. The language, tools, and built-in math functions enable you to explore multiple approaches and reach a solution faster than with spreadsheets or traditional programming languages, such as C/C++ or Java™."

The ML2VR application is from the Duke University Visualization Technology Group and is available from Sourceforge ML2VR page. ML2VR uses an OpenGL-intercept technique to capture 3D rendering in the Matlab diagram window and transmit geometries and colors to the immersive display. Using accompanying Matlab scripts, interactions in the immersive space can be conveyed back to Matlab to affect the script and allow immediate feedback to the immersed user.

ML2VR sample visualization

How to Run ML2VR in Immersive Mode

First of course you will need a valid copy of Matlab. In addition to the basic Matlab package, ML2VR requires this package:

• Instrument Control Toolbox — used for the socket communication

Linux

Open two shells, one to run Matlab, and one to run the geometry viewing and interacting application. (Note: these do not necessarily have to reside on the same computer, though for now the configuration process described herein does make that assumption.) The VR application must be run first as the doppelgänger OpenGL will attempt to connect with it when Matlab initiates.

ML2VR application shell

% export LD_LIBRARY_PATH=<directory of boost libraries compatible with the Matlab executable>

E.g.:% export LD_LIBRARY_PATH=/usr/local/MATLAB/R2012a/bin/glnxa64

% ./freevr_app

Matlab shell

% export LD_LIBRARY_PATH=<directory of doppelgänger libGL.so>

NOTE: if your LD_LIBRARY_PATH requires other directories for matlab, then be sure to maintain them in the library path.

% matlab

Matlab script A simple way to "just get started" and convert an existing Matlab script to work with ML2VR in a basic way is to add this Matlab code to the top of the script:

% Code to interface with the ML2VR application:
clear;
addpath('../vr_interface');
vr_config;
vi=vr_interface(ServerIP,ServerPort);

close all;
figure('Renderer','OpenGL');

% Code to enable the grab-the-world interface:
vvh=vr_virtual_hand(vi);
vvh.enable(true);
vvh.set_hand_button(0);

Of course, you need to make sure the path to the vr_interface script is correct — safest to hardcode it to the full path. This example assumes you are running in a sibling directory to the vr_interface directory.

Now run your modified script in Matlab:

>> myscript_vr

MS-Windows

(to be documented)

How to Build ML2VR w/ FreeVR

This section specifically instructs on how to compile ML2VR using the FreeVR immersive interface library.

1. Install missing dependencies

   Commonly needed packages include:

   • FreeVR version 0.6
   • boost_system (1.41 works, but must match that used by Matlab)
   • boost_thread (1.41 works, but must match that used by Matlab)

2. Download the latest source (version 4_1_2013 as of this writing [04/11/13])

   http://sourceforge.net/projects/ml2vr/

3. Untar the source ball

   E.g. % tar -zxf ml2vr_4_1_2013.tar.gz

4. Compile the interceptor (doppelgänger) OpenGL library:

   % cd linux

5. Confirm (change when necessary) that the interceptor program will relay OpenGL calls to the correct real libGL:

   % $EDITOR ../intercept/src/gltrace.c

   • search for "libGL.so" and make sure it points to the correct one

     NOTE: pay particular attention to whether it points to the 32 or 64 bit version of the real OpenGL library

   % $EDITOR ../make/Makefile.intercept

   • Point the CFLAGS to find the correct boost libraries

     E.g. /home/avl/Apps/ML2VR/boost_1_44_0

   % make

Compile the VR application:

1. Configure ML2VR

   % cd freevr_app

   % $EDITOR Makefile

   Point FREEVRDIR to the FreeVR directory

2. Compile ML2VR

   % make

How to Build ML2VR w/ Syzygy

... (to be documented)

How to Configure ML2VR

There are three configuration files associated with ML2VR, plus the shell running the Matlab application must be configured to use the doppelgänger (intercepting) copy of the OpenGL library.

• Create a /etc/ml2vr/intercept.cfg file:

  Here is a sample intercept.cfg file for use with Linux:

### OpenGL interceptor configuration file for ML2VR
debug_file   /tmp/intercept.log
opengl_port  9000
client       127.0.0.1

• Create a ml2vr.cfg file in the directory from which the ML2VR application will be run:

  Here is a sample ml2vr.cfg file for use with Linux:

### ML2VR application configuration file:
opengl_port    9000
interface_port 9999
swap_port      9001
swap_ip        127.0.0.1
swap_clients   1

• Edit the vr_interface/vr_config.m Matlab script segment to set matching parameters that will be read within the Matlab visualization scripts:

  Here is a sample vr_config.m file that matches the above configurations:

%% ML2VR Matlab configuration script
ServerIP='127.0.0.1';
ServerPort=9999;

• set the LD_LIBRARY_PATH environment variable to point to the location of the doppelgänger OpenGL:

  % export LD_LIBRARY_PATH=/home/avl/Apps/ML2VR/ml2vr_20130314b/dev/intercept/linux

ML2VR interface code

The ML2VR package includes the directory matlab_files/vr_interface which contains the class and methods used for communicating with the ML2VR immersive application. This provides the utilities needed for making an interactive immersive Matlab application.

For graphics only, the code segment listed above in the How to Run ML2VR section is sufficient. This section briefly describes the classes and methods used to interface back to a Matlab script from within the immersive display.

• vr_interface class
  • Initialization methods
    • vrinstance = vr_interface(ServerIP, ServerPort)
    • vrinstance.set_return_type({'pos3' | 'mat4'})
    • vrinstance.set_transform_matrix(<matrix>)
  • Event Queue methods
    • [event,button,pos] = vrinstance.get_button_event()
    • [event,axis,pos,value] = vrinstance.get_joystick_event()
  • Value Polling methods
    • state = vrinstance.get_button_state(buttonID)
    • value = vrinstance.get_joystick_state(axisID)
    • tform = vrinstance.get_sensor_state(sensorID)

• vr_virtual_hand class
  • Initialization methods
    • vrhandinstance = vr_virtual_hand(vrinstance)
    • vrhandinstance.enable({true | false})
    • vrhandinstance.set_hand_button(buttonID)

• vr_sphere class
  • Initialization methods
    • sphereinstance = vr_sphere(size)
    • sphereinstance.setPos(x,y,z)
    • sphereinstance.setColor(r,g,b)

• vr_cursor class
  • Initialization methods
    • cursorinstance = vrcursor(vrinstance)
    • cursorinstance.setup_ray_preset()

Example Matlab scripts

The ML2VR package includes four directories of example Matlab scripts that demonstrate all the available features of ML2VR. These example collections are:

• framework_examples — examples for doing the most basic operations
• interaction_examples — examples of a variety of ways of interacting between the immersive application and the Matlab simulation
• viz_examples — two examples that demonstrate visualization interactions possibilities
• robot_demo — a simple robot simulation that can be directly affected by the immersed user

Usage Tidbits

• Matlab comments are '%' — putting two together causes them to appear in the script comment window
• The Matlab command license('inuse') will report what packages are used by a loaded script
• Renderings generated by the Matlab plot3() command can be made to generate ML2VR enabled OpenGL calls by pushing it through a 3D transformation. Eg.:

  h = plot3(masscharge, time, spectra);

  set(h, 'Parent', hgtransform); — See [hgtransform] and [makehgtform].

• ...

Usage Issues

• OpenGL lighting calls are not yet implemented in the ML2VR immersive application, so lighting commands within the Matlab scripts should be removed.
• Do not name a Matlab script to end in a "+" — Matlab interprets this and will attempt to load a script with a different name. Eg. instead of "script1+.m" it would try to load just "script1.m" !
• ...

Development Issues

This section is for issues that are of interest to those working directly on the ML2VR code-base.

• Lighting on cube cursor (and pointer in some examples) is discontinuous across screens — this can be fixed by setting the light position each frame.
• Can there be a button from the VR application that terminates the loop in the Matlab script? I suppose this could just be coded as part of the while conditional
• Perhaps a built-in clipping capability can be added to the ML2VR immersive code. Such a feature was part of the prototype and is being considered for re-inclusion.
• Perhaps there can be navigation control that is built-in to the VR application rather than relying on the Matlab script to do the work. This feature exists, but must be enabled within the Matlab script using the vr_virtual_hand class.
• I'm not 100% convinced that the vr_config.m file is really used.
• I'd like to see more comments in the Matlab scripts in the vr_interface directory.

See Also

• Ryan McMahan's blog
• The FIVE lab
• IGERT announcement
• Matlab Wikipedia entry