Requirements

You need a working reacTIVison setup. Basically you need a camera working in your environment. If you are a Linux user and own a V4L2 compatible camera you might be interested in the article A V4L2 camera engine for reacTIVision. For further information on reacTIVision installation and setup please see the reacTIVision homepage.

You also need a working InstantPlayer setup. See the InstantReality homepage for more information and supported platforms. I am using the SuSE rpm on a Gentoo box (you can read about how to do this here).

This solution only works in the InstantPlayer X3D browser. The reacTIVision integration is based on MIDI and uses the non-X3D-conform MIDI input feature of the InstantPlayer, so other X3D browsers will not work out.

Download

The necessary files for integrating reacTIVision into the InstantPlayer X3D browser can be downloaded here:

• instantplayer_reactivision-0.1.tgz

The package contains the following files:

instantMidi.xml:

MIDI configuration file for reacTIVision

FiducialManager_PROTO.x3d:

X3D files containing the Fiducial and FiducialManager prototypes

test_reactivisionMidi.x3d:

Test scene using two fiducials.

Introduction

Using the manager

To integrate fiducial tracking into the X3D scene simply load the Fiducial and FiducialManager prototypes and instantiate a manager, containing as many Fiducial nodes as you want to track (please note that the current implementation is limited to 17 fiducials). The following example contains a manager with four fiducials and Logger node for debugging output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<?xml version=“1.0” encoding=“UTF-8”?>
<!DOCTYPE X3D PUBLIC “ISO//Web3D//DTD X3D 3.0//EN” “http://www.web3d.org/specifications/x3d-3.0.dtd”>
<X3D xmlns:xsd='http://www.w3.org/2001/XMLSchema-instance' profile='Full' version='3.0' xsd:noNamespaceSchemaLocation='http://www.web3d.org/specifications/x3d-3.0.xsd'>
  <Scene DEF='scene'>
    
  <ExternProtoDeclare name='Fiducial' url='FiducialManager_PROTO.x3d#Fiducial' />  
  <ExternProtoDeclare name='FiducialManager' url='FiducialManager_PROTO.x3d#FiducialManager' />
  
    <FiducialManager DEF='fiducialManager' >
      <Group containerField='fiducialsGroup'>
        <Fiducial DEF='fiducial0' index='0' />
        <Fiducial DEF='fiducial1' index='1' />
        <Fiducial DEF='fiducial2' index='2' />
        <Fiducial DEF='fiducial3' index='3' />      
      </Group>        
    </FiducialManager>
          
    <Logger DEF='logger' level='3' />
    <ROUTE fromNode='fiducial0' fromField='visible' toNode='transObj0' toField='render'/>
    <ROUTE fromNode='fiducial0' fromField='angle' toNode='oi2' toField='set_fraction'/>    
    <ROUTE fromNode='fiducial1' fromField='visible' toNode='transObj1' toField='render'/>      
  </Scene>
</X3D>

Configuring reacTIVision MIDI output

The reacTIVision integration is based on the MIDI output functionality of the reacTIVision server. The output can be configured via an XML file which is passed to the server when started on the command line. The XML file for integrating reacTIVision into InstantPlayer is named instanMidi.xml and it looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
<?xml version='1.0' encoding='ISO-8859-1'?>      
<midi>  
  <map fiducial=“0” type=“hfader” control=“0” min=“0” max=“1”/>
  <map fiducial=“0” type=“vfader” control=“1” min=“0” max=“1”/>
  <map fiducial=“0” type=“knob” control=“2” min=“0” max=“1”/>
  <map fiducial=“0” type=“note” note=“0” />
  
  <map fiducial=“1” type=“hfader” control=“3” min=“0” max=“1”/>
  <map fiducial=“1” type=“vfader” control=“4” min=“0” max=“1”/>
  <map fiducial=“1” type=“knob” control=“5” min=“0” max=“1”/>
  <map fiducial=“1” type=“note” note=“1” />
  
  <!— more fiducials … —>
  
  <map fiducial=“17” type=“hfader” control=“51” min=“0” max=“1”/>
  <map fiducial=“17” type=“vfader” control=“52” min=“0” max=“1”/>
  <map fiducial=“17” type=“knob” control=“53” min=“0” max=“1”/>
  <map fiducial=“17” type=“note” note=“19” />    
</midi>

The position and angle of a fiducial is mapped onto three succesive MIDI controller numbers. The fiducial presence is mapped to a note number corresponding to the index of the fiducial. This makes four <map> entries per fiducial.

Normally you should not need to edit this file.

Testing the system

You can use the provided test_reactivisionMidi.x3d file to test your reacTIVision - InstantPlayer setup.

Starting InstantPlayer

First start InstantPlayer with the X3D test scene containing the FiducialManager. The FiducialManager initializes the MIDI input of the InstantPlayer called HID MidiIn:

1
$ InstantPlayer test_reactivisionMidi.x3d

You should see a windows that looks like the following, with two items a the bottom left labeled “F1” and “F2” representing two fiducials. Since we did not start the reacTIVision application yet, these items currently do not move.

After starting InstantPlayer the output of the following command should contain a HID MidiIn entry:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
$ cat /proc/asound/seq/clients
Client info
  cur  clients : 2
  peak clients : 3
  max  clients : 192
Client   0 : “System” [Kernel]
  Port   0 : “Timer” (Rwe-)
  Port   1 : “Announce” (R-e-)
Client 128 : “MidiIn Backend” [User]
  Port   0 : “HID MidiIn” (-We-)
  Input pool :
    Pool size          : 200
    Cells in use       : 0
    Peak cells in use  : 0
    Alloc success      : 0
    Alloc failures     : 0

Starting reacTIVision

To make the items move, start the reacTIVision server in MIDI mode like this:

1
2
$ ./reacTIVision -m /path/to/instantMidi.xml
opening midi device: HID MidiIn

Please make sure that reacTIVision is using InstantPlayer’s HID MidiIn as it’s MIDI input device, otherwise the tracking data will be send to the wrong application.

Using the fiducial number 0 and 1 from the reacTIVision website, you can now move and rotate the items by holding the fiducial markers in front of the camera.

Limitations

• The current implementation is limited to 17 fiducials. This is due to the limit of 64 fields per X3D prototype/node. Internally every fiducial requires three fields (3*17=51) plus two additional fields for add/delete messages. The IOSensor node contains 11 fields itself, therefore 51+2+11=64. tO overcome this limitation multiple IOSensors could be used but that has not been tried yet.