Time flies when you’re having fun

Saying goodbye

Lots happened the past year, the most notable projects I have worked on as a scientific programmer were “Interactive Networks” (a collaboration with the SNE department) and “Twilight” (a graph visualization and exploration tool for multi-touch devices). I am still busy creating some short videos from the HD footage we shot, so expect them in the near future.

I am very glad that I was offered the opportunity to participate in both projects. Therefore I would like the following people who made it possible for me to work for the University of Amsterdam (UvA):

I would also like to thank:

Finally I would like to thank Dr. Dick van Albada, Edwin Steffens, Erik Hitipeuw, Alexis Salin, my fellow students at the UvA and everyone I forgot.

A new challenge

After a very brief meeting in Amsterdam during the IEEE Tabletop 2008, I got in touch again with Dr. Chia Shen during the IDC 2009. Shen is a Senior Scientist and Program Manager at the Initiative in Innovative Computing (IIC) at Harvard University. She is well known for her work as Senior Research Scientist at MERL (the Mitsubishi Electric Research Labs).

During the breaks at IDC 2009, we had a talk about the work I was doing at the UvA and my involvement in the NUI Group. After showing some work I have been doing on network visualization (Twilight) for multi-touch devices it seemed that there were some similarities between the work I was doing at the UvA and the research that was going on at Harvard University. While discussing some of the ongoing projects, I was asked whether I was interested to join the SDR Lab as a research fellow. After some consideration, I proudly accepted her job offer.

SEAS Building

The next months I will be working on several interesting projects. One of the tasks will be continuing the development of INVOLV, a multi-touch application developed by Michael Horn and Chia Shen in collaboration with the Encyclopedia of Life (EoL), the Tree of life (ToL) and many biology professors and scientists since May 2008. The application visualizes the data of the EoL and the ToL in a unique way using a Voronoi diagram and is used for learning and discovery.

A video of Michael Horn explaining the INVOLV project can be found below:

Living in the USA

It has only been a week since my arrival in the USA, but so far it has been a pleasant experience.

I have not made a lot of photos yet, but this one was made nearby the place where I am staying (safety first?):
Strange pole?

More photos will be posted on Flickr @ flickr.com/photos/multigesture/


NUI Book and GSOC 2009

NUI Book

To help out fellow multi-touch users and developers, the NUI Group community decided to create a book sharing information about multi-touch technologies. Originally we intended to release this book during the IDC2009, unfortunately we were only able to finish a concept version. This concept version was available at the NUI Group booth.

mttechcoverNUI Group press release:

“We are proud to announce the release of the first edition of the NUI Group Community Book – Multi-Touch Technologies. This book comprises state of the art articles from our Community Wiki, as well as several chapters written by leading experts in the field specifically for this publication. Detailing basic hardware technologies as well as the process of developing software suited for multi-touch systems, this book provides a technical introduction to the world of multi-touch.

This book aims to provide an introduction to the multi-touch world for all backgrounds of people, and also include sufficient information to serve as a general “handbook” for those acquainted with the field, including code examples, suppliers, and building methodology.

Multi-Touch Technologies is a unique publication in its domain in that it is released at no charge under the Creative Commons 3.0 Attribution Share-Alike license – all content found in this book is free to be re-distributed and modified under a similar license, with attribution.”

View the full press release.
Download your copy here: First_Edition_Community_Release.pdf

Comments and suggestions can be posted in the feedback topic @ NUI Group forums.

* update 15 July *
Mirror of the NUI Book:
English version
Chinese version (special thanks to Kelvin / mt2a for translating the book, feedback can be posted here).

Google Summer of Code 2009

2009socwithlogo

Like last year, the NUI Group will be coaching students to work on open source multi-touch projects. This year, I will be helping out by mentoring a student who will add hand tracking to Community Core Vision (formally known as tBeta).

Project: Recognition, Tracking and Association of Hands, Fingers, and Blobs: A Community Core Upgrade
Student: Thiago de Freitas Oliveira Araujo
Mentor: Laurence Muller

Follow his progress on his blog and his code on nuicode or google code.

Hand/Fingers occlusion testcase

occluded_fingers_for_dummies

Matching a hand model (using Active Appearance Models)

More videos can be found here.


Multitouch Media Application Pro v3


Watch it in HD on Youtube or Vimeo!

Changelog for v3:

  • Fixed compatibility issue with Adobe AIR 1.5 (Fullscreen bug)
  • This version should autodetect your screensize to support any screen resolution.
  • Added a Google Maps object
  • Added support for adding pictures on the fly (for example if you want send a picture from your cellphone with bluetooth, read the readme.txt for the instructions)
  • Added a exit-button, reset view-button, canvaslock and displaylock buttons.
  • Temporary removed DCIM support due to slowdowns

Want to try it out yourself?
Download it now: Multitouch Media Application Pro v3 (16828) - 2.25 MB

Please read the install instructions in readme.txt (and download the latest Adobe AIR 1.5.x).

note: If you want to try out the bluetooth feature, I recommend installing BlueSoleil to handle the pairing of the devices and file transfers.


Interactive Displays 2009

interactive_displays_2009_banner

It has been a while since my last blog post, so I will try to catch up a bit.

Earlier this year, Chris Moore invited me to join the Interactive Displays Conference 2009 (IDC) in San Jose, CA, USA. The IDC is an event for hardware manufacturers, developers, researchers and end user of interactive display technologies. The event was organized by pira-international and supported with great keynote speakers: Jeff Han from Perceptive Pixels and Andy Wilson & Steven Bathiche from Microsoft Research.

The NUI Group was also present with a booth to demonstrate some of the current activities from the community. Key members (Seth Sandler, Chris Moore, Alex Popovich, Jimmy Hertz and Thomas Hansen (Author of PyMT) ) from the NUI Group community and NUI INC. where present to demonstrate their hardware and software achievements.

Jimmy Hertz presented his amazing and huge H2O AquaTouch Water Screen at the booth. Alex Popovich demonstrated his multi-camera solution based on custom drivers created for the PS3.

Jimmy setting up his touch screen
3510640691_6406798b43

The NUI Group booth (with free NUI Group T-shirts!)
3510625625_00e1f28265

Alex’s multicamera solution with 3 PS3 cameras
3534192219_a647479aa0

A nice summary of the conference can found on the NUI Group blog. More photos can be found on the Flickr streams from Joan Woodbrey and Thomas Hansen

Seth, Justin and me heading to the car.
3511413410_ccb4004084

Photograph credits: Thomas Hansen

I would like to thank Chris and the other present NUI Group members for this amazing event, I really enjoyed my stay in San Jose and San Francisco.


Interactive Networks at SC08 (part two)

November last year we presented the Interactive Networks concept at Supercomputing 2008. Recently I got hold of more photo and video content.

Photos and videos

In my previous post on Interactive Networks, I gave a brief explanation of the concept. My colleague Rudolf Strijkers (Project lead) created a page with pictures and videos from the demo we presented. This page also includes a video of Rudolf explaining the Interactive Network concept to Gordon Cook. Only the video with Gordon Cook is in English, but I guess that most of the videos are self explanatory. The videos and photos be found here: Supercomputing 2008 media

COOK Report

After Gordon Cook was given a brief introduction to the Interactive Networks concept, he invited Rudolf Strijkers and Cees de laat for a more in-depth interview. The article (abstract) can be found here: ICT and E-Science as an Innovation Platform in The Netherlands “A National Research and Innovation NetworkWhat Can the US Learn from Dutch Experience?”

Posters

tno-sc08-interactivenetworks uva-sc08-multi-touch_table

More posters from our booth can be found here: clickme!

Pictures of multitouch systems presented at SC 2008

RENCI Vis Group Multi-Touch

The guys from Renaissance Computing Institute were kind enough to show us their table. We could even take a peek under the black curtain that contained all the *magic*.

IMG_0021.JPG IMG_0022.JPG IMG_0023.JPG IMG_0024.JPG
IMG_0038.JPG IMG_0045.JPG IMG_0046.JPG IMG_0047.JPG

More photo and video content can be found on their blog.

EVL | Multi-Touch LCD Table Prototype

EVL showing a rain simulator on their LCD based multitouch table.

IMG_0030.JPG IMG_0032.JPG
Checkout the video of the table here. More pictures of the setup can be found here.

Enthought

A stable, portable and responsive multitouch system using a python backend.

IMG_0026.JPG IMG_0027.JPG IMG_0028.JPG IMG_0029.JPG

“Me painting like a pro, looks better than Picasso … right?”

MS Surface

This doesn’t really needs an explanation…

IMG_0049.JPG IMG_0050.JPG IMG_0051.JPG IMG_0052.JPG

“Dear Santa, please send me a MS coffee table this year.”

(more…)


fIRC Android Chat Client released!

A few months ago Google and T-mobile introduced the T-mobile G1 (aka googlephone). What makes this smartphone so special, is the fact that the operating system (Android) is open source. While most end-users wouldn’t care about this, developers have more freedom in improving or fixing parts of the OS.

As usual we Europeans get all the ‘new’ electronic devices 6 months later than the US or Asia. Often these devices cost more as well (for example iPhones, PSPs, NDS) because someone decided that 1 USD == 1 EUR. Luckily I was able to get a T-mobile G1 just before I left the states 🙂 .

Today I released my first application on the android marketplace (something similar like the apple store). It is an IRC chat client called fIRC:

fIRC chat is a free IRC client.

Current features:

  • Connect to irc servers such as Freenode and EFnet
  • Supports only one channel for now
  • Only landscape mode for now…

Available user commands:

  • To change your current nickname: /nick changeme
  • To request a user list: /userlist

Todo:

  • Server message output
  • Multichannel support
  • PM support
  • A better way to visualize the userlist (another listview?)
  • Nicer GUI for the chat area

A few screenshots:

Suggestions, bug reports and comments can be posted here. The project page can be found here.


Interactive Networks at SC08

After my graduation in July, I was asked to join the Scientific Visualization and Virtual Reality group at the University of Amsterdam. The project that I would be working on was network visualization on a multitouch table that was going to be presented at Supercomputing 2008 in Austin, TX.

Because our old table (which we used for the multitouch performance experiments) was moved to the NEMO Science Center in Amsterdam, we had to develop a new multitouch system from scratch. Based on our experiences of building the previous multitouch table, we carefully choose our new hardware components (such as the camera and digital projector).

Supercomputing 2008


Supercomputing is an international conference for High Performance Computing, Networking, Storage and Analysis (SC08, November 15-21, 2008 Austin Convention Center). In collaboration with the Systems- and Networking Engineering Group of the University of Amsterdam we developed software and hardware for our project called Interactive Networks.

Our project consists out of several parts:

  • The multitouch table; used as an input/interaction device (Rear illumination)
  • Multiple (linux) computers using custom kernal software called streamline
  • A controller that processes events from the multitouch table to the computer nodes using streamline (and vice versa).
  • The Interactive Network Visualizer (the visualization and manipulation software on the MT table)

The project focuses on stream manipulation in a token based network. In Amsterdam we created a physical network of computer nodes (16+). Each of these computer nodes has the streamline module installed. Depending on the task of the node, a node can be the following:

  • A producer (A node that is capable of delivering a video/content stream)
  • A router (A node that is able to route packages)
  • A consumer (A node that is capable of receiving and displaying the stream content)

In the current setup we have 4 producers (each with different video content), 8 routers and 4 consumers (Mac mini’s connected to HD screens with VLC installed to display UDP video streams).

In order to stream video content from the producer to a consumer node, users can simply switch to the ‘route planner’ mode and draw a line from node to node. When the user completes a path (stream), the result is submitted to the controller. The controller checks if the nodes are connected and instructs the nodes how the incoming packages should be handled. Normally it is difficult to distinguish different data streams from each other. However, in our system we added a ‘token’ (color) to each stream. This allows us to distinguish a stream on a particular node and transmit it to its destination. Our system has support for multicasting and streams containing cycles (unique!).

Because streams are tokenized we can also manipulate particular streams on the node itself. By double tapping a node, we can view the internals of this particular node. All streams that passes through this node can be viewed and modified. For this particular demo we added an option to add a sampler. The sampler allows the user to set a value of how many packages should be dropped.

By using standard Linux tools we are also able to measure the network throughput on a node, the load on a node, the used bandwidth between two nodes and the jitter on a connection. These values are visualized in our software.

Videocontent

For our demo we stream high definition videos from The Netherlands to the USA over an 4 GBit internet connection. We are using movies that were made in Blender, namely: Big Buck Bunny and the Elephants dream.

Live videofeed

On our dutch stand we have three live camera streams showing different projects, be sure to check those out: webcam overview

Other multitouch projects at SC08:

We certainly aren’t the only ones with a multitouch table at SC08, so far I’ve seen the following:

  1. RENCI Vis Group Multi-Touch Blog
    Developing multitouch hardware and applications for research and experimentation.
  2. EVL | Multi-Touch LCD Table Prototype (Tactile)
    A high resolution multitouch device created from a LCD panel Video!
  3. Enthought (Scientific Computing Solutions)
    A fast and responsive multitouch solution using a python backend. They are located at the stand of Western Scientific
  4. MS Surface (on the HPC stand)

Photos

Photos of our setup can be found after the break!

(more…)


IEEE Tabletops and Interactive Surfaces 2008

Tommorow the 3rd edition of IEEE Tabletop will take place in Amsterdam.

On the first day of the conference there will be a bootcamp on building your own multitouch surfaces.

At the bootcamp the ‘experts’ will share their knowledge and experiences on building multitouch surfaces. The following topics will be covered:

  • Infrared Illumination
  • Silicone & Projection surfaces
  • Cameras, Optics, Filters & Projectors
  • Software
  • Hardware Integration
  • Community & Network

I will be at one of the stands presenting the software part together with Ulrich von Zadow.

Hope to see you soon,
– Laurence

* Update *
We now have a website running with the bootcamp information.
– Project page: http://ifgi.uni-muenster.de/multi-touch-bootcamp
– Some photos of the conference: click me!

* Update 2 * (22/03/2009)
IEEE tabletop bootcamp page update.
Multi-Touch Surfaces: A Technical Guide (mirror)


Thesis work

For those who are interested, I have uploaded my master’s thesis and final presentation.

1. Master’s thesis:
Multi-touch displays: design, applications and performance evaluation (5184)
Mirror: SCS – Master of Science theses
(1.4 mb)

2. Final presentation:
multitouch_presentation.pdf (4956)
(26.6 mb, includes embedded wmv videos)

3. Multitouch video:
multitouch-videomix1.wmv (3474)
(76.3 mb, high quality wmv)

*update 17-09-2008*
4. Multitouch experiment videos:
10 multitouch experiment videos

*update 20-10-2009 (mirrored files)*
4. Multitouch experiment videos:
10 multitouch experiment videos

4. Experiment videos:
Multitouch experiment videos (youtube playlist)


I graduated!

After almost a year, I finally finished my thesis and presented it. A brief overview of my work can be found in the video below. The thesis itself will become available later this week and will of course be posted on this blog!


Multi-touch video puzzle game

The same game I have showed here but this time played on the multi-touch table.

p.s. Sorry for the ending 😛 it wasn’t me!


tDesk released!

Last year I demonstrated how Windows XP could be (ab)used with multi-touch interaction.

A few weeks ago I made some adjustments to make it compatible with the latest touchlib and this is the result:

tDesk v2 allows the user to move and resize multiple windows at the same time by using multi-touch gestures. Because Windows XP/Vista was not designed for multi-touch there are some issues with this type of input. In multi-touch mode it is not possible to do a mouse click, therefor I added an option to switch between multi-touch and mouse simulation mode.

Please keep in mind that when the mouse simulation mode is activated only single touch input is allowed. Do not use this software on unstable multi-touch systems.

Finally, use this software at own risk 🙂 .

Install instructions are in: readme.txt !

Downloads:
tDesk v2 (4074) - 13.32 kB
Touchlib (speedfix) (4810) - 785.69 kB – rev393

Any tips or problems can be posted in the comment section.

* update *
In some cases tDesk does not seem to work properly with touchlib. Try downloading rev393 and overwrite your current touchlib dll files.


Touchlib Speedfix and MMA Pro update

Touchlib

A few weeks ago I was busy tweaking the Touchlib sourcecode and noticed a bug/feature in the code which caused the image processing pipeline to stall for 32 milliseconds. This might not sound like a big issue however on our system it caused the system to be less responsive.

I have compiled a version without this ‘sleep’ function. This version might not be safe for single core cpu’s (I could not test it out myself, hopefully someone could check it out). Because the sleep function is removed, the videoplayback (using cvcapture + *.avi) is not working properly.

Check out this version of touchlib and report any problems in the comment section 🙂 !

Download:
Touchlib (speedfix) (4810) - 785.69 kB compiled libraries, revision 393 speedfix (21 apr 2008)

MMA Pro v2d

Changelist:
– Added smoothing, scaled images should look a lot better
– Added an View Reset and Exit button
– Increased the size of the canvas
– Compatible with Adobe Air v1.0

Install instructions are in: readme_pro.txt !

Download:
Multitouch Media Application Pro v2d (4146) - 2.22 MB
Adobe AIRrequired

Example video showing how to load photos dynamicly into MMA Pro

Wong!

Wong is a game based on pong which uses the multitouch input to create dynamicly sized paddles.

A game of Wong on our multitouch table:

The game is made by Seth Sandler and is part of the audiotouch application suite: http://code.google.com/p/audiotouch/


Real-Time Fluid Dynamics on a Multi-touch Table

Smoke improved…

One of the demo applications which comes with Touchlib is Smoke. The technique which is used to create ‘smoke’ is based on the algorithm presented by Jos Stam in his paper Real-Time Fluid Dynamics for Games.

Often multi-touch enthusiast use this application to demonstrate their multi-touch displays.

Some time ago I modified the Smoke application which allows users to draw objects into the scene. The application is extended to have a new drawing mode which is allow (multiple) users to draw objects/boundaries into the scene. This can be done while the simulation is running. If sources are placed in the scene this means you can view in real-time how flow will behave on a particular shaped object.

In the demonstration video I have added 3 different coloured sources and the logo of my university is used as an object to perform the simulation on.

A higher quality wmv can be downloaded here:
RTFD.wmv

Multi-touch video puzzle game

I had this idea for a while and even the basic puzzle game framework (using photo’s) was already finished. After messing around with FFmpeg I managed to export the video frames from an mpeg2 (.vob) file and convert them to an OpenGL texture. The only problems left were the synchronization and the audio stream. Because FFmpeg is really horrible coded (you need special hacks to get it even to compile under windows), I never managed to get those working. For those who are interested, this site: dranger.com has helped me a lot.

After searching around for an alternative I decided to drop the cross platform support for this application and use MS DirectShow. Usually I do not like the DirectX due its ‘clean’ coding style. But after giving DirectShow a try I was surprised how easy it was to get a basic demo working which played a mpeg2 file without any problems. The audio and video streams where synchronized automatically.
With some help of a friend I merged the video player code into the puzzle application.

This is the result (recorded with Fraps):

In the video I used the mouse as input device, but the application works with the multi-touch table as well.


MMA Pro update

Minor fixes:
– Fullscreen should be working now
– It is should be possible to use any screen resolution now (previous version used 1024 x 768, hardcoded)
– Keyboard can be locked on the stage (prevents the keyboard from jumping around while you’re typing 😉 )

And as usual: Don’t forget to read the readme_pro.txt !

Download:
Multitouch Media Application Pro v2c
Adobe AIRrequired


Multitouch @ SOCO Amsterdam

My 2 minutes of fame…

Last saturday MMA Pro was presented on national television (NED3, VPRO) in the show called “In de ban van ‘t ding”. The documentary created by Dahl TV and Bright Magazine is about the (new) technologies which have an impact on our daily lives.

In this weeks episode Erwin visits SOCO Amsterdam. SOCO (Social Contemporary Center for the Arts) is a new exhibition/project space in Amsterdam. Peter Distol and Ralph Das are giving a demonstration of the table we have build together at the SOCO. The table uses front illumination and touchlib for touch detection and processing.

A part of the episode can be viewed here (it is in Dutch):

The whole episode can be viewed here:
Uitzending gemist

The applications which have been demonstrated:
– Smoke (basic demo from touchlib)
– Newswell (Ralph Das)
– Multitouch Media Application (showing Jimi Hendrix!) (Laurence Muller)

Currently there are two multi-touch tables available to the general public.

Multitouch Media Application Pro Update

The previous release of MMA Pro only worked with Adobe AIR beta1. A few users have contacted me about the troubles when installing MMA Pro v2a with Adobe AIR beta3 installed.

I have updated the application to make it compatible with Adobe AIR beta3 (Thank You Adobe! for renaming zillion functions).

The application requires Adobe AIR beta3 runtimes, so do not forget to install them first. Besides the fixes, there is nothing new.
Ofcourse this version also supports adding images on-fly. Just connect your digital camera (or memorycard) and all your images will be downloaded to the desktop. Your camera should be using the DCIM standard).

Don’t forget to read the readme_pro.txt !

Download:
Multitouch Media Application Pro v2b
Adobe AIRrequired