[Version - I]

[Due to loss of official video documentation, this video captured at rehearse by iphone is the only documentation I had from last installation]

[Project Description]
Interactive installation that addresses rippled mental images as the product of perceived external stimulus [Kinect sensor (Microsoft)] and internal bodily response [Bionomadix wireless bio-sensors (Biopac System Inc.)]

[Concept]
This installation gives user a reconstructed 3-D space that is modulated by user’s bodily input in real-time also as a technological mediation using the metaphor of Ambient Vision.

[Technology]
The virtual space is reconstructed using 3D information from Kinect sensor and the space is then distorted based on user’s biometrics input. Graphics rendering is done using OpenFrameWorks and Audio synthesis is done using Max/MSP in real-time. For graphics effects, vertex and DOF shader are used. Biometrics data acquisition is done using Biopac MP150 hardware, Bionomadix wireless PPG sensor, and AcqKnowledge software (Biopac System Inc.).

[Installation]
Ambient Vision [version 2: Kinect + iPhone Biometric App]
Kinect Cube @ COLLIDER 4: SPECTACLE, Emily Davis Gallery, Akron, March 19, 2012 – April 13, 2012
Ambient Vision [version 1: Kinect + Bionomadix]
End Of Year Show, Media Arts and Technology, University of California Santa Barbara, June 9th, 2011

Reference
- Ambient Vision
- OpenFrameWorks
- M594CM Real-time Graphics Programming
- OpenKinect
- GLSL
- VIJNANA [識]
- Kant structure
- Bionomadix Wireless Physiological Sensing – BioPAC System Inc
- Esoma Exercise System for cardio rehab
- [Follow-Up@2011/11/29] I found “Emptiness is Form” on Scott Snibbe’s website today while he was giving a talk at MAT seminar. What a surprise!

Fight or Flight explores the artistic representation of heart rate signals visually and sonically. The abstract behaviors of boids(swarm) reflect the state of calmness and chaos in real-time.

2012.04.09 – 2012.04.13. “F || F (Fight or Flight)”, PRIMAVERA UCSB’s Festival of Contemporary Arts & Digital Media, CA, USA

–> Fight || Flight Poster
–> Primavera Festival of Contemporary Arts and Digital Media

BioCymatics v1.0 [1min, graphics]

BioCymatics v2.0 [6min HD, Cymatics graphics rendering and granular sound synthesis, iPhone & laptop]

BioCymatics v2.0 [2min, Cymatics graphics rendering and granular sound synthesis, iPhone & laptop]

In this video, two different versions of cymatic patterns are rendered on iphone and laptop, respectively. Both are driven by user’s heart rate. The laptop version of cymatic patterns receives heart rate value from iphone and is reactive to granular sound generated by iphone.

Reference

- Chladni plate interference surfaces
- Granular Synthesis
- “He+ARTs” heart rate iphone app

This allows researchers/musician/artist to explore (via dynamically mapping)/compose multimodal representation of real-time physiological signals in immersive environment, such as AlloSphere Research Facility.

Reference

- Device Server

- Allosphere Research Facility

- BioPac System Inc.

Demo application of multimodal interaction (speech, gesture, physical activity, proximity. Photo used with permission)

Research intern : Solutions for experience sharing via novel interpretation and visualization of real-world sensor data

2012.01.20. Presentation. “Multimodal Interaction as Communicative Media of Narrative Form for Experience Sharing” , The 6th international conference on Design Principles and Practices, UCLA, Los Angeles, CA, USA

2012. Péter Pál Boda & Yuan-Yi Fan, “Multimodal and Multisensory Interaction as Communicative Media of Narrative Form for Experience Sharing”, Design Principles and Practices: An International Journal

There are 21 books from Chomsky used in the data visualization. Each book title is used as filter when parsing data from the SPL’s database, and further used to query recent news title from Google News API.

The dimension in this 3D data visualization is explained as the following:
z-axis : time, 2006 – 2010
y-axis : book titles in chronicle order, 1957 – 2010
x-axis : the transaction count at SPL is represented in red, the checkout duration is represented in blue

The most interesting finding, as shown in the following figures, is how Google News API interprets these book titles, namely, the query results from Google News API reveals relevant/irrelevant meaning to these book titles, for example, Figure 2. and Figure 4.

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

Reference

- M259 Visualizing Information
- Seattle Public Library
- Processing
- Noam Chomsky

Harmonics in frequency spectrum of human pulse pressure waveform

Image documentation of BioRhythm installation

Video documentation of BioRhythm installation

Interactive installation that addresses frequency relation of harmonic pulse spectrum in real time

Origin

BioRhythm originated from my early interests in 2008 in harmonics on spectrum of pulse pressure wave. Fascinated by this simple and elegant phenomenon, several approaches have been taken upon this resonance phenomenon in Hemodynamics (Harmonic Analysis of Arterial Pressure Pulse Waves) : an experiment has been designed to investigate relation between audio stimuli and blood vessel wall displacement using ultrasound. While there wasn’t enough satisfactory results from scientific approach, a different approach was chosen to express this phenomenon. The choice was made so that it can reach audiences in various fields.

More on origin

I’m fascinated with simple patterns in organic body, harmonics that the human heart generates with the artery system, and currently designing tool and looking for forms to represent these patterns. Sphygmograph is one of my interest for its nature that involves interpretation of haptic sensory information and discourse in objectivity and subjectivity in interpretation of pulse information. Here are some of my exploration since 2008: Sphygmograph (2010), Electrocardiograph (2010),Photoplethysmograph (2011), and Ultrasound (2009).

Inspired by simple and elegant phenomenon on frequency spectrum of pulse pressure wave, harmonics (or overtone that roughly falls on 1, 2, 3, 4, 5, and up to 9Hz), with very long history and recent scientific approach to correlate these harmonics peaks as index for corresponding visceral organs); this is the first-known approach to represent resonance of pulse signal according to ancient mappings to pentatonic scale, cardinal direction, and colors in forms of real- time visualization and spatial sonification.

Publication

Ryan McGee, Yuan-Yi Fan, Reza Ali, BioRhythm – a Biologically-inspired Audio-Visual Installation, in Proc. of NIME 2011, The 2011 International Conference on New Interfaces for Musical Expression, Oslo, Norway, May 30-June 1, 2011, pp. 80-83.

Installation

2011.06.01. “BioRhythm”, Installation at NIME, Oslo, Norway

2011.02.28. “BioRhythm”, Installation at TEDActive2011 AlloSphere Open House, Santa Barbara, CA, USA

2010.08.19. “BioRhythm”, Installation at Mindshare L.A., Los Angeles, CA, USA

2010.06.07. “BioRhythm”, Invited Presentation at Biopac System Inc., Santa Barbara, CA, USA

2010.05.26. “BioRhythm”, Installation & Critique Session, End of Year Show, Media Arts and Technology, UC Santa Barbara, CA, USA

How I learned about all this

Professor Wei-Kung Wang was the chair at Institute of Biomedical Engineering at National Yang Ming Medical College from 1987 to 1989, where I got my MS. I read about his pop-sci book “Music movement which — Was mad” in 2007 and all his publications on resonance theory as well as harmonic pulse analysis since then. And I was so inspired and gained deeper interests in cardiovascular science after having a short talk with Professor Shu Chien at 6th Wu Ta-You Science Camp- Prospect for Cardiovascular Science, Aug. 10-15 2007. While reading “Biomechanics: Circulation”, ”Biomechanics: Mechanical Properties of Living Tissues”, and “Biomechanics: Motion, Flow, Stress, and Flow” in class, as a side note, I also found out Professor Chien works closely with Professor Yuan-Cheng Fung at UCSD Bioengineering and Institute of Engineering in Medicine. Then I accidentally found recent collaboration between National Yang Ming Medical College and University of California San Diego:

- 2010 Establishment of the International Research-intensive Center of Excellence (I-RiCE) for Advanced Bioengineering Research between the University System of Taiwan (UST) and University California San Diego (UCSD)
- 2011 The International Research-Intensive Center of Excellence (I-RiCE) on Advanced Bioengineering; This Center is an international collaborative research partnership between UCSD and two universities from the University System of Taiwan (UST, including the National Yang-Ming University and the National Chiao-Tung University)

Recommended popular science publication on “Resonance – the Missing Phenomenon in Hemodynamics”

W.K. Wang, “Music movement which -­‐ Was  mad”, regular script Chinese, the bulk culture, on September 07, 2002 published, ISBN: 9867975502 (王唯工 ,二零零二 ,氣的樂章,大塊文化)

Further reading, a complete list of publications –>

“ECG-Breathe-Ambient” is remix of my previous work “POM” presented at experimental electronic music concert. “ECG-Breathe-Ambient” presented at the CREATE concert “Critical Point”, Lotte Lehmann Concert Hall, University of California Santa Barbara on Thursday 13 October 2011.

It is an experimental sonification remix piece that is based on sonification of electrocardiographic data of myself in a 30-min meditation process and ambient sound sample recorded in the mean time. It describes the distorted sense of time, the patterns in the spectrum of ECG signal in the process of letting go thougths, as well as the involuntary attention shifts between perceiving inner sound and ambient sound.

Reference
- Information about CREATE concert “Critical Point” –> 2011_UCSB_CREATE_Concert_Critical_Point
- More about CREATE –> Center for Research in Electronic Art Technology