BioCymatics v1.0 [1min, graphics]

BioCymatics v2.0 [6min HD, Cymatics graphics rendering and granular sound synthesis, iPhone & laptop]

BioCymatics v2.0 [2min, Cymatics graphics rendering and granular sound synthesis, iPhone & laptop]

In this video, two different versions of cymatic patterns are rendered on iphone and laptop, respectively. Both are driven by user’s heart rate. The laptop version of cymatic patterns receives heart rate value from iphone and is reactive to granular sound generated by iphone.

Reference

- Chladni plate interference surfaces
- Granular Synthesis
- “He+ARTs” heart rate iphone app

This allows researchers/musician/artist to explore (via dynamically mapping)/compose multimodal representation of real-time physiological signals in immersive environment, such as AlloSphere Research Facility.

Reference

- Device Server

- Allosphere Research Facility

- BioPac System Inc.

Demo application of multimodal interaction (speech, gesture, physical activity, proximity. Photo used with permission)

Research intern : Solutions for experience sharing via novel interpretation and visualization of real-world sensor data

January 2012, Presentation Proposal accepted, ‘Multimodal Interaction as Communicative Media of Narrative Form for Experience Sharing’, 6th International Conference on Design Principles and Practices, University of California, Los Angeles, USA

Yuan-Yi Fan & Péter Pál Boda (2012), “Multimodal Interaction as Communicative Media of Narrative Form for Experience Sharing” (submitted to Design 2012 International Conference on Design Principles and Practices & An International Journal, 20-22 January, 2012, UCLA, Los Angeles, CA, USA)

Reference
- Adviser at Nokia Research Center Hollywood : Péter Pál Boda

There are 21 books from Chomsky used in the data visualization. Each book title is used as filter when parsing data from the SPL’s database, and further used to query recent news title from Google News API.

The dimension in this 3D data visualization is explained as the following:
z-axis : time, 2006 – 2010
y-axis : book titles in chronicle order, 1957 – 2010
x-axis : the transaction count at SPL is represented in red, the checkout duration is represented in blue

The most interesting finding, as shown in the following figures, is how Google News API interprets these book titles, namely, the query results from Google News API reveals relevant/irrelevant meaning to these book titles, for example, Figure 2. and Figure 4.

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

Reference

- M259 Visualizing Information
- Seattle Public Library
- Processing
- Noam Chomsky

Harmonics in frequency spectrum of human pulse pressure waveform

Image documentation of BioRhythm installation

Video documentation of BioRhythm installation

Real-time audio visual interactive installation based on mapping harmonic peaks on spectrum of pulse pressure wave to color and pentatonic scale

Origin

BioRhythm originated from early research interests in harmonics (and frequency relation) on spectrum of human pulse pressure wave in 2008. Fascinated by this simple and elegant phenomenon, several approaches have been taken upon this resonance phenomenon in Hemodynamics (it’s corresponding analysis method : Harmonic Analysis of Arterial Pressure Pulse Waves) : an experiment has been designed to investigate relation between audio stimuli and blood vessel wall displacement using ultrasound. While there wasn’t enough satisfactory results from scientific approach, a different approach was chosen to express this phenomenon: via collaboration with spatial sound designer and visual artist, this content-driven-technology project turned into an interactive media arts installation. The choice was made so that it can reach audiences in various fields.

More on origin

I’m fascinated with patterns in physiological signal, activity of the human heart (specifically frequency relation of harmonics in spectrum of pulse pressure wave), and currently designing instrumentation tool and looking for forms to represent these patterns in realtime. Sphygmograph 2010 among other instrumentation tools for heart activity (Electrocardiograph 2010, Photoplethysmograph 2011, Ultrasound 2009; these are links to my approaches since 2008) is of my most interest for its nature that involves interpretation of haptic signs(styles fo touch) and thus discourse in objectivity and subjectivity in interpreted knowledge of pulse. Featured works below are around how patterns are generated from organic body through negotiation with its environment, and have been realized in forms of interactive audiovisual installation and bio-inspired data sonification as samples for electronic music composition. Next step will be looking for physical forms to represent patterns from organic body in response to environment in realtime. My recent iphone app 2011 is an example of enabling object that allows easy network setup for heart-rate-based interactive installation using mobile device.

P.O.M. 2010 and its remix 2011 [ElectroCardioGraphic data sonification - accepted and performed at CREATE concert; awarded in electro-acoustic music composition]
Inspired by discussion in brain-mind dichotomy, state of consciousness, and poly-vagal perspective on autonomous nervous system; I measured my own ECG signal during a 30-minute mediation process, composed these samples after signal processing. Later, I performed a remix of the original composition at CREATE concert.

BioRhythm 2010-2011 [I initiated this collaborated work based on realtime PhotoPlethysmoGraph signal - presented at L.A., USA , Oslo, Norway, TEDx open house at Allosphere, BioPAC company, and other public venue]
Inspired by simple but elegant phenomenon on frequency spectrum of pulse pressure wave, harmonics (or overtone roughly falls on 1, 2, 3, 4, 5, and up to 9Hz), with very long history and debatable scientific approach to correlate these harmonics peaks as index for corresponding visceral organs); this is the first-known approach to represent signal characteristics according to ancient mappings to pentatonic scale, cardinal direction, and colors in forms of realtime computer graphics and spatial sonification. Next step will be looking for physical forms to represent frequency relation of this signal in realtime.

To reach larger audience, I chose to express this elegant and simple phenomenon in multimedia installation form in various approaches.

Publication

Ryan McGee, Yuan-Yi Fan, Reza Ali, BioRhythm – a Biologically-inspired Audio-Visual Installation, in Proc. of NIME 2011, The 2011 International Conference on New Interfaces for Musical Expression, Oslo, Norway, May 30-June 1, 2011, pp. 80-83.

Installation

End of Year Show Media Arts and Technology University of California Santa Barbara 2010, MAT, UCSB, 5/26/2010

BioPac System Inc., Santa Barbara, CA, 6/7/2010

Mindshare L.A., Los Angeles, CA, 8/19/2010

TEDActive2011 AlloSphere Open House, Santa Barbara, CA, 2/28/2011

NIME, Oslo, Norway, 6/1/2011

Future work

I collecte, compose, perform, and keep looking for various forms to represent frequency relation of harmonic pulse analysis.

How I learned about all this

Professor Wei-Kung Wang was the chair at Institute of Biomedical Engineering at National Yang Ming Medical College from 1987 to 1989, where I got my MS. I read about his book “Music movement which — Was mad” in 2007 and all his publications on resonance theory as well as harmonic pulse analysis since then. And I was so inspired and gained deeper interests in cardiovascular science after having short talks with Professor Shu Chien at 6th Wu Ta-You Science Camp- Prospect for Cardiovascular Science, Aug. 10-15 2007. While reading “Biomechanics: Circulation”, ”Biomechanics: Mechanical Properties of Living Tissues”, and “Biomechanics: Motion, Flow, Stress, and Flow” in grad class, I also found out Professor Chien works closely with Professor Yuan-Cheng Fung at UCSD Bioengineering and Institute of Engineering in Medicine. Recent collaboration between National Yang Ming Medical College and University of California San Diego:

- 2010 Establishment of the International Research-intensive Center of Excellence (I-RiCE) for Advanced Bioengineering Research between the University System of Taiwan (UST) and University California San Diego (UCSD)
- 2011 The International Research-Intensive Center of Excellence (I-RiCE) on Advanced Bioengineering; This Center is an international collaborative research partnership between UCSD and two universities from the University System of Taiwan (UST, including the National Yang-Ming University and the National Chiao-Tung University)

Recommended popular science publication on “Resonance – the Missing Phenomenon in Hemodynamics”

W.K. Wang, “Music movement which -­‐ Was  mad”, regular script Chinese, the bulk culture, on September 07, 2002 published, ISBN: 9867975502 (王唯工 ,二零零二 ,氣的樂章,大塊文化)

Selected academic publication related to “Resonance – the Missing Phenomenon in Hemodynamics” (Publications by Wei-Kung Wang)

1989_Specific frequency properties of renal and superior mesenteric arterial beds in rats [Cardiovasc. Res., 23: 465-467]

1991_Resonance the missing phenomenon in hemodynamics [Circ Res. 1991 Jul;69(1):246-9]

1992_The filter properties of the arterial beds of organs in rats [Acta Physiol. Scand., 145: 401-406]

1993_Spectral analysis of radial pulse in patients with acute, uncomplicated myocardial infarction [Jpn. Heart J., 34: 131-143]

1994_Resonance in rat kidney system [Am. J. Physiol., 267: H1544-H1548]

1994_Alteration of pulse in human subjects by three Chinese herbs [Am. J. Chin. Med., 22: 197-203]

1995_Effect of acupuncture at Tsu San Li (St-36) on the pulse spectrum,” Am. J. Chin. Med., 23: 121-130]

1996_Pulse Spectrum Analysis of Hospital Patients with Possible Liver Problems [to update]

1997_Pressure wave propagation in arteries. A model with radial dilatation for simulating the behavior of a real artery [to update]

1998_Transmission efficiency of the aortic bifurcation in rats [to update]

1999_Pulse analysis of patients with severe liver problems: studying pulse spectrums to determine the effects on other organs [IEEE Eng. Med. Biol. Mag., 18: 73-75]

2000_Effect of length on the fundamental resonance frequency of arterial models having radial dilation [to update]

2000_The importance of pulsatile microcirculation in relation to hypertension [to update]

2001_Applying the resonance equation to the blood pressure waveform variation in aorta bending and renal ligation of rats [to update]

2001_Harmonic variations of arterial pulse during dying pulse of rats [to update]

2002_Hemodynamics with total energy [to update]

2003_Effects of aorta bending and renal ligation on the blood pressure a simulation on arterial stiffening [to update]

2003_Influencing the heart rate of rats with weak external mechanical stimulation [PACE-Pacing Clin. Electrophysiol., 26: 36-43]

2003_Pulse Analysis as a Possible Real-Time Biomarker Complementary to SGPT and SGOT for Monitoring Acute Hepatotoxicity [Toxicol. Mech. Methods, 13: 181-186]

2003_The Physical Conditions of Different Organs Are Reflected Specifically in the Pressure Pulse Spectrum of the Peripheral Artery [to update]

2004_Losing harmonic stability of arterial pulse in terminally ill patients [Blood Press. Monit., 9: 255-258, 2004]

2004_Pressure pulse velocity is related to the longitudinal elastic properties of the artery [to update]

2004_The Natural Frequencies of the Arterial System and Their Heart Rate [to update]

2005_Organ specific ligation-induced changes in harmonic components of the pulse spectrum and regional vasoconstrictor selectivity in Wistar rats [Exp. Physiol., 91: 163-170]

2005_Raising harmonic variation of arterial pulse in dying rat [Am. J. Chin. Med., 33: 73-85]

2005_The eccentric position of the heart in the mammalian body and optimal energy transfer in single tube models

2006_Frequency matching and arterial transverse kinetic energy [to update]

2008_The ventricular-arterial coupling system can be analyzed by the eigenwave modes of the whole arterial system [to update]

2008_Microcirculatory effect of different skin contacting pressures around the blood pressure [to update]

2008_The possible role of arterial radial vibration in heart rate and blood pressure matching [Proc. Inst. Mech. Eng. Part H-J. Eng. Med., 222: 773-779, 2008]

2008_The ventricular-arterial coupling system can be analyzed by the eigenwave modes of the whole arterial system [to update]

2009_Age-Related Changes in Specific Harmonic Indices of Pressure Pulse Waveform [to update]

2009_The exponential decay characteristic of the spectral distribution of blood pressure wave in radial artery [to update]

2010_Effects of Captopril on Specific Harmonic Indexes of the Peripheral Pressure Pulse Waveform [to update]

2010_Theory and Applications of the Harmonic Analysis of Arterial Pressure Pulse Waves [to update]

2011_Differences in Pulse Spectrum Analysis Between Atopic Dermatitis and Nonatopic Healthy Children [The Journal of Alternative and Complementary Medicine. April 2011, 17(4): 325-328] (with 林昭庚 (Jaung-Geng Lin))

Further reading, a complete list of publications –>

“ECG-Breathe-Ambient” is remix of my previous work “POM” presented at experimental electronic music concert. “ECG-Breathe-Ambient” presented at the CREATE concert “Critical Point”, Lotte Lehmann Concert Hall, University of California Santa Barbara on Thursday 13 October 2011.

It is an experimental sonification remix piece that is based on sonification of electrocardiographic data of myself in a 30-min meditation process and ambient sound sample recorded in the mean time. It describes the distorted sense of time, the patterns in the spectrum of ECG signal in the process of letting go thougths, as well as the involuntary attention shifts between perceiving inner sound and ambient sound.

Reference
- Information about CREATE concert “Critical Point” –> 2011_UCSB_CREATE_Concert_Critical_Point
- More about CREATE –> Center for Research in Electronic Art Technology

[ Work in progress ]

I’ve been interested in various instrumentation methods of human heart activity, such as sphygmograph, photoplethysmograph, and electrocardiograph. Sphygsmograph has the longest history among above three documented in both ancient China and Greek texts [See reference 1]. Sphygsmograph particularly has discourses upon subjectivity versus objectivity due to the nature of how physician perceives patient’s pulse, which is by “touch”.
Interpretation of haptic sign is the essential inspiration of this project. In ancient China, it’s called qiemo (palpating the mo; pulse-taking); there are similar enthusiasm on pulse in early western medicine, such as extensive discussion on the close association of pulse and palpitation and of sphygmos and palmos, keywords for further reading “Rufus of Ephesus”, “Aegimius”, “Galen”, “Hippocratic”, and “Praxagoras of Cos”.
In ancient Chinese medical texts, physician judge patient’s condition by applying different pressure (between physician fingers and patient’s inner wrist) when taking pulse. Styles of touch is, therefore, the medium that transforms patterns in pulse signal into tactile language/knowledge.
This physical installation is composed of hijack board, force sensitive resistor, iPod, and machined acrylic case. Visualizing spectrum of user’s touch in disorienting waveform(by subtle changes in repetition and contrast) is a metaphor of that styles of touch is one of the determinants for perceived patterns from pulse signal.

Reference
- (Part – I Styles of Touching) Kuriyama, Shigehisa (1999). The expressiveness of the body and the divergence of Greek and Chinese medicine. New York: Zone Book

- Bridget Riley

- Hijack Board

This paper surveys current mobile technology and bio-sensing, video game in health communication, reality mining, social network and chronic diseases, social cognitive theory, and propose a mobile game that encourages users to have a healthier lifestyle through help from virtual community.

Reference
- Health Communication (Prof. Debra Lieberman at Dept. of Communication, UC Santa Barbara)
- Social Cognitive Theory

Inspired by the American legal dramedy “Boston Legal – Kill, Baby, Kill”, recent use of social media in politics, and the Taiwanese Presidential Debate for the coming 2012 Taiwanese Presidential Election, as well as the Talking Head project by my close friend Joshua Dickinson, I decide to spend my final week of winter break on this project, and see where the experiment may lead to. I started this project on 12/28 and the voting started since 1/8 and will last for 7 days. This project was a study of how audiences may favor the politician regarding his/her speech style by voting. With minimal language manipulation, this is also an experiment to how visual images may contribute to the voting result. All images were rendered using the Talking Head Engine. The materials include presidential debate videos from 2011/12/03 and 2011/12/17. The clips were segmented from the last 75-second video from every debate section of the original videos. These sections include proposition, conclusion, and examination by 5 major news agencies as well as 12 civic groups.

The talking head engine “takes a video clip, stripes the audio, analyzes and intelligently segments this audio into small sections marking their point in time, and then gives each of these sections a rating in terms of frequency and amplitude. These segments are then sorted, usually from loudest to softest. The program then returns to the original video clip, grabbing “snapshots” at each of the loudest points, which it then combines into a single image, giving us a still portrait of a speaker at his most emphatic moments”.

For this website, user can vote based on the very first impression of those three speech visualization on frontpage. User can further rate each speaker’s speech style in the end of the post “Visualizing Speech Styles in 2012 Taiwanese Presidential Debate”. User can view voting results after voting or click on Top-rated tab to see rating results.

I’ve been interested in how technology influences media, and on a larger scale, human communication. The following photo was taken at the Newseum, DC Washington, USA, and it’s interesting to see how will the boundary be re-shaped by enabling technology in the near future.

The following are survey on the use of social media on 2012 Taiwan Presidential Election (1/14/2012), briefing on presidential candidates and participatory groups in the debate, as well as the tweet on Taiwan President Election with hashmark of major news agencies(Reuters, Xinhua, CNA, NHK, AFP, AP).

Survey on Use of Social Media in 2012 Taiwan President Election
- Ma Ying-jeou (KuoMinTang)
[Youtube: Yes]
[Twitter: Yes]
[Facebook: Yes]
[Plurk: Yes]
[Website: Yes]
[Google+: Yes]
[App: No]

- Tsai Ing-wen (Democratic Progressive Party)
[Youtube: Yes]
[Twitter: Yes]
[Facebook: Yes]
[Plurk: Yes]
[Website: Yes]
[Google+: Yes]
[App: Yes]

- James Soong Chu-yu (People First Party)
[Youtube: No]
[Twitter: No]
[Facebook: Yes]
[Plurk: No]
[Website: No(unofficial)]
[Google+: No]
[App: No]

Education briefing (sources: wikipedia)
- Ma Ying-jeou
[LL.B. National Taiwan University, 1972]
[LL.M. New York University Law School, 1976]
[S.J.D. Harvard Law School, 1981]

- Tsai Ing-wen
[College of Law, National Taiwan University, 1978]
[M.S. in Legal Science, Cornell University, 1980]
[PhD, London School of Economics, 1984]

- James Soong Chu-yu
[Bachelor Degree in Diplomacy, National Chengchi University, 1964]
[M.A. in Political Science, University of California, Berkeley, 1967]
[M.S. in Library Science, The Catholic University of America, 1971]
[PhD in Political Science, Georgetown University, 1974]

Presidential Debate
- 12/3/2011 1st presidential debate (president candidates)
- 12/10/2011 2nd presidential debate (vice-president candidates)
- 12/15/2011
- 12/17/2011 3rd presidential debate (president candidates)
- 12/23/2011 policy presentation
- 12/30/2011 policy presentation
- 1/4/2012 policy presentation
- 1/6/2012 policy presentation

5 News Agency in 1st and 3rd Presidential Debate QA Session
- Central News Agency
- United Daily News Group
- China Times
- Liberty Times
- Next Media

12 Civic Groups in 2nd Presidential Debate QA Session
- Taiwan Healthcare Reform Foundation
- Consumer’s Foundation, Chinese Taipei
- Taiwan Association for Human Right
- 248 Agriculture Market
- Taiwan Environmental Protection Union
- Judicial Reform Foundation
- Fairtax
- Awakening Foundation
- Taiwan Alliance for Advancement of Youth Right and Welfare
- National Teachers’ Association R.O.C.
- Taiwan Labor Front
- SocialHousingTW

Recent Social Media on Politics
- Internet Politics, 1994
- US Presidential Election, 2008 [Non-classical, Non-symmetrical]
- Egypt, 2011 [No Leader]

Reference (in alphabetic order)
- 2012 Taiwan Presidential Election on wikipedia [Chinese version, more detail][English version, concise]
- American dramedy Boston Legal (Kill, Baby, Kill)
- Appropriation Arts
- Map of world press freedom at Newseum, Washington DC, USA
- More on Taiwan Public Television Service Online
- Taiwan Public Television Service Online Youtube Channel for Presidential Debate
- “Talking Heads” by Joshua Dickinson
- Video materials from PTS Live Streaming Service
- William Safire