The MEGA System Environment

The MEGA System Environment is the main scientific/technological result and concrete product of this project. It has been used along the project in public events, including museum exhibits and artistic performances. 

The MEGA System Environment includes the following systems and components:

1. The EyesWeb open software platform version 3.2.1, developed by DIST as a kernel.

2. A collection of software modules for analysis and synthesis of expressive gestures, integrated or connected in the MEGA System Environment:

The EyesWeb Expressive Gesture Processing Library (see description) including a collection of blocks and patches contained into three main sub-libraries (DIST):

-     The EyesWeb Motion Analysis Library: a collection of modules for real-time motion tracking and extraction of expressive cues from human full-body motion.

-     The EyesWeb Space Analysis Library: a collection of modules for analysis of occupation of 2D (real as well as virtual) spaces.

-     The EyesWeb Trajectory Analysis Library: a collection of modules for extraction of features from trajectories in 2D (real as well as virtual) spaces.

The CSC – DEI Library providing a set of EyesWeb block divided into four branches:

-     Expressive Analysis with the aim at analysing in real time the expressive content of a performance in MIDI format.

-     Expressive Synthesis aiming at conveying different sensitive content likes expressive intentions and emotions by opportunely introducing systematic deviations into a MIDI score.

-     Spatialization: a library intended to provide a tool for the management of the virtual 3D space to be simulated in generic Concert Halls.

-     Generic Blocks: integrations to the EyesWeb Generic Library.

DEI-KTH Expressive Director: based on the CSC-DEI Library, the Expressive Director combines two different approaches for computer modelling of expressiveness in music performance: the KTH rules and the CSC Expressiveness model. It allows an user to transform in real time the expressive intention of a musical piece.

IPEM Ear: a real-time auditory model as an acoustical front end and modules for real-time extraction of features, based on auditory model, related to loudness, roughness, pitch, onsets and rhythm, including statistical features related to time-dependent means, variances, densities. “Ear” is integrated within EyesWeb and also integrated within Pure Data (a more complete pd installation can be downloaded here). It is an attempt to model different aspects of human hearing. It consists of a set of 40 bandpass filters on a critical-band scale (cochlear filterbank). In each of the filters, there is an analysis of the loudness, roughness, virtual pitch and spectral pitch. These outputs can be visualized and processed in EyesWeb and Pure Data.

Director Musices version 2.5 (KTH), the KTH rule system for music performance, summarizing 20 years of research in the field. It has been integrated into the EyesWeb architecture for real-time control of musical expressiveness. A new version of Director Musices (version 2.4) used for generating expressive performances by rules was released. The new version includes the generation of preprocessed data to be used in the Expressive Synthesis library by DEI. Several new rules have been included and the midi input parser has been completely rewritten. This version now works on most Windows operating systems as well as Macintosh.  

GEM SoundAnalysis Library (see description): a library intended to add audio analysis features to the EyesWeb platform, including:
  • Frequency analysis
  • Frequency-domain cues
  • Pitch detection ( autocorrelation )
  • Vibrato (Vibrato Through hand/arm gesture analysis)

GEM EyesWeb SoundSinanTools Library, intended to provide a set of tools for sinusoidal modelling and sound processing to the EyesWeb platform. Some of the features provided by the library are sinusoidal analysis, file generation from audio, sound resynthesis from sinusoidal analysis file, time-scale and pitch modifications. It is included in the next EyesWeb distribution.

GEM SoundSynth Library (see description)

GEM WK4Tools (see description)