Visualization Techniques
Real Time Animation for Music
Visualization, in this context, refers to the creation of animated graphics that respond in real time to the volume and pitch of an auditory source, e.g. music.
Rather than create animation using video editing programs, e.g. Adobe's After Effects, or dedicated animation software such as Maya and Blender, two small companies pioneered another approach which used the physical properties of an auditory source to move predetermined sprites or the X/Y/Z coordinates of mathematical objects across a multi-layered screen, with each layer sampling or moving those coordinates at independent and variable rates. The auditory elements could determine the brightness/intensity of the object, the color of the objects, the position of each axis over time on the screen, the coordinates at which those axes enter and exit the screen, the modification of those coordinates when encountering a succeeding sprite/mathematical formula, and several other options. The two products that offered this different approaches to real-time animation were "Vusic," and "G-Force" led by Andy O'Meara at SoundSpectrum. Neither of these two platforms have survived, although SoundSpectrum does maintain a website. Several other companies displaying real-time visual synchronization now participate in the market, but none had the visual variety and potential of Sound Spectrum.
The expectation of real-time performance was an impressive aspect for this kind of software, and it only became possible with the development of fast computer processing power. In fact, the appearance of real-time is still an illusion...but it's fast enough. The real-time feature meant that the software would display a moving graphic/object -- or process the modification of an object -- "at the same time" as the auditory and visual stimuli were being perceived. This real-time feature eliminated the traditional calculations that had to be performed to synchronize the two stimuli. With this new approach, synchronization of the visuals to the music was built into the software product. And, the synchronization was flexible, meaning it could follow the natural tempo variations that occur during the live performance of music.
When used with a musical group, the visual response could be isolated to a single member (with single micing or direct line-level inputs in the case of electrical pickups) or applied to the overall sound emanating from the group via overhead micing. Music majors with an aptitude for analytical geometry could see this as a new career path, and math students with a penchant for music could demonstrate a new, employable skill. Support for the proliferation of this new application would come from major hardware companies that already serviced the entertainment industry. Or so I thought.
I was naive to believe that the addition of visual graphics to the performance of music represented a new, middle ground where the arts and sciences could merge, that audience members of the classical idiom now had something to watch while they listened so intensely ...something that added to the auditory experience. I envisioned academia adding new departments to their ivory towers as they incorporated this discipline into their curricula, and hardware manufacturers would embrace this technology since it offered a new application for their products.
I was wrong on all accounts. Outside of bars and night clubs, the only industries to adopt the real-time generation of graphics into their environment were traveling, pop concert producers and the occasional museum. Road crews traveling with big-name performers had a few more pieces of equipment to load and take care of. It was just one more thing that could go wrong during a concert...as if anyone could tell. Companies like Roland that tried to incorporate real-time software into its product line didn't find a ready market. But in the pop music culture, the pulsing graphics added one more distraction, one more stimulant to be attached to the quantitized beat...which is exactly what it wanted.
In today's classical concert environment, projection mapping is the new "wow" and has taken over the use of synchronized visualization software-- as defined herein. Accompanying visuals, if used, are more likely to be prepared by a software technician proficient in AfterEffects, Premier and 3D animation platforms, rather than a musician with a secondary degree mathematics. A relaxed form of synchronization is cued and adjusted by a stage crew member (with a more sophisticated title.) Visualization software, if it's used, is more likely to be a loop of pre-recorded video. The overall effect is devoid of any profundity and contributes nothing to an understanding -- or the aesthetic impact -- of the music. In the pop scene, the visuals are intended to add shock value rather than artistic nuance. They are the jingles of the visual medium; something catchy. It's all about marketing...not art.
In my support for the visualization of music, I had forgotten that "cool" eventually becomes passee. I was pathetically naive about the power of marketing. In an era where the producer/marketeer -- not the artist -- is king, where "sampling" is embraced and disassociated from the concept of theft, where the highest paid musician readily admits -- and demonstrates at every turn -- that he or she cannot play an instrument or actually sing, I have severely misjudged the aesthetic values of our culture. None the less, I present my feeble attempts to use visualization within the classical music idiom.
If Round (Excerpt)
There is this long-standing admonition in poetry to avoid the abstract, which explains the ubiquitous use of metaphors and the words like, and as. That's why "Love is like a red, red rose, and "Hope is the thing with feathers." However, I found I could challenge that caveat and remain in the abstract throughout the poem by adding performance graphics to the mix.
I also discovered that the addition of performance graphics diminished the tolerance for repetition. A musical line, phrase, motiff, etc. is amenable to being sliced up, inverted, repeated and developed over an extended period of time. But the tolerance for that kind of manipulation with a visual expression is much shorter. While the repetition of themes, patterns, motiffs, movements, etc., are common in music, they quickly becomes intolerable when portrayed visually. The eye seems to tire more quickly than the ear.
If Round
was composed for a piano composition contest that required the entry to be based on poetry. I remembered Stravinsky's proclamation that "music is, by its very nature, essentially powerless to express anything at all, whether a feeling, an attitude of mind, or psychological mood, a phenomenon of nature, etc." Therefore, embedding the text into an accompanying visual component, or at least having the text read by a narrator, was the best way to satisfy the essential requirement for participation in this contest. Otherwise, the claim "based-on" would be a sham.
Obviously, I did not win or place. To be honest, the compositions from other entries were much better, and when I pointed out that the other contestants had not adhered to the stipulation that the music was to be based on poetry, I was ignored. So much for my musicology training and Stravinsky's admonition against imposing learned-associations on music. "Expression has never been an inherent property of music. That is by no means the purpose of its existence," he wrote. I still believe Stravinsky was correct.
This composition, titled "If Round" is about shapes. How's that for being abstract? The square blocks at the beginning evolve from straight lines in the preceding imagery as the first line of the poem asks "What good has any shape found?" The blocks, in turn, transform into round shapes as the poem settles on an examination of round shapes. The rotating coil image in the beginning is responding in real-time to the pitch of the piano notes. Different areas of the coil are programmed to respond to different frequencies coming from the piano, and that response to varying frequencies can be seen once the coil rotates to a horizontal position.
The program "If Round" offers three options for performance: 1) The text can be read live by a narrator during performance; 2) The text can be generated by the visualization software program, but the effects imposed on the text will be limited; 3) A more visually sophisticated rendition of the text can be generated from a second computer (as shown in the video above), where the graphic output of two computers are combined through a video mixer.
Edge (Full)
For example, the piano has an edge….a physical edge. If you extend your hands to the extreme ends of the keyboard, they come to an edge beyond which they fall off the piano. This piece of music gradually pushes the hands to the extreme edges of the piano. The best place to watch a performance of this piece would be directly above the pianist, where the placement of the hands will define the edges established by the music. And, from that overhead perspective, it would be possible to observe the gradual extension of those edges during the course of the piece.
At the beginning of this piece, the tonal edges are very close...barely a note apart. First comes a hammered grace note; then as a trill. The notes are so close together they produce a sharp edge, which in musical terminology is called "dissonance". Those edges gradually expand … to the left, to the right, outward, then collapsing back, and outward again. As the piece progresses, this dissonance abates, and the edges becomes less sharp and more mellow.
With the addition of the visual component, the manifestation of the "edge concept" includes the physical sides of the screen and the shape and placement of the images.
The first image -- disregarding the titles -- is a straight line, the simplest edge. The first notes of the piece are placed towards the middle of the keyboard and are so close together they form a very distinct edge, both physically and aurally. As the notes expand the aural edges, the images move outward from the single, vertical line, thereby extending the visual edge. Within the first, few moments of the piece, the visual edges change orientation from horizontal to vertical, and the surrounding braces form an artificial edge on an area of the screen that is defined only for a moment. The visual edge is in flux.
Visual edges can also be found in the horizontal and vertical striations of colors. For example, in the middle of the piece there is a large, moon-shaped background that is graded in to multiple shades of the same color. The edges of each shade are clearly visible.
Where one thing begins and another ends, there is an edge.