CMJ Editor's Notes

Adrian Freed Berkeley, California, USA adrian@CNMAT.Berkeley.edu This is a plea to the computer music community to aim higher in developing new graphical tools and better graphical user interfaces (GUIs) for computer music applications. It has been ten years now since user interfaces such as Digidesign Inc.'s Sound Designer and my own MacMix were designed. It is distressing to see how many recently built programs are simple permutations of the same graphical elements. The interactive techniques used in MacMix (direct manipulation with mouse of 2-D polygonal objects in overlapping rectangular windows) were obsolete when I used them by at least a decade. My defense is that computing performance of the day did not afford more exciting techniques. Now the situation is completely turned around. With RISC, computing performance exceeds the requirements of the old bitmap 2-D graphics paradigm by a huge factor. First we should dismantle the roadblocks hindering more interesting use of GUIs--portability and compatibility--as these requirements have led to conservative choices. Of course, as researchers, we have no excuse for favoring these requirements over innovation. Fortunately the personal computer and workstation industry has just successfully rallied around a strong, reliable, and portable standard for 3-D graphics: OpenGL (Neider 1993). Among features especially relevant to the computer music developer, OpenGL includes: * a client/server architecture; * strong integration with X and other windowing systems, such as Microsoft Windows and the Apple Macintosh Toolbox; * a broad range of geometric primitives including points, lines, polygons, images, and bitmaps; * Ianguage bindings for C and C++; and * only 120 basic library functions. As for device compatibility, I believe there is little point supporting small screens or monochrome screens. The price-leading monitors are now color. Most new computers have enough video RAM for 16-bit color. By the time new software matures, computers will have 24-bit, million-pixel graphics support as standard features. Proposals Following is a list of suggestions for future GUIs for computer music. 1. Thoughtful use of color is required--not just "pastel desktop and menus" or color spectrograms (Wolff and Yaeger 1993). Page 237 of Technical Note 3, "Visualization of floating-point data" illustrates some of the pitfalls here. Color is a virtually unexplored and potentially powerful element in "music visualization." 2. Three-dimensional object representations--with careful use of perspective, fog, depth cuing, and transparency--aid in the manipulation and processing of large numbers of objects (see e.g., Card, Robertson, and Mackinlay, 1991; Mackinlay, Robertson, and Card, 1991; Robertson et al. 1991, 1993; and Bier et al. 19931. The multidimensionality of musical data begs for higher dimensions of control and representation that are impossible with the current "paper on a desktop" metaphor. 3. Abandonment of the "data in files" model in favor of the mature database and multidimensional search techniques used in geometric modeling, as described by Nielson and coworkers (1994) and Matousek (1994). Many of the interesting questions we are asking are about properties of very large collections of different objects, e.g., databases of analyzed sounds of a musical instrument over many pitches, loudnesses, and playing styles. We need to move beyond limited, filebased, single-data-type applications toward models that support richer data types, visualization paradigms, and distributed storage, such as the model behind the rapidly evolving World-Wide Web. 4. Support for input devices with higher-control bandwidth and dimensionality than the mouse, as in Bier (1993). We need to integrate new kinds of keyboards, a broader range of physical gestures, vocalization, and even non-human control sources. 5. Integration of real-time sound synthesis throughout the graphical interface, e.g., cursors on a frequency axis that synthesize a continuous reference tone as they are moved (Gaver and Smith 1990; Gaver, Smith, and O'Shea, 1991; Gaver 1993). Suggested Reading 3-D Interface Paradigms These references describe some of the recent work lat Xerox PARC and elsewhere) on advanced software interfaces for the presentation and manipulation of complex and very large information structures. Bier, E. A., et al. "Toolglass and Magic Lenses: The See-through Interface." Proceedings of the 1993 SIGGRAPH Conference. New York: Association for Computing Machinery pp. 73-80. Card, S. K., G. G. Robertson, and J. D. Mackinlay. 1991. "The Information Visualizer, an Information Workspace." Proceedings of the 1991 ACM Conference on Computer Human Interaction. New York: Association for Computing Machinery, pp. 181-188. Mackinlay J. D., G. G. Robertson, and S. K. Card. 1991. "The Perspective Wall: Detail and Context Smoothly Integrated." Proceedings of the 1991 ACM Conference on Computer Human Interaction. New York: Association for Computing Machinery, pp. 173-179. Robertson, G. G., S. K. Card, and J. D. Mackinlay. 1993. "Information Visualization using 3-D Interactive Animation." Communications of the ACM 36(4): pp. 56-57. Robertson, G. G., J. D. Mackinlay, and S. K. Card. 1991. "Cone Trees: Animated 3-D Visualizations of Hierarchical Information." Proceedings of the 1991 ACM Conference on Computer Human Interaction. New York: Association for Computing Machinery, pp. 189-194. Auditory Icons William Gaver has developed several software user interfaces to integrate non-speech audio into the Macintosh Finder and the Xerox PARC Alternate Reality Kit. Gaver, W. 1993. "Synthesizing Auditory Icons." Proceedings of INTERCHI '93, pp. 228-235. Gaver, W., and R. Smith. 1990. "Auditory Icons in Large-scale Collaborative Environments." In D. Diaper et al. (eds.), Human-Computer Interaction--INTERACT'90. North-Holland: Elsevier Science Publishers B. V., pp. 735-740. Gaver, W., R. Smith, and T. O'Shea. 1991. "Effective Sounds in Complex Systems: The ARKola Simulation." Proceedings of CHI'91, pp. 85-90. OpenGL The OpenGL graphics interface library is stable and is well documented in the following references. Neider, J. OpenGL Programming Guide: The Official Guide to Learning OpenGL, Release 1. Reading, Massachusetts: AddisonWesley, 1993. OpenGL Reference Manual: The Official Reference Document for OpenGL. Reading, Massachusetts: Addison-Wesley 1993. Data Representation and Access These problems, addressed in the scientific visualization community, are the same ones we face in timbre representation: Matousek, J. 1994. "Geometric Range Searching." ACM Computing Surveys 26(4):421-461. Nielson, G. M., et al. 1994. "Research Issues in Data Modeling for Scientific Visualization." IEEE Computer Graphics and Applications 14(2):70-73. Tips and Pitfalls Graphics programming offers its own special challenges, but a mature literature on effective and efficient graphical programming techniques is now available: Arvo, J. (ed.). 1991. Graphics Gems 11, Academic Press. Glassner, A. (ed.). 1990. Graphics Gems, Academic Press. Heckbert, P. (ed.), 1994. Graphics Gems IV, Academic Press (includes Macintosh or IBM-PC floppy disk). Kirk, D. (ed.). 1992. Graphics Gems 111, Academic Press (includes Macintosh or IBM-PC floppy disk). Wolff, R. S., and L. Yaeger. 1993. Visualization of Natural Phenomena. Santa Clara, California: Telos Publishers. Encouraging Beginnings The following papers were part of a 1995 International Computer Music Conference session devoted to GUIs for computer music. They are included in the Proceedings of the conference. Bargar, R., et al. "Defining Spectral Surfaces." Choi, I., R. Bargar, and C. Goudeseune. "A Manifold Interface for a High Dimensional Control Space." Polfreman, R., and J. Sapsford-Francis. "A Human Factors Approach to Computer Music Systems User-Interface Design." Taube, H., and T. Kunze. "Capella: A Graphical Interface for Algorithmic Composition." Conclusion I hope this note spawns wide discussion of GUI design for computer music applications and hereby offer an electronic forum to track these discussions--http://www.cnmat. berkeley.edu/~adrian/GUIforCM. I welcome and look forward to your contributions.

	Vol. 19 Issue 4 Editor's Notes	CMJ Library >
	Improving Graphical User Interfaces for Computer Music Applications
	Adrian Freed Berkeley, California, USA adrian@CNMAT.Berkeley.edu This is a plea to the computer music community to aim higher in developing new graphical tools and better graphical user interfaces (GUIs) for computer music applications. It has been ten years now since user interfaces such as Digidesign Inc.'s Sound Designer and my own MacMix were designed. It is distressing to see how many recently built programs are simple permutations of the same graphical elements. The interactive techniques used in MacMix (direct manipulation with mouse of 2-D polygonal objects in overlapping rectangular windows) were obsolete when I used them by at least a decade. My defense is that computing performance of the day did not afford more exciting techniques. Now the situation is completely turned around. With RISC, computing performance exceeds the requirements of the old bitmap 2-D graphics paradigm by a huge factor. First we should dismantle the roadblocks hindering more interesting use of GUIs--portability and compatibility--as these requirements have led to conservative choices. Of course, as researchers, we have no excuse for favoring these requirements over innovation. Fortunately the personal computer and workstation industry has just successfully rallied around a strong, reliable, and portable standard for 3-D graphics: OpenGL (Neider 1993). Among features especially relevant to the computer music developer, OpenGL includes: * a client/server architecture; * strong integration with X and other windowing systems, such as Microsoft Windows and the Apple Macintosh Toolbox; * a broad range of geometric primitives including points, lines, polygons, images, and bitmaps; * Ianguage bindings for C and C++; and * only 120 basic library functions. As for device compatibility, I believe there is little point supporting small screens or monochrome screens. The price-leading monitors are now color. Most new computers have enough video RAM for 16-bit color. By the time new software matures, computers will have 24-bit, million-pixel graphics support as standard features. Proposals Following is a list of suggestions for future GUIs for computer music. 1. Thoughtful use of color is required--not just "pastel desktop and menus" or color spectrograms (Wolff and Yaeger 1993). Page 237 of Technical Note 3, "Visualization of floating-point data" illustrates some of the pitfalls here. Color is a virtually unexplored and potentially powerful element in "music visualization." 2. Three-dimensional object representations--with careful use of perspective, fog, depth cuing, and transparency--aid in the manipulation and processing of large numbers of objects (see e.g., Card, Robertson, and Mackinlay, 1991; Mackinlay, Robertson, and Card, 1991; Robertson et al. 1991, 1993; and Bier et al. 19931. The multidimensionality of musical data begs for higher dimensions of control and representation that are impossible with the current "paper on a desktop" metaphor. 3. Abandonment of the "data in files" model in favor of the mature database and multidimensional search techniques used in geometric modeling, as described by Nielson and coworkers (1994) and Matousek (1994). Many of the interesting questions we are asking are about properties of very large collections of different objects, e.g., databases of analyzed sounds of a musical instrument over many pitches, loudnesses, and playing styles. We need to move beyond limited, filebased, single-data-type applications toward models that support richer data types, visualization paradigms, and distributed storage, such as the model behind the rapidly evolving World-Wide Web. 4. Support for input devices with higher-control bandwidth and dimensionality than the mouse, as in Bier (1993). We need to integrate new kinds of keyboards, a broader range of physical gestures, vocalization, and even non-human control sources. 5. Integration of real-time sound synthesis throughout the graphical interface, e.g., cursors on a frequency axis that synthesize a continuous reference tone as they are moved (Gaver and Smith 1990; Gaver, Smith, and O'Shea, 1991; Gaver 1993). Suggested Reading 3-D Interface Paradigms These references describe some of the recent work lat Xerox PARC and elsewhere) on advanced software interfaces for the presentation and manipulation of complex and very large information structures. Bier, E. A., et al. "Toolglass and Magic Lenses: The See-through Interface." Proceedings of the 1993 SIGGRAPH Conference. New York: Association for Computing Machinery pp. 73-80. Card, S. K., G. G. Robertson, and J. D. Mackinlay. 1991. "The Information Visualizer, an Information Workspace." Proceedings of the 1991 ACM Conference on Computer Human Interaction. New York: Association for Computing Machinery, pp. 181-188. Mackinlay J. D., G. G. Robertson, and S. K. Card. 1991. "The Perspective Wall: Detail and Context Smoothly Integrated." Proceedings of the 1991 ACM Conference on Computer Human Interaction. New York: Association for Computing Machinery, pp. 173-179. Robertson, G. G., S. K. Card, and J. D. Mackinlay. 1993. "Information Visualization using 3-D Interactive Animation." Communications of the ACM 36(4): pp. 56-57. Robertson, G. G., J. D. Mackinlay, and S. K. Card. 1991. "Cone Trees: Animated 3-D Visualizations of Hierarchical Information." Proceedings of the 1991 ACM Conference on Computer Human Interaction. New York: Association for Computing Machinery, pp. 189-194. Auditory Icons William Gaver has developed several software user interfaces to integrate non-speech audio into the Macintosh Finder and the Xerox PARC Alternate Reality Kit. Gaver, W. 1993. "Synthesizing Auditory Icons." Proceedings of INTERCHI '93, pp. 228-235. Gaver, W., and R. Smith. 1990. "Auditory Icons in Large-scale Collaborative Environments." In D. Diaper et al. (eds.), Human-Computer Interaction--INTERACT'90. North-Holland: Elsevier Science Publishers B. V., pp. 735-740. Gaver, W., R. Smith, and T. O'Shea. 1991. "Effective Sounds in Complex Systems: The ARKola Simulation." Proceedings of CHI'91, pp. 85-90. OpenGL The OpenGL graphics interface library is stable and is well documented in the following references. Neider, J. OpenGL Programming Guide: The Official Guide to Learning OpenGL, Release 1. Reading, Massachusetts: AddisonWesley, 1993. OpenGL Reference Manual: The Official Reference Document for OpenGL. Reading, Massachusetts: Addison-Wesley 1993. Data Representation and Access These problems, addressed in the scientific visualization community, are the same ones we face in timbre representation: Matousek, J. 1994. "Geometric Range Searching." ACM Computing Surveys 26(4):421-461. Nielson, G. M., et al. 1994. "Research Issues in Data Modeling for Scientific Visualization." IEEE Computer Graphics and Applications 14(2):70-73. Tips and Pitfalls Graphics programming offers its own special challenges, but a mature literature on effective and efficient graphical programming techniques is now available: Arvo, J. (ed.). 1991. Graphics Gems 11, Academic Press. Glassner, A. (ed.). 1990. Graphics Gems, Academic Press. Heckbert, P. (ed.), 1994. Graphics Gems IV, Academic Press (includes Macintosh or IBM-PC floppy disk). Kirk, D. (ed.). 1992. Graphics Gems 111, Academic Press (includes Macintosh or IBM-PC floppy disk). Wolff, R. S., and L. Yaeger. 1993. Visualization of Natural Phenomena. Santa Clara, California: Telos Publishers. Encouraging Beginnings The following papers were part of a 1995 International Computer Music Conference session devoted to GUIs for computer music. They are included in the Proceedings of the conference. Bargar, R., et al. "Defining Spectral Surfaces." Choi, I., R. Bargar, and C. Goudeseune. "A Manifold Interface for a High Dimensional Control Space." Polfreman, R., and J. Sapsford-Francis. "A Human Factors Approach to Computer Music Systems User-Interface Design." Taube, H., and T. Kunze. "Capella: A Graphical Interface for Algorithmic Composition." Conclusion I hope this note spawns wide discussion of GUI design for computer music applications and hereby offer an electronic forum to track these discussions--http://www.cnmat. berkeley.edu/~adrian/GUIforCM. I welcome and look forward to your contributions.