91ɬÂþ

Abstract

Haptic technologies bring touch-based interactions into computer interfaces, opening new possibilities for entertainment, music and art, virtual reality, education, and accessibility. In many of these applications, haptic interactivity is experienced as part of a multisensory experience, often including complementary visual or auditory elements. Such multisensory design, when done well, can improve the experiences of users, for example by providing richer feedback to performing musicians and creating interactive audio-haptic diagrams for blind and low-vision students. Despite this, most software for haptic design has limited support for other sensory modalities, requiring ongoing effort from the designer to follow a suitably multisensory approach. It should not be surprising then that haptic effects are often added to an experience after visuals or sounds have been created.

This thesis explores possibilities for software to support audio-haptic design throughout the design process, and studies its impact on the practitioners that use it. Instead, we take the approach of focusing on a particular use case—representing visual media for blind and visually impaired individuals—but designing the software with extensibility in mind so that it can be adapted to other applications. Our work to advance audio-haptic design research and the engineering state of the art consists of two tracks:

First, the thesis presents a software architecture for automatically creating audio-haptic representations of visual web content, and studies its ability to meet the needs of accessibility practitioners. The results of retrospective interviews with practitioners who used the architecture are presented to aid others in adapting the system and techniques used therein to their own projects. Our results suggest that the architecture is useful when implementing, refining, and evaluating more mature audio-haptic representations. However, it does not adequately support practitioners engaged in prototyping possible solutions to a problem.

To address this limitation, in the second track we study authoring tools for haptic and audio-haptic design as a means of prototyping. We begin with a systematic review of these tools to identify historical trends likely to be relevant to the users or creators of authoring tools. This is discussed through the lens of a novel taxonomy developed through interviews with practitioners and the review itself. Building upon work in other domains, we developed and evaluated a reinforcement learning computer agent that encourages exploration in haptic and audio-haptic design activities. This agent only trains on user feedback, facilitating its integration into other tools. We identify how such agents can be built for audio-haptics using thematic analyses of interviews with participants in these activities.

Through these efforts, we develop techniques to create software to support audio-haptic design from the brainstorming of new ideas to the deployment of experiences to end users. While some practitioners may be able to use the artifacts unchanged, our results provide useful guidance to those working outside our particular context.