Designing across senses : a multimodal approach to product design

What Is This Book About? From the keyboard, mouse, and touchscreen, to voice-enabled assistants and virtual reality, we have never had more ways to interact with technology. Called modes, they allow people to enter input and receive output from their devices. These inputs and outputs are often designed together in sets to create cohesive user interfaces (UIs). These modes reflect the way our senses, cognitive functions, and motor skills also work together in sets called modalities. Human modalities have existed for far longer than our interface modes, and they enable us to interact with the physical world. Our devices are only beginning to catch up to us. We can now jump and move around in our living rooms to play a game using Microsoft’s motion-tracking peripheral, Kinect. We can ask Domino’s to deliver a pizza using the Amazon Echo. We often use several modalities together in our daily activities, and when our devices can do the same, they are considered multimodal UIs. Most UIs are already multimodal, but because they’re so familiar we don’t tend to think of them that way. In fact, almost all designed products and environments are multimodal. We see a door and knock on it, waiting for it to open or to hear someone inside ask who it is. We use our fingers to type on a keyboard, and see characters appear on the screen in front of our eyes. We ask Siri a question and see the oscilloscope-like waveform to let us know we are being heard. We receive a phone call and feel the vibration, hear the ringtone, and see the name of the person on the screen in front of us. We play a video game and are immersed in sensory information from the screen, speakers, and the rumble shock controller in our hands. Multimodal products blend different interface modes together cohesively. They allow us to experience technology the same way we experience our everyday lives: across our senses. Good multimodal design helps us stay focused on what we are doing. Bad multimodal design distracts us with clumsy or disjointed interactions and irrelevant information. It pulls us out of our personal experience in ways that are at best irritating and at worst dangerous. As technology is incorporated into more contexts and activities in our lives, new types of interfaces are rapidly emerging. Product designers and their teams are challenged to blend modalities in new combinations for new products in emerging categories. They are being asked to add new modalities to the growing number of devices we use every day. This book provides these teams with an approach to designing multimodal interactions. It describes the human factors of multimodal experiences, starting with the senses and how we use them to interact with both physical and digital information. The book then explores the opportunities represented by different kinds of multimodal products and the methodologies for developing and designing them. Following this approach will develop multimodal experiences for your users. You will be able to deliver successful products that earn trust, fulfill needs, and inspire delight. Who Should Read This Book This book is for designers who are developing or transforming products with new interface modes, and those who want to. It will extend knowledge, skills, and process past screen-based appraoches, and into the next wave of devices and technologies. The book also helps teams that are integrating products across broader physical environments and behaviors. The senses and cognition are the foundation of all human experience, and understanding them will help blend physical and digital contexts and activities successfully. Ultimately, it is for anyone who wants to create better products and services. Our senses are the gateway to the richness, variety, delight, and meaning in our lives. Knowing how they work is key to delivering great experiences. [Preface] Chapter 1: Returning to Our Senses If a Tree Falls in the Forest... The Sound of Violence Experience Is Physical People Have Modalities Devices Have Modes Human Modalities + Device Modes = Interfaces Physical Information: The New Data Sensing, Understanding, Deciding, and Acting: The New Human Factors Focus: The New Engagement Multimodality Makes a Wider Range of Human and Product Behaviors Possible How Multimodality Affects Design Multimodal Design Is Cross-Disciplinary Summary Chapter 2: The Structure of Multimodal Experiences The Human Slice of Reality: Umwelt and Sensibility Assembling Multimodal Experiences: Schemas and Models The Building Blocks of Multimodal Experience Summary Chapter 3: Sensing The Three Main Categories of Stimuli Defining the Senses: Dimension, Resolution, and Range Sensory Focus: Selecting, Filtering, and Prioritizing Information Reflexes Our Senses and Their Unique Properties Vision Hearing Auditory Interfaces Touch (Somatosensory or Tactile Abilities) Smell (Olfactory Ability) Taste (Gustatory Ability) Sixth Senses and More Summary Chapter 4: Understanding and Deciding The Foundations of Understanding: Recognition, Knowledge, Skills, and Narratives Aware and Non-Aware: Fast and Slow Thinking Agency: Balancing Self-Control and Problem Solving Motivation, Delight, Learning, and Reward: Creating Happiness Summary Chapter 5: Acting About Anthropometrics Task Performance Nonverbal Communication Precision Versus Strength Inferring Versus Designating Intent Summary Chapter 6: Modalities and Multimodalities Modalities: How We Use Our Senses We Shape Our Modalities, and They Shape Us Attributes and Abilities of Modalities Applying Modalities to Design Multimodalities Trusted Version and Performance Optimization Attributes and Abilities of Multimodalities Common Categories of Multimodalities Applying Multimodality to Design Summary Chapter 7: The Opportunity: Transforming Existing Products and Developing New Ones Key Applications of IoT: Monitor, Analyze and Decide, Control and Respond “Disruptive” Technologies Beginning Inquiry Summary Chapter 8: The Elements of Multimodal Design Using Physical Information Constructing Knowledge, Interactions, and Narratives Summary Chapter 9: Modeling Modalities and Mapping User Experiences Behaviors Shared Between Users and Devices Demanding Contexts and Complex Interactions Call for Alignment Experience Maps and Models for Multimodality Key Considerations of Multimodal Design Example Maps and Models Summary Chapter 10: Form Factors and Configurations Creating Multimodal Properties Configuring Interface Modes Mapping Modal Behaviors to Modal Technologies Summary Chapter 11: Ecosystems Device Ecosystems Information Ecosystems Physical Ecosystems Social Ecosystems Specialized Ecosystems Cloud Architectures: Distributing Resources Through Connectivity Ecosystem and Architecture: Applying Ecosystem Resources to Multimodal Design Sensing Experiences: Answering the Door—A Doorbell, Ring, and the August Lock Understanding and Deciding Experiences: Determining Distance—a Pedometer, Apple Watch, and Lyft App Acting Experiences: Writing and Drawing—A Pencil, a Tablet, and the Apple Pencil Summary Chapter 12: Specifying Modalities: States, Flows, Systems, and Prototypes Introduction: A Prototype Is a Custom Measuring Tool Practice Makes Perfect The Media of Multimodal Products: Information and Interactions, Understandings and Behaviors The Product Development Process for Multimodal Products Defining Design Requirements Specifying Multimodalities Summary Chapter 13: Releasing Multimodal Products: Validation and Learning Release Is About Process Alpha Release Beta Release Public Release The Out-of-the-Box Experience Summary Further Reading Glossary [ Index ] Today we have the ability to connect speech, touch, haptic, and gestural interfaces into products that engage several human senses at once. This practical book explores examples from current designers and devices to describe how these products blend multiple interface modes together into a cohesive user experience. Authors Christine Park and John Alderman explain the basic principles behind multimodal interaction and introduce the tools you need to root your design in the ways our senses shape experience. This book also includes guides on process, design, and deliverables to help your team get started. The book covers several topics within multimodal design, including: New Human Factors: learn how human sensory abilities allow us to interact with technology and the physical world New Technologies: explore some of the technologies that enable multimodal interactions, products, and capabilities Multimodal Products: examine different categories of products and learn how they deliver sensory-rich experiences Multimodal Design: learn processes and methodologies for multimodal product design, development, and release "Today we have the ability to connect speech, touch, haptic, and gestural interfaces into products that engage several human senses at once. This practical book explores examples from current designers and devices to describe how these products blend multiple interface modes together into a cohesive user experience. Authors Christine Park and John Alderman explain the basic principles behind multimodal interaction and introduce the tools you need to root your design in the ways our senses shape experience. This book also includes guides on process, design, and deliverables to help your team get started."--Provided by Amazon.com