User Interface Technology

Question 1

User Interfaces

Answer 1

• The user inteface is the part of a system through which a user can interact, composed of software and/or hardware that supports input, output or both.
• Not restricted to digital and interactive systems
• Examples:
• Graphical user interface (GUI) of any app or system,
  e.g. of search engine, or powerpoint
• Voice user interface in a smart speaker (e.g. Alexa)
• Buttons, switches, wheels and levers in a cockpit
• Control panel of everyday devices,
  e.g. microwave oven, alarm cock, …

Question 2

Human-Computer Input/Output

Answer 2

• Input from the user to the system via sensors/controls
• Output from the system via displays/actuators

Question 3

Input Technologies

Answer 3

Question 4

Examples of Input Tasks

Answer 4

• Filling in a form
  
  • Pointing/selection, text input
• Mobile food order
  
  • Media capture, pointing/selecting
• Data-entry device (assignment 3)
  
  • Directly mapped controls
• Whack-A-Mole (assignment 2)
  
  • Pointing/selection

Question 5

Input devices

Answer 5

• “An input device is a transducer from the physical properties of the world into the logical parameters of an application” (B.Buxton)
• Keys, buttons, touch sensors - sensing discrete input
• Pointing devices - sensing spatial input in one or more dimensions
• Input devices can also be based on touchless sensing
  • Audio - for voice control
  • Video - e.g., mid-air gestures, facial expressions, …
  • Depth sensors - body interface, proxemic interaction

Question 6

Control Interfaces

Answer 6

• Conventional control interfaces
• Input/controls: Buttons, knobs, pedals, …
• Output/feedback: Lights, beeps, instruments
• Physical: hard controls
• Single-purpose
• Separate input and output
• Hard-wired control-display relationships

Question 7

Computer Interfaces

Answer 7

• Physical interface
  
  • Multi-purpose I/O devices
  • Keyboard, pointing device, microphone, camera
  • Displays, speaker, printer
• Soft interface
  
  • Graphical controls and views
  • Rendered on physical display
  • Controlled with input devices

Question 8

Hard versus soft interfaces

Answer 8

• Hard controls: input distinct from output, rigid control-display mappings
• Soft interfaces: blending input and output, flexible and dynamic control-display mappings, using general purpose input/output hardware
• Device trends: fewer components, soft controls are cheaper and more flexible
• Physicality remains important for many tasks and situations
  
  • e.g., faster typing, eyes-free input, industrial application, safety-critical

Question 9

Soft interfaces

Answer 9

• Soft interfaces transform possibilities for control and interaction
• More controls in less space
• Use of the same display space can instantly change during the interaction
• Use the same physical input devices for any number of different controls
  
  • Small set of input events
  • What they control depends on the state of the interface at the time of input

Question 10

Blending Input/Output

Answer 10

• Soft interfaces blend the distinction between control and display
• Soft controls are rendered on a display and have the properties of displays
• Feedforward on available actions
  
  • Static labels, Preview on hover
• Feedback on system status
  
  • e.g., Switch on/off, Position on scale
• Feedback during the control action
  
  • e.g., continuous feedback in adjusting values on a scale

Question 11

Control-Display Relationships

Answer 11

• A control-display relationship is a mapping from input to output
• Soft interfaces lead to complex control-display relationships
• Every possible action on a soft control that produces a visible response is a control-display relationship
• Soft controls can respond to different physical control events
  
  • Click, double-click, drag, shift-click, control-click, keyboard shortcut, …
• Controls can trigger different responses (displays) depending on context
  
  • Different modes of the interface

Question 12

Modes

Answer 12

• Modes are different interpretations of the user input by the system, depending on the state of the interface
• Same input, different results
  
  • Caps Lock switches keyboard mode
  • Modes in camera app change effect of shutter button
  • Language modes in text entry effect prediction
• Modeless interfaces
  
  • same user input will have the same result
  • regardless of the system state

Question 13

Usability of Modes

Answer 13

• Modes solve the problem of supporting more
  functions than there are controls
• e.g. Photoshop:
  
  • mouse input in different modes
  • switched by selecting tools (brush, eraser, …)
  • cursor shape to provide feedback on mode
• Usability issues
  
  • Hard to discover how modes are switched
  • Not clear what the current mode is
  • Common with hard control interfaces (appliances)

Question 14

Mode errors

Answer 14

• Mode errors occur when the user is not
  aware of the current mode
• Example: Airbus-320 crash, Strasbourg 1991
• Modes for specifying descent as FPA (angle) or VS (vert. speed)
• Pilot input “-3.3” assuming FPA mode (same as 800 ft/min)
• But the control was in VS mode, with the input interpreted as -33 x 100 feet/min

Question 15

Pointing devices

Answer 15

• Pointing devices enable input of spatial data, for a wide range of tasks
  
  • Object selection, positioning, manipulation, drawing, quantifying input, …
• Pointing devices are characterized by:
  
  • Continuous sensing in motor space (= input space)
  • Continuous mapping of the sensed property to control a point in display space (= output space)
  • A specific control-display relationship defined by a transfer function

Question 16

Devices and Technologies

Answer 16

• “Classic” pointing devices for graphical user interfaces (2D)
  
  • Mouse for pointing on desktop computers
  • Touchpad for pointing on laptops
  • Joystick on game controllers
  • Trackball (“mouse upside down”)
  • Pointing stick (trackpoint)
• Touchscreen and stylus are also pointing devices (in addition to other functions)
• Pointing devices for 3D environments (6 DOF)
• Eye trackers for gaze pointing (accessibility), foot controls, …
• Sensors in mobile devices (e.g. accelerometer, 3DOF), depth cameras (body input)

Question 17

Direct/Indirect Control

Answer 17

• Direct-control: pointing directly on the display
  
  • Input space / motor space = output space
    (1:1 mapping)
    
    • Touchscreen (single-touch, multi-touch)
    • Passive stylus
    • Active stylus (sense pressure, tilt; button
      to mode switch)
• Indirect-control: input space separate from the display
  
  • Motor space is separate from display (can
    have a gain factor)
  • Device representation by a marker/cursor
    on the display
    
    • e.g. Mouse, Touchpad, Graphics tablet

Question 18

Absolute / Relative Input

Answer 18

• Input with indirect control devices can be classified as absolute or relative
• Absolute mapping of input: a point in motor space corresponds to a point in the display space
• e.g. Graphics tablet
• Relative mapping: input is relative to the current position in display space
  
  • Can be performed from any position in motor space
  • Mouse (cannot sense absolute position, only displacement)
  • Touchpad (although it can absolute position)

Question 19

Control-display gain (CD gain)

Answer 19

• Control-display gain is a scale factor in mapping input to movement on the display:
• Example
  
  • Mouse moves 3cm and cursor moves 3cm: CD gain = 1.
  • Mouse moves 3cm and cursor moves 6cm: CD gain = 2.
• Absolute input may be scaled with a fixed CD gain
• Relative input can involve dynamic CD gain, based on a transfer function

Question 20

Transfer function

Answer 20

• The transfer function describes the mapping of input from motor space to display space
• Transfer functions for the mouse are highly
  optimized and non-linear
  • CD gain dependent on mouse acceleration
  • Increasing gain in the acceleration phase,
    for gross positioning
  • Reducing gain when the user decelerates,
    for fine positioning

Question 21

Property sensed

Answer 21

• Isotonic controllers
  
  • Device is free moving: resistance is low and
    constant
  • Property sensed: displacement
• Isometric controllers
  
  • Device does not move: infinite resistance
  • Property sensed: force
• Elastic controllers
  
  • Resistance increases with displacement
  • Can sense displacement or force

Question 22

Order of control

Answer 22

• The property sensed by an input device can be mapped to control position or velocity
• Position control (= zero-order control)
  
  • Input mapped to position – how far the cursor moves
• Rate control (= first-order control, or: velocity control)
  
  • Input mapped to speed – how fast the cursor moves

Question 23

Comparison of pointing devices

Answer 23

• Touchscreens are fastest but least accurate
• Mouse consistently found to be best for both speed and accuracy
• Touchpad not as efficient but has lower acquisition time from the keyboard
• Pointing stick not as efficient for very small
  movements but users can keep hands in keyboard home position
• People with motor disabilities prefer joysticks or trackballs
  
  • location of the device remains fixed (easier
    reach, wheelchair mounting)
  • can be operated with small residual
    movements
• Touch-sensitive devices require least force but that also increases risk of inadvertent input

Question 24

Pointing Devices - Key Points

Answer 24

• Devices can be classified by how their input is mapped
  
  • Direct/indirect
  • Absolute/relative
  • Position-control/rate-control
  • Transfer function and CD Gain
• All pointing devices have speed-accuracy trade-offs
  
  • They differ in how efficient they are for different tasks
  • For example, arrow keys better for nudging the cursor than the mouse,
    but not for moving the cursor across the screen