Essays /

123 Dfrd3 Essay

Essay preview

How to evaluate Public Displays
Florian Alt1 , Stefan Schneegaß1 , Albrecht Schmidt1 ,
Jörg Müller2 , Nemanja Memarovic3

VIS, University of Stuttgart


Telekom Innovation Laboratories, TU Berlin

[email protected]

University of Lugano

[email protected]
After years in the lab, interactive public displays are finding their way into public spaces, shop windows, and public institutions. They are equipped with a multitude of sensors as well as (multi-) touch surfaces allowing not only the audience to be sensed, but also their effectiveness to be measured. The lack of generally accepted design guidelines for public displays and the fact that there are many different objectives (e.g., increasing attention, optimizing interaction times, finding the best interaction technique) make it a challenging task to pick the most suitable evaluation method. Based on a literature survey and our own experiences, this paper provides an overview of study types, paradigms, and methods for evaluation both in the lab and in the real world. Following a discussion of design challenges, we provide a set of guidelines for researchers and practitioners alike to be applied when evaluating public displays.

Figure 1: Studying public displays in the lab (left) / field (right) hence providing opportunities for researchers to create novel interaction techniques. Hence, there is an emerging need for both practitioners and researchers, to understand how to best evaluate public displays with regard to effectiveness, audience behavior, user experience and acceptance, and social as well as privacy impacts. Since today no commonly accepted guidelines exists as to how (applications for) public displays should be designed, the evaluation of these is both crucial and challenging for several reasons. First, display deployments are often opportunistic. As new infrastructure or real estate is being created, the premises are often augmented with public displays, having only little knowledge of the audience. Second, simulations of the environment a display is deployed in are difficult, as there are no (dynamic) models yet (e.g., of the stream of visitors passing through a pedestrian area). As a result, evaluation has to be conducted in context (both in the real world and in the lab). Third, there is not one single goal that public displays (or their content) try to achieve. Ads most likely strive for maximizing attention, interactive games may want to create an engaging experience, informative applications (e.g., a public transport schedule) may aim at maximizing usability, and some displays may be deployed to show warnings to passers-by or support the fast evacuation of a building. Fourth, measuring the effectiveness of a display is difficult. Compared to the internet, it is often not possible to monitor user interaction, but sensors might in the future allow to extract richer information based on the interaction. This, however, might raise privacy concerns (e.g., when using a camera in public space), hence determining and restricting the means for evaluation. In order to tackle these challenges, we set out to provide guidelines for evaluating public displays. Our work is grounded in a comprehensive literature survey, based on which we identified common study types, paradigms, and methods including their respective advantages and disadvantages. We discuss and validate them, ultimately deriving guidelines that can help researchers and practitioners to choose an evaluation method for their public display.

Public Displays, Evaluation, Digital Signage, Methods

Categories and Subject Descriptors
H.5.2 [Information Interfaces and Presentation]: User Interfaces—Evaluation/methodology

General Terms
Experimentation, Human Factors



Interactive public displays are leaving the labs and are being deployed in many places. Nowadays, they permeate public spaces, shop windows as well as malls, workspaces, and public institutions and are equipped with sensors, such as cameras, that enable presence and motion sensing. At the same time, new (consumer) devices and software enter the market (e.g., the Microsoft Kinect), Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. PerDis 2012, June 04 - 05 2012, Porto, Portugal.

Copyright 2012 ACM 978-1-4503-1414-5/12/06 ...$10.00.




esed qualitatively based on subjective feedback, e.g., in focus groups to collect the target group’s view and concerns [11] or quantitatively based on questionnaires [23].

Our research is based on an extensive literature review with the goal to identify methods and tools that are used to evaluate public display. As of January 2012, 522 papers can be found in the ACM Digital Library that are concerned with public displays or digital signage. Most of these papers evaluate their concepts and deployments. Even nowadays, more than 30 years after Hole-in-Space1 , one of the first public display installation, neither design guidelines for public displays exists, that cover a broad spectrum of systems and applications, nor do generally accepted evaluation guidelines. However, several ideas have been around in recent years.

Cheverst et al. [11] reported on challenges of evaluating situated displays deployed in a community setting. Storz et al. published lessons learned from the deployment of the eCampus public display network [36] that provides useful information for informing the design of public display (networks) but only little information with regard to evaluation. Mankoff et al. [20] looked at the evaluation of ambient displays, focusing mainly on effectiveness and usability. Starting from Nielsen’s usability heuristics, they created a modified set to be used for the evaluation. Finally, Matthews et al. [21] used activity theory to evaluate peripheral displays. They identified an initial set of evaluation metrics (appeal, learnability, awareness, effects of breakdowns, and distraction), that vary depending on the importance of the display, but do not focus on evaluation methods. Though many research papers provide useful lessons learned or recommendations based on their findings (e.g., [15, 36]), most previous work either focusses on a rather specific application domain (community/situated displays, ambient displays), draws from findings of their deployment(s) only, or treats evaluation only on the side / on a high level. To overcome these limitations, we base our findings on a comprehensive literature review, identifying research questions, research types, and approaches to research and methods used in public display research.

Table 1 provides a summary of research projects classified by the tackled research questions. Furthermore, we distinguished whether the evaluation was conducted prior to creating a prototype by asking users or running an ethnographic study, or whether a prototype was evaluated in a lab study, field study, or in the context of a deployment (see section 4.3 Paradigms).



• User Performance: Measuring effectiveness from a user’s perspective is often done when evaluating novel interaction
techniques, e.g., based ...

Read more


-1 -1414 -2 -4503 -5 -8 /12/06 /en/home/ /getty/his/ 01 03 04 05 08 09 1 1.0 10 10.00 1021 1030 11 12 12.0 12.5 12.7 123 13 137 139 14 14.3 146 15 15.0 16 16.0 169 17 176 18 183 19 1988 1994 2 2.9 20 2001 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 206 21 214 215893 22 221 228 2287 2296 23 24 243 244011 25 26 26.9 261 27 270 28 285 29 29.5 294 3 30 31 32 32.0 33 34 35 35.5 36 36.0 37 37.4 38 38.4 39 4 4.1 4.1.1 4.1.2 4.1.3 4.2 4.3 4.3.1 4.3.2 4.3.3 4.3.4 4.3.5 40 42 457 460 47 49 5 5.0 5.9 522 58 6 6.0 6.1 6.2 7 791 798 7th 8 80 9 978 abl abstract accept accompani accord account achiev acknowledg acm action activ ad adapt address adher adj advantag advertis advis agreement aim al albrecht alik allow almer almost also alt alt1 ambient ame ami amiquin analysi analyz and/or andpop annot anonym anoth answer app appeal appear appli applic approach appropri area around artifact ask aspect ass1 assess assum assumpt attent attract attrakdiff attribut audienc augment autom automat autopoies averag avg awar b bailli balakrishnan ballaga bangor base baselin baudisch baur bear beaudouin beaudouin-lafon becom bederson begin behav behavior benefit benko berlin besid best beyer bial bias bind blind borcher bore breakdown brereton briefli brignul broad build built busi business-ori butz buzeck c camera campus cannot capabl carri cart case categor categori caus causal center centr certain challeng chang characterist check chehimi cheverst chi children choos citat citi citywal classic classifi classroom clear click code coffe coher collect collomb combin come comment commerci common communal communic communiti community/situated compar complet complex compon comprehens compromis comput concept concern conclus condit conduct confound consid consider consum content context contextu continuum contrast control conversi copi copyright correl could covari cover creat credit critic crucial cscw cue cultur cumbersom current custom cylindr d dalton data davi day dec decid decreas deeper degre demonstr depend deploy deployment-bas deploymentbas depth deriv describ descript descriptor design detect determin develop devic dey defin dfrd3 diesel differ difficult dig digit digifi direct disadvantag discov discuss display distant distinguish distract distribut dix document domain done door downsid drag drag-and-drop draw drop druin drunk due dynam döbelt e e-campus e.g ea earli ecampus ecolog econom ed effect effort eiderbäck either elhart emerg empir enabl encompass engag engin enhanc enter entic entir environ environment envis equip error ese especi estat et etc ethic ethnograph ethnographi european evacu evalu evaluation/methodology evan even ever everi everyday evolut examin exampl except exclud exel exercis exist expect expens experi experiment explain explicit explor extens extern extract eye f fact factor fail famili fantasi fast fatah fee feedback feng field figur final finney first fitton fitzmauric fitzpatrick florian focus focuss follow footag form format foster found fourth framework freehand friday front full fume function funsquar furthermor futur g galani game gather gaze gellersen gen general generaliz generat get gi give given glanc glass go goal good grant ground group guidelin gustafson h h.5.2 hall hand hands-on hansen happen harbo hard hardwar hart hascoët haulsen hazlewood hci head heidelberg heikkinen help henc herm hertrich heurist hide high high-level higher hinder hoc hochheis hole hole-in-spac hole-in-space1 honeypot honor hosio hotmobil hotspot hour howev hsieh huang huber human human-comput hutchinson hypothes hypothet i.e iciw idea identifi identifi ieee ilmonen impact implic implicit import impos imposs improv inc incent includ increas index indistinguish inform infrastructur initi inkpen innov insight inspir instal institut integr intellig intend intent interact interest interfac intern internet interven intervent interview introduc introduct investig involv influenc isakov isol issu iter izadi j jacucci januari jay [email protected] john journal june jungl jurmu jörg k kaufmann keyword khan kientz kind kinect kleimola klose knowledg kortum kostako koster kray kruger krüger kubitza kukka kurtenbach kurvinen l lab laboratori lack lafon laitinen land langheinrich larg large-scal last lastnam lazar lead leader learn learnabl least leav led leder lee left legal leisur leisure-ori lesson level lewen librari life light lightspac like limit lindén list liter literatur littl live load local locat log london long long-term longer longitudin look lot low lugano lure m mackay made main mainten major make mall mani mankoff mannequin manual market marshal matejka mattheiss matthew maxim may mean measur med media mediat memarov memarovic3 mental mere meschtscherjakov method metric mich microsoft might miller mine minim mirlach mobid mobil model modifi monitor month morgan morn morrison most motion motiv move movement much multi multi-touch multipl multitud mum municip must müller müller2 n nasa nasa-tlx nasatlx natur necessari need neither nemanja [email protected] network new nielsen nigel night nischt nnub note notic notion novel novelti nowaday number ny o object observ obtain obvious occur often offic ojala one opinion opportun opportunist opposit optic optim order orient ortel other otherwis oulasvirta overal overcom overview own p page paletta paper paper-bas paradigm parallel parisi part partial particip particular pass passedbi passer passers-bi past pedestrian peltonen penn peopl perdi perform period peripher perman permeat permiss permit persist person perspect pervas phase phenomenon phone phonecam phonecam-bas phonetouch photo pick pictur place plaisant platform play point poor popular porto portug posit possess possibl post post-hoc potenti power practic practition pre pre-defin prefer premis prepar presenc present prevent previous principl prior priorit privaci privacywar privat probe problem proc process programm progress project projector promot prompt proper properti prototyp provid profit public publish pure push push-andpop qualit quantit quantifi question questionnair r rais random rare rate rather rattenburi raycast reader real real-world realism realist realiti realiz realli reason receiv recent recognit recommend record redhead redistribut reduc refer refut regard reitberg relat relationship remov replic report represent republish requir research resolut resourc respect restrict result reveal review refin reflect reflectivesign richard richer rid right rigor risk road rodden roger roh role rouncefield roussel rukzio rule run saarikko sacrific salovaara sampl sas scale schedul schieck schiew schmidt schmidt1 schneegaß1 school schrammel scientific screen sd second section see seen select self self-gener semi semi-structur sens sensit sensor server servic session set sever shape share sheridan shirazi shoemak shoot shop short shorter show shown shut side sign signag signific similar simul sinc singl situat size small social socialis softwar solut sometim son soon space space1 specific spectrum spend spot spotlight springer springer-verlag stage stakehold standard start state statist staveland stefan storz stream strike strive structur studi stuttgart sub subject subquest sufficient suit suitabl summari summat support surfac survey sweep sweet swim system tabl tackl take taken target task taylor technic techniqu technolog telekom term test thank theoret theori therefor thing third though three throughout thumb thus time time-consum tlx today tool touch touchprojector toward tradit trajectori transit transport treat tremend tri triangul trigger tscheligi tu turn twink two type typic ubi ubi-hotspot ubicomp ubioulu ubiquit uist ultim uncov understand union univers unknown urban us usa usabl usag use user usual v valid valu valuabl vari variabl variat various vast verlag version video video-bas view viewer vis vision visitor vogel vol volum w wall wall-siz walter want warn way well well-found westerlund wherea whether wide wild wiley wilmsmann wilson window within within-subject without work workload workspac world would wray y year yet york zaidan zone zurmaar field files filming final find finding first firstname fix