Establishing meaningful cut points for online user ratings

Subjective perceptions of websites can be reliably measured with questionnaires. But it is unclear how such scores should be interpreted in practice, e.g. is an aesthetics score of 4 points on a seven-point-scale satisfactory? The current paper introduces a receiver-operating characteristic (ROC)-based methodology to establish meaningful cut points for the VisAWI (visual aesthetics of websites inventory) and its short form the VisAWI-S. In two studies we use users' global ratings (UGRs) and website rankings as anchors. A total of 972 participants took part in the studies which yielded similar results. First, one-item UGRs correlate highly with the VisAWI. Second, cut points on the VisAWI reliably differentiate between sites that are perceived as attractive versus unattractive. Third, these cut points are variable, but only within a certain range. Together the research presented here establishes a score of 4.5 on the VisAWI which is a reasonable goal for website designers and highlights the utility of the ROC methodology to derive relevant scores for rating scales. Practitioner Summary: We demonstrate the benefit of ROC-based methods by finding relevant cut points for online user ratings. Specifically, we establish that a score of `4.5' as a meaningful cut point for the VisAWI, a scale for measuring aesthetic appeal of websites and its short form, the VisAWI-S.