Automatic dialog systems have become a mainstream part of online customer service. Many such systems are built, maintained, and improved by customer service specialists, rather than dialog systems engineers and computer programmers. As conversations between people and machines become commonplace, it is critical to understand what is working, what is not, and what actions can be taken to reduce the frequency of inappropriate system responses. These analyses and recommendations need to be presented in terms that directly reflect the user experience rather than the internal dialog processing.

This paper introduces and explains the use of Actionable Conversational Quality Indicators (ACQIs), which are used both to recognize parts of dialogs that can be improved and to recommend how to improve them. This combines benefits of previous approaches, some of which have focused on producing dialog quality scoring while others have sought to categorize the types of errors the dialog system is making. We demonstrate the effectiveness of using ACQIs on LivePerson internal dialog systems used in commercial customer service applications and on the publicly available LEGOv2 conversational dataset. We report on the annotation and analysis of conversational datasets showing which ACQIs are important to fix in various situations.

The annotated datasets are then used to build a predictive model which uses a turn-based vector embedding of the message texts and achieves a 79% weighted average f1-measure at the task of finding the correct ACQI for a given conversation. We predict that if such a model worked perfectly, the range of potential improvement actions a bot-builder must consider at each turn could be reduced by an average of 81%.

The Grand Seaside Hotel is a large five-star hotel in an Australian Coastal town. It is a place that not only aspires to provide excellent service but that also seeks to reconcile two apparently divergent demands: the need for customized service and the efficient management of business operations. To commit staff to the provision of service excellence, management has introduced a customer service programme that relies on various forms of training and rewards, as well as a guest response system. The customer service programme, particularly the use of guest questionnaires, appear as disciplinary strategies that aim to produce service encounters in which both staff and guests are ‘normalized’. The main loci of ethnographic data collection for this paper are regular Management Briefings. Through data collected from these, the paper investigates the use of the ‘imaginary’ in constituting service encounters and guest expectations. It interprets these in terms of Foucault's Panoptic analysis to identify the immanent mechanisms of discipline in these customer service programmes.