The DialPort Portal
The DialPort Portal is an on-going NSF-funded research project built by the Dialog Research Center (DialRC) at Carnegie Mellon's Language Technologies Institute. The primary aim of the DialPort Portal is to allow dialog researchers to collect data with real users.
To best assess the strengths and weaknesses of a dialog system, it is imperative to test in a realistic setting: a back-and-forth interaction with a real user. Evaluating a dialog system on a static dataset does not allow us to sufficiently understand it's strengths and weaknesses, particularly in terms of dialog-level qualities such as (1) coherence, (2) consistency, (3) topic depth and (4) error recovery. Furthermore, paid users (i.e., Amazon Mechanical Turk workers) will often tolerate errors in a system in order to quickly finish a HIT. In contrast, real users will be less tolerant and therefore better reflect system errors. The ultimate goal of dialog system research is to have back-and-forth interactions with real users, the Portal allows the dialog research community to evaluate systems in the same manner.
When systems are collecting data, you can check out the DialPort Portal and interact with the live systems! If you are interested in using the DialPort Portal for your research, please do not hesitate to reach out.
DialPort Interface
Previous NextA screenshot of the DialPort Portal interface. Users, generally recruited through Facebook Advertising, come to the Portal website and are randomly paired with a dialog system. They can have a back-and-forth interaction with the dialog system. At any point, the user is able to provide feedback to the system developers in the form of a like/dislike, a written review or a correction of the system response.
Users tend to have long conversations (7+ turns on average) and leave a lot of feedback! The long conversations and the user feedback was very valuable to the participants in our DSTC9 track!
Research on the DialPort Portal
The DialPort Portal has been a valuable tool for the dialog research community. In total, the DialPort Portal has collected 11,991 dialogs and 94,317 turns with real users. Over 20 different systems have been assessed on the DialPort Portal. Below are a few examples of research that used the DialPort portal.
SlugBot: Alexa Prize Challenge Dialog System
SlutBot is an Alexa Prize System developed by the University of California at Santa Cruz. Since the DialPort Portal can provide the user speech signial, in addition to the transcribed text, UCSC ran experiments on the Portal with SlugBot. We collected 3188 dialogs with real users, with 18,809 turns.
The DialPort Portal can serve as a valuable resource for researchers working on open-domain dialog systems. Real users enjoy interacting with state-of-the-art social bots and often have long conversations and leave tons of feedback!
DSTC9 Track: Interactive Evaluation of Dialog
We ran a DSTC9 track on the DialPort Portal! This track was open to any dialog researcher and was conducted with the objective of challenging researchers to built dialog systems that could interact with real users. Our track was very succesful, with over 18 teams participating from many different academic institutions and companies. Throughout the track we collected over 7000 dialogs with real users, with over 60,000 dialog turns (average of 7.8 turns per dialog).
Several publications were produced from this DSTC9 track. Our track overview paper describes the track and gives an overview of the results. Several teams also published papers at the AAAI-2021 workshop: [1], [2]. We anticipate that other teams will publish papers in the near future and we hope that the data we collected on the DialPort Portal during this challenge will serve as a valuable resource for research!
Mr. Clue
We ran a series of studies for Mr. Clue, a game-based dialog system, built by the University of Southern California. We collected 1783 dialogs with over 13,735 turns. The DialPort Portal allowed the researchers at USC to experiment with several different variants of Mr. Clue, to ultimately determine the setting that real users enjoyed most!
Get in touch
If you are interested in using the DialPort Portal for your research, or simply want to learn more about our work, please get in touch with us! The Portal is a free resource for the research community and we will be happy to help you collect dialogs with real users!