The REAL Challenge – Call for Participation

The Dialog Research Center at Carnegie Mellon (DialRC) is organizing the REAL Challenge. The goal of the REAL Challenge (dialrc.org/realchallenge) is to build speech systems that are used regularly by real users to accomplish real tasks. These systems will give the speech and spoken dialog communities steady streams of research data as well as platforms they can use to carry out studies. It will engage both seasoned researchers and high school and undergrad students in an effort to find the next great speech applications.

Why have a REAL Challenge?

Humans greatly rely on spoken language to communicate, so it seems natural that we would be likely to communicate with objects via speech as well. Some speech interfaces do exist and they show promise, demonstrating that smart engineering can palliate indeterminate recognition. Yet the general public has not yet picked up this means of communication as easily as they have the tiny keyboards. About two decades ago, many researchers were using the internet, mostly to send and receive email. They were aware of the potential that it held and waited to see when and how the general public would adopt it. Practically a decade later, thanks to providers such as AmericaOnline, who had found how to create easy access, everyday people started to use the internet. And this has dramatically changed our lives. In the same way, we all know that speech will eventually replace the keyboard in many situations when we want to speak to objects. The big question is what is the interface or application that will bring us into that era.

Why hasn’t speech become a more prevalent interface? Most of today’s speech applications have been devised by researchers in the speech domain. While they certainly know what types of systems are “doable”, they may not be the best at determining which speech applications would be universally acceptable.

We believe that students who have not yet had their vision limited by knowledge of the speech and spoken dialog domains and who have grown up with computers as a given, are the ones that will find new, compelling and universally appealing speech applications. Along with the good ideas, they will need some guidance to gain focus. Having a mentor, attending webinars and participating in a research team can provide this guidance.

The REAL challenge will combine the talents of these two very different groups. First it will call upon the speech research community who know what it takes to implement real applications. Second, it will advertise to and encourage participation from high school students and college undergrads who love to hack and have novel ideas about using speech.

How can we combine these two types of talent?

The REAL Challenge is starting with a widely-advertised call for proposals. Students can propose an application. Researchers can propose to create systems or to provide tools. A proposal can target any type of application in any language. The proposals will be lightly filtered and the successful proposers will be invited to a workshop on June 21, 2014 to show what they are proposing and to team up. The idea is for students to meet researchers and for the latter to take one or more students on their team. Students will present their ideas and have time for discussion with researchers. A year later, a second workshop will assemble all who were at the first workshop to show the resulting systems, measure success and award prizes.

Student travel will be taken care of by DialRC through grants.

Preparing students

Students will have help from DialRC and from researchers as they formulate their proposals. DialRC will provide webinars on such topics as speech processing tool basics and how to present a poster. Students will also be assigned mentors. Researchers in speech and spoken dialog can volunteer to be a one-on-one mentor to a student. This consists of being in touch either in person or virtually. Mentors can tell the students about what our field consists of, what the state of the art is, and what it is like to work in research. They can answer questions about how the student can talk about their ideas. If you are a researcher in speech and/or spoken dialog and you would like to be a mentor, please let us know at realchallenge@speechinfo.org

What is an entry?

The groups will create entries. Here are the characteristics of a successful entry:

• there is a stateful interaction (not stateless, not on-off)

• the interaction is sustained over multiple turns

• language is central to the entry – it is the primary medium of exchange (not necessarily the only medium, but it is not peripheral to the main use of the entry)

• the entry makes a meaningful contribution to the interaction (so, it does not just pass messages)

• the entry must do some meaningful processing (not just passing messages like an email router). It has to make meaningful contributions to the interaction.

How can we assess success?

Success will be judged on the basis of originality, amount of regular users and of data and on other criteria to be agreed upon by the Challenge scientific committee and the participants.

Possible prize areas for an entry include:

• how much usage it gets

• how engaging it is / how novel is the interaction

• how good it is as a platform for future research – a platform is defined here as the output/result of an entry that would be of use for the research community. A platform is not just a computer program toolkit. It could, for example, be used the following year as the basis for a competition (like best ASR or best belief tracking)

Details of the measures of success will be refined at the workshop with input from the participants.

Timeline

The REAL Challenge was announced at several major conferences during the summer of 2013: SIGDIAL, Interspeech, ACL. It is also being announced to younger participants through their schools and hacker websites.

March 20, 2014 : Proposals due

April 20, 2014: Feedback on proposals and invitations to attend the workshop sent out.

June 21, 2014 : Workshop in Baltimore Maryland USA.

Early summer of 2015 : Resulting systems are presented a year after the first workshop.

What advantage is there for a student to participate?

For students, participation in the REAL Challenge will present several unique opportunities:

• the chance to work in a group with real researchers, on a real world problem

• the chance to see how ideas are turned into reality

• the chance to make something that works and that people actually use

• the chance to learn about new technology and use it to solve new problems

• the chance to observe what careers in technology are like and to be in contact with possible future employers

What does this Challenge contribute to the speech community?

For researchers, participation reaps several benefits:

• the number and type of speech applications will be greatly expanded

• there will be more datasets available for research

• there will be more platforms to run studies on and to use in speech and spoken dialog classes

• the enrichment that comes from mentoring

Why should industrial research groups be interested in the Challenge?

Industrial research groups should be interested to see:

• which types of applications actually appeal to the general public and which ones fail, which could be revenue-generating

• how students learn to apply the latest speech technologies in novel directions and which of these students could become future collaborators

Organization

This Challenge is run by the Dialog Research Center at Carnegie Mellon (DialRC)

REAL Challenge Scientific Committee

Alan W Black, Carnegie Mellon University, USA

Maxine Eskenazi, Carnegie Mellon University, USA

Helen Hastie, Heriot Watt University, Scotland

Gary Geunbae Lee, Pohang University of Science and Technology, South Korea

Sungjin Lee, Carnegie Mellon University, USA

Santoshi Nakamura, Nara Institute of Science and Technology, Japan

Elmar Noeth, Friedrich-Alexander University Erlangen-Nuremberg, Germany

Antoine Raux, Lenovo, USA

David Traum, University of Southern California, USA

Jason Williams, Microsoft Research, USA

Contact information:

Website : http://dialrc.org/realchallenge

Email : realchallenge@speechinfo.org