|
 |
ISCApad Archive » 2016 » ISCApad #221 » Events » Other Events » (2016-12-xx) CfP Dialog State Tracking Challenge 5 (DSTC5) |
ISCApad #221 |
| Friday, November 11, 2016 by Chris Wellekens |
3-3-3 (2016-12-xx) CfP Dialog State Tracking Challenge 5 (DSTC5)
Dialog State Tracking Challenge 5 (DSTC5) @SLT 2016 San Juan Porto Rico
Call for Participation
=================================
* MOTIVATION
Dialog state tracking is one of the key sub-tasks of dialog management, which defines the representation of dialog states and updates them at each moment on a given on-going conversation. To provide a common testbed for this task, the first Dialog State Tracking Challenge (DSTC) was initiated [1], and then two more challenges (DSTC 2&3) [2][3] had been organized keeping the aim at human-machine conversations. On the other hand, the fourth challenge (DSTC 4) which has been most recently completed [4] has shifted the target of state tracking to human-human dialogs. In the challenge, a dialog state was defined for each sub-dialog segment level as a frame structure filled with slot-value pairs representing the main subject of the segment. Then, trackers were required to fill out the frame considering all dialog history prior to each turn in a given segment.
The previous DSTCs have contributed to the spoken dialog research community by providing opportunities for sharing the resources, comparing results among the proposed algorithms, and improving the state-of-the-art. However, the impacts of the outcomes from the challenges could be restricted to English dialogs only, because all the resources including the corpora, ontologies, and databases were collected under monolingual settings in English.
In the fifth challenge, we introduce a cross-lingual dialog state tracking task addressing the problem of adaptation to a new language. The goal of this task is to build a tracker in the target language with given the existing resources in the source language and their translations generated automatically by machine translation technologies to the target language. In addition to this main task, we propose a series of pilot tracks for the core components in developing end-to-end dialog systems also in the same cross-lingual settings. We expect that these shared efforts on cross-lingual tasks would contribute to progress in improving the language portability of state-of-the-art monolingual technologies and reducing the costs for building resources from the scratch to develop dialog systems in a resource-poor target language.
* DATASETS
At the beginning of the challenge, TourSG corpus which was used in DSTC 4 will be provided as a training set in the source language English. TourSG consists of 35 dialog sessions on touristic information for Singapore collected from Skype calls between three tour guides and 35 tourists. All the recorded dialogs have been manually transcribed and annotated with various labels.
In addition to the original dialogs in English, their translations generated by a machine translation system to Chinese which is the target language in the challenge will be also given along with the word alignment information, so that participants will not need to run their own system to generate the translated pairs of the dialogs.
Then, a test set will be released to evaluate the trackers developed in the first phase. It consists of Chinese dialogs collected and annotated under the equivalent conditions to the English dataset TourSG. At the beginning of the test phase, only the unlabelled set will be given with their English translations which were also generated by machine translation. The full annotations for the test set will be available after the challenge period.
* PROPOSED TASKS
Main task:
- Dialog state tracking at sub-dialog level: Fill out the frame of slot-value pairs for the current sub-dialog considering all dialog history prior to the turn.
Pilot tasks (optional):
- Spoken language understanding: Tag a given utterance with speech acts and semantic slots.
- Speech act prediction: Predict the speech act of the next turn imitating the policy of one speaker.
- Spoken language generation: Generate a response utterance for one of the participants.
- End-to-end system: Develop an end-to-end system playing the part of a guide or a tourist.
Open track (optional):
- Proposed by teams willing to work on any task of their interest over the provided dataset.
* IMPORTANT DATES
- 01 Apr 2016: Registration opens
- 14 Apr 2016: Training set is released
- 18 Jul 2016: Registration closes
- 21 Jul 2016: Test set is released
- 27 Jul 2016: Entry submission deadline
- 29 Jul 2016: Evaluation results are released
- 19 Aug 2016: Paper submission deadline
- December 2016: Workshop is held @ SLT 2016
* ORGANIZING COMMITTEE
Seokhwan Kim (I2R, Singapore)
Luis Fernando D?Haro (I2R, Singapore)
Rafael E. Banchs (I2R, Singapore)
Matthew Henderson (Google, USA)
Jason D. Williams (Microsoft, USA)
Koichiro Yoshino (NAIST, Japan)
* CONTACT DETAILS
Seokhwan Kim: kims AT i2r.a-star.edu.sg
Luis Fernando D?Haro: luisdhe AT i2r.a-star.edu.sg
1 Fusionopolis Way, #21-01, Singapore 138632
Fax: (+65) 6776 1378
* REFERENCES
[1] Jason D. Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The Dialog State Tracking Challenge. In Proceedings of the 14th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), Metz, France.
[2] Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014. ?The Second Dialog State Tracking Challenge?. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), Philadelphia, USA.
[3] Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014. ?The Third Dialog State Tracking Challenge?. In Proceedings of IEEE Spoken Language Technology Workshop, South Lake Tahoe, USA.
[4] Seokhwan Kim, Luis Fernando D'Haro, Rafael E. Banchs, Jason D. Williams, Matthew Henderson. 2016. ?The Fourth Dialog State Tracking Challenge?. In Proceedings of the 7th International Workshop on Spoken Dialogue Systems (IWSDS 2016), Saariselkä, Finland. |
 | Back |  | Top |