Albayzin 2012 Language Recognition Evaluation

The Albayzin 2012 Language Recognition Evaluation (Albayzin 2012 LRE) is supported by the Spanish Thematic Network on Speech Technology (RTTH) and organized by the Software Technologies Working Group (GTTS) of the University of the Basque Country, with the key collaboration of Niko Brümmer, from Agnitio Research, South Africa, for defining the evaluation criterion and coding the script used to measure system performance. The evaluation workshop will be part of IberSpeech 2012, to be held in Madrid, Spain from 21 to 23 November 2012. 
As in previous Albayzin LRE editions, the goal of this evaluation is to promote the exchange of ideas, to foster creativity and to encourage collaboration among research groups worldwide working on language recognition technology. To this end, we propose a language recognition evaluation similar to those carried out in 2008 and 2010, but under more difficult conditions. This time the application domain moves from TV Broadcast speech to any kind of speech found in the Internet, and no training data will be available for some of the target languages (aiming to reflect a common situation for low-resource languages). 
The change in the application domain pursues two objectives: first, the task should reflect a practical application (in this case, indexing of multimedia content in the Internet); and second, the task should be challenging enough for state-of-the-art systems to yield a relatively poor performance. 
Audio signals for development and evaluation will be extracted from YouTube videos, which will be heterogeneous regarding duration, number of speakers, ambient noise/music, channel conditions, etc. Besides speech, signals may contain music, noise and any kind of non-human sounds. In any case, each signal will contain a minimum amount of speech. As for previous evaluations, each signal will contain speech in a single language, except for signals corresponding to Out-Of-Set (OOS) languages, which might contain speech in two or more languages, provided that none of them are target languages. 
Overall, the Albayzin 2012 LRE introduces some interesting novelties with regard to previous Albayzin LRE editions and NIST Language Recognition Evaluations. The most remarkable novelties are the type of signals used for development and test and the evaluation criterion. All the details can be found in the Albayzin 2012 LRE Plan.

Registration

Deadline: July 16th 2012 Procedure: Submit an e-mail to the organization contact: luisjavier.rodriguez@ehu.es, with copy to the Chairs of the Albayzin 2012 Evaluations: javier.gonzalez@uam.es and javier.tejedor@uam.es, providing the following information:

• Group name
• Group ID
• Institution
• Contact person
• Email address
• Postal address

Data delivery

Starting from June 15th 2012, and once registration data are validated, the training (108 hours of broadcast speech for 6 target languages) and development (around 2000 audio segments including 10 target languages and Out-Of-Set languages) datasets will be released via web (only to registered participants).

Schedule

• May 18 2012: The evaluation plan is released and registration is open.
• June 15 2012: Training and development data are released via web.
• July 16 2012: Registration deadline.
• September 3 2012: Evaluation data are released via web and system submission is open.
• September 24 2012: Deadline for submitting system results and system descriptions.
• October 15 2012: Preliminary results and evaluation keyfile are released via web.
• November 21-23 2012: Albayzin 2012 LRE Workshop at IberSpeech 2012, Madrid, Spain.

Contact

Luis Javier Rodríguez Fuentes Software Technologies Working Group (GTTS) Department of Electricity and Electronics (ZTF-FCT) University of the Basque Country (UPV/EHU) Barrio Sarriena s/n 48940 Leioa - SPAIN
web: http://gtts.ehu.es e-mail: luisjavier.rodriguez@ehu.es phone: +34 946012716 fax: +34 946013071