ALBAYZIN 2014 SEARCH ON SPEECH EVALUATION

The Spanish Thematic Network on Speech Technology (RTTH) and the ISCA Special Interest Group
on Iberian Languages (SIG-IL) are pleased to announce the ALBAYZIN 2014 Search on Speech Evaluation,
which will be carried out as part of Iberspeech 2014, a biennial event gathering the Spanish researchers
on speech Technology. This year’s event will take place in Las Palmas de Gran Canaria (Spain),
on November 19-21, 2014 (see http://iberspeech2014.ulpgc.es for details).

Research groups worldwide are invited to participate in this evaluation.
Here we just provide the key points. The full evaluation plan can be found in:

http://iberspeech2014.ulpgc.es/index.php/albayzin/search-on-speech-evaluation

**TASKS**

The ALBAYZIN 2014 Search on Speech evaluation involves searching in audio content a list of terms/queries.
This evaluation focuses on retrieving the appropriate audio files that contain any of those terms/queries.
Four different tasks are defined:

1) KEYWORD SPOTTING (KWS), where the input to the system is a list of terms, which is known
when processing the audio and hence word-based recognizers can be effectively used to hypothesize detections.

2) SPOKEN TERM DETECTION (STD), where the input to the system is a list of terms (as in the KWS task),
but terms/queries are unknown when processing the audio. This is the same task as in NIST STD 2006 evaluation
[1] and Open Keyword Search 2013 [2].

3) QUERY-BY-EXAMPLE SPOKEN TERM DETECTION (QbE STD), where the input to the system
is an acoustic example per query and hence a prior knowledge of the correct word/phone transcription
corresponding to each query cannot be made. This task must generate a set of occurrences for each
query detected in the audio files, along with their timestamps as output, as in the STD task.
QbE STD is the same task as those proposed in MediaEval 2011, 2012 and 2013 [3].

4) QUERY-BY-EXAMPLE SPOKEN DOCUMENT RETRIEVAL (QbE SDR), where the input to the system
is composed of several acoustic examples per query and hence a prior knowledge of the correct word/phone
transcription corresponding to each query cannot be made. This task must generate an output score for each
of the provided queries, which reflects the probability that each of the queries appears in each audio file,
and no information about the timestamp is required. Formally, given a spoken example of a given query q
and a spoken document x (whose transcriptions are unknown), a QbE SDR system must carry out some kind
of detection procedure and output a score s ∈ R, the higher (the more positive) the score the higher the likelihood
that q appears in x. Note that systems are neither required to make a strong decision about whether or not
q appears in x, nor to provide the time marks of the place (or places) where q appears. Systems are just required
to produce a score, which must be computed by automatic means, with no human supervision.
This is the same task as that proposed in MediaEval 2014 Query-by-Example Search on Speech (QUESST) [4].

**REGISTRATION**

Interested groups must register for the evaluation before July 15th 2014, by contacting the organizing team at:

javiertejedornoguerales@gmail.com
luisjavier.rodriguez@ehu.es

with CC to the Chairs of Iberspeech 2014(iberspeech2014@ulpgc.es), and providing the following information:

    Research group (name and acronym)
    Institution (university, research center, etc.)
    Contact person (name)
    Email

**SCHEDULE**

•      June 30, 2014: Release of training and development data

•      July 15, 2014: Registration deadline

•      September 3, 2014: Release of evaluation data

•      September 30, 2014: Deadline for the submission of system outputs and description papers

•      October 15, 2014: Results distributed to participants

•      November 19-21, 2014: Evaluation Workshop at Iberspeech 2014


For more information, please follow the link to the Albayzin 2014 Search on Speech Evaluation:

http://iberspeech2014.ulpgc.es/index.php/albayzin/search-on-speech-evaluation

**REFERENCES**

[1] http://www.itl.nist.gov/iad/mig/tests/std/2006/index.html

[2] http://www.nist.gov/itl/iad/mig/openkws13.cfm

[3] Florian Metze, Xavier Anguera, Etienne Barnard, Marelie Davel and Guillaume Gravier. 'Language Independent Search in Mediaeval's Spoken Web Search Task'. Computer Speech and Language, Special Issue on Information Extraction & Retrieval, 2014.

[4] http://multimediaeval.pbworks.com/w/page/79432139/QUESST2014