In this newsletter:

  Fall 2011 LDC Data Scholarships recipients

 LDC data - now on Blu-ray

  LDC at NWAV 2011 

New publications:

LDC2011S08
2008 NIST Speaker Recognition Evaluation Test Set

LDC2011T11
Arabic Gigaword Fifth Edition 

LDC2011T12
Spanish Gigaword Third Edition

Fall 2011 LDC Data Scholarships recipients

LDC is pleased to announce the student recipients of the Fall 2011 LDC Data Scholarship program!  The LDC Data Scholarship program provides university students with access to LDC data at no-cost.  Data scholarships are offered twice a year to correspond to the Fall and Spring semesters.  Students are asked to complete an application which consists of a data use proposal and letter of support from their academic adviser.  

LDC received many strong applications from students attending universities across the globe.  We've reviewed all the applications, and after careful consideration, we have selected four scholarship recipients!   These students will receive no-cost copies of LDC data:

Haris B C - Indian Institute of Technology Guwahati (India), Electronics & Electrical Engineering.  Haris has been awarded a copy of 2005 NIST Speaker Recognition Evaluation Training Data (LDC2011S01) and 2005 NIST Speaker Recognition Evaluation Test Data (LDC2011S04) to evaluate the performance of a sparse representation speaker verification system.

Friðjón Guðjohnsen - Reykjavik University (Iceland), Computer Science.  Friðjón has been awarded a copy of Treebank-3 (LDC99T42) to be used in the development of tagging methods to improve the accuracy of tagging Icelandic texts.

Leili Javadpour - Louisiana State University (USA), Engineering Science.  Leili has been awarded a copy of BBN Pronoun Coreference and Entity Type Corpus (LDC2005T33) and Message Understanding Conference (MUC) 7 (LDC2001T02) for her work in pronominal anaphora resolution.

Jad Makhlouta - American University of Beirut (Lebanon), Electrical and Computer Engineering.  Jad has been awarded a copy of LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1 (LDC2010L01) for his work in Arabic text mining.

Please join us in congratulating our student recipients!   Look for our upcoming announcements about the submissions deadlines for the Spring 2012 LDC Data Scholarship program.

LDC data - now on Blu-ray!

LDC is pleased to announce that the Blu-ray revolution has come to linguistic data!  We are now offering select databases on Blu-ray Disc (BD).  With BD, we'll be able to distribute some of our larger databases using a smaller number of discs.   BDs also have the potential to be read more quickly than DVDs, which means faster access to data.  To introduce our Blu-ray option, we would like to announce that the following databases will now be distributed on BD in addition to DVD-ROM:

LDC2011T07 English Gigaword Fifth Edition, now available on 1 Blu-ray Disc

LDC2006T13 Web 1T 5-gram Version 1, now available on 2 Blu-ray Discs

Organizations with licenses to English Gigaword Fifth Edition will be given the opportunity to 'swap' their DVDs for BDs.  New licensees for Web 1T 5-gram have the option to select BD or DVD media.

We expect to extend the BD option over time to other corpora in the catalog and to new releases.

LDC at NWAV 2011

NWAV’s 40th Anniversary Conference  will be hosted by Georgetown University from October 27-30 and LDC will be on-hand to celebrate! Please stop by the LDC exhibition at any point during the main conference and be sure to attend LDC’s pre-conference workshop on “Demographic Coding for Sociolinguistic Corpus Archive Preparation” from 4.00 – 6.00 pm on Thursday, October 27. This workshop will be hosted by LDC Executive Director Christopher Cieri and Malcah Yaeger-Dror of the University of Arizona. It has two stated goals:

1) to catalog the need for more detailed demographic categories based on field experience that other researchers can and should exploit both for their own immediate analyses, and to facilitate sharing among research groups, and

 2) to encourage the use of a core set of demographic metadata coding options.

NWAV registration options can be found here. We hope to see you there! Please visit LDC’s Facebook page to follow our conference activities.

New publications

(1) 2008 NIST Speaker Recognition Evaluation Test Set was developed by LDC and NIST (National Institute of Standards and Technology). It contains 942 hours of multilingual telephone speech and English interview speech along with transcripts and other materials used as test data in the 2008 NIST Speaker Recognition Evaluation (SRE).

NIST SRE is part of an ongoing series of evaluations conducted by NIST.  They are intended to be of interest to all researchers working on the general problem of text independent speaker recognition. The 2008 evaluation was distinguished from prior evaluations, in particular those in 2005 and 2006, by including not only conversational telephone speech data but also conversational speech data of comparable duration recorded over a microphone channel involving an interview scenario.

LDC previously released the 2008 NIST SRE Training Set in two parts as LDC2011S05 and LDC2011S07.

The speech data in this release was collected in 2007 by LDC at its Human Subjects Data Collection Laboratories in Philadelphia and by the International Computer Science Institute (ICSI) at the University of California, Berkeley. This collection was part of the Mixer 5 project, which was designed to support the development of robust speaker recognition technology by providing carefully collected and audited speech from a large pool of speakers recorded simultaneously across numerous microphones and in different communicative situations and/or in multiple languages. Mixer participants were native English and bilingual English speakers. The telephone speech in this corpus is predominantly English, but also includes the above languages. All interview segments are in English. Telephone speech represents approximately 368 hours of the data, whereas microphone speech represents the other 574 hours.

English language transcripts in .cfm format were produced using an automatic speech recognition (ASR) system.

2008 NIST Speaker Recognition Evaluation Test Set is distributed on 9 DVD-ROM.

2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$2000.

*

(2) Arabic Gigaword Fifth Edition is a comprehensive archive of newswire text data that has been acquired from Arabic news sources over several years by LDC. Arabic Gigaword Fifth Edition includes all of the content of the fourth edition of Arabic Gigaword (LDC2009T30) plus new data covering the period from January 1, 2009 through December 31, 2010.

Nine distinct sources of Arabic newswire are represented in this distribution:

Asharq Al-Awsat (aaw_arb)

Agence France Presse (afp_arb)

Al-Ahram (ahr_arb)

Assabah (asb_arb)

Al Hayat (hyt_arb)

An Nahar (nhr_arb)

Al-Quds Al-Arabi (qds_arb)

Ummah Press (umh_arb)

Xinhua News Agency (xin_arb)

The seven-character codes shown above represent both the directory names where the data files are found, and the 7-letter prefix that appears at the beginning of every file name. The 7-letter codes consist of the three-character source name IDs and the three-character language code ('arb') separated by an underscore ('_') character. The three-character language code conforms to the ISO 639-3 standard.

In addition to adding new data, the following updates were made:

Repeated documents in Asharq Al-Awsat data from 2008 were removed.

Document formatting and docid duplication problems were corrected in Agence France Presse  data.

Significant duplication of content in 2007-2008 An Nahar data was detected, and the duplicated documents were removed.

Arabic Gigaword Fifth Edition is distributed on 1 DVD-ROM.

2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$6000.

*

(3) Spanish Gigaword Third Edition was produced by LDC. It is a comprehensive archive of Spanish newswire text data that has been acquired over several years by LDC. Spanish Gigaword Third Edition includes all of the content of the second edition (LDC2009T21) and adds data collected from January 1, 2009 through December 31, 2010.

The three distinct international sources of Spanish newswire in this edition, and the time spans of collection covered for each, are as follows:

Agence France-Presse, Spanish (afp_spa) May 1994 - Dec 2010

Associated Press, Spanish (apw_spa) Nov 1993 - Dec 2010

Xinhua News Agency, Spanish (xin_spa) Sep 2001 - Dec 2010

The seven-letter codes in the parentheses above include the three-character source name abbreviations and the three-character language code ('spa') separated by an underscore ('_') character. The three-letter language code conforms to LDC's internal convention based on the ISO 639-3 standard.

All text data are presented in SGML/XML form, using a very simple, minimal markup structure; all text consists of printable ASCII, whitespace, and printable code points in the 'Latin1 Supplement' character table, as defined by both ISO-8859-1 and the Unicode Standard (ISO 10646) for the 'accented' characters used in Spanish. The Supplement/accented characters are rendered using UTF-8 encoding.

Spanish Gigaword Third Edition is distributed on 1 DVD-ROM.

2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$4500.