®®®® SIIA Público

Título del libro: Vardial 2017 - 4th Workshop On Nlp For Similar Languages, Varieties And Dialects, Proceedings
Título del capítulo: Discriminating between similar languages using a combination of typed and untyped character n-grams and words

Autores UNAM:
HELENA MONTSERRAT GOMEZ ADORNO;
Autores externos:

Idioma:

Año de publicación:
2017
Palabras clave:

Classification (of information); Computational linguistics; Digital subscriber lines; Learning algorithms; Classification approach; Feature representation; Frequency threshold; Lexical features; N-grams; Representation method; Single-step; Term Frequency; Threshold-value; Word and characters; Support vector machines


Resumen:

This paper presents the CIC UALG's system that took part in the Discriminating between Similar Languages (DSL) shared task, held at the VarDial 2017 Workshop. This year's task aims at identifying 14 languages across 6 language groups using a corpus of excerpts of journalistic texts. Two classification approaches were compared: a single-step (all languages) approach and a two-step (language group and then languages within the group) approach. Features exploited include lexical features (unigrams of words) and character n-grams. Besides traditional (untyped) character n-grams, we introduce typed character n-grams in the DSL task. Experiments were carried out with different feature representation methods (binary and raw term frequency), frequency threshold values, and machine-learning algorithms - Support Vector Machines (SVM) and Multinomial Naive Bayes (MNB). Our best run in the DSL task achieved 91.46% accuracy. © 2017 Association for Computational Linguistics


Entidades citadas de la UNAM: