Nalazite se na CroRIS probnoj okolini. Ovdje evidentirani podaci neće biti pohranjeni u Informacijskom sustavu znanosti RH. Ako je ovo greška, CroRIS produkcijskoj okolini moguće je pristupi putem poveznice www.croris.hr
izvor podataka: crosbi

Evaluation of Language Models over Croatian Newspaper Texts (CROSBI ID 245960)

Prilog u časopisu | izvorni znanstveni rad | međunarodna recenzija

Beliga, Slobodan ; Ipšić, Ivo ; Martinčić-Ipšić, Sanda Evaluation of Language Models over Croatian Newspaper Texts // Information Technology and Control, 46 (2017), 4; 425-444. doi: 10.5755/j01.itc.46.4.18367

Podaci o odgovornosti

Beliga, Slobodan ; Ipšić, Ivo ; Martinčić-Ipšić, Sanda

engleski

Evaluation of Language Models over Croatian Newspaper Texts

Statistical language modeling involves techniques and procedures that assign probabilities to word sequences or, said in other words, estimate the regularity of the language. This paper presents basic characteristics of statistical language models, reviews their use in the large set of speech and language applications, explains their formal definition and shows different types of language models. Detailed overview of n-gram and class- based models (as well as their combinations) is given chronologically, by type and complexity of models, and in aspect of their use in different NLP applications for different natural languages. The proposed experimental procedure compares three different types of statistical language models: n-gram models based on words, categorical models based on automatically determined categories and categorical models based on POS tags. In the paper, we propose a language model for contemporary Croatian texts, a procedure how to determine the best n-gram and the optimal number of categories, which leads to significant decrease of language model perplexity, estimated from the Croatian News Agency articles (HINA) corpus. Using different language models estimated from the HINA corpus, we show experimentally that models based on categories contribute to a better description of the natural language than those based on words. These findings of the proposed experiment are applicable, except for Croatian, for similar highly inflectional languages with rich morphology and non-mandatory sentence word order.

Statistical language model ; Natural language regularity ; Word-based language model ; Category-based language model ; Brown algorithm ; POS class ; N-gram ; Perplexity ; Croatian corpora

nije evidentirano

nije evidentirano

nije evidentirano

nije evidentirano

nije evidentirano

nije evidentirano

Podaci o izdanju

46 (4)

2017.

425-444

objavljeno

1392-124X

10.5755/j01.itc.46.4.18367

Povezanost rada

Računarstvo, Informacijske i komunikacijske znanosti

Poveznice
Indeksiranost