LanguageCrawl: a generic tool for building language models upon common Crawl.
The exponential growth of the internet community has resulted in the production of a vast amount of unstructured data, including web pages, blogs and social media. Such a volume consisting of hundreds of billions of words is unlikely to be analyzed by humans. In this work we introduce the tool Langu...
| Publicado en: | Language Resources & Evaluation Vol. 55; no. 4; pp. 1047 - 1076 |
|---|---|
| Autores principales: | , |
| Formato: | Artículo |
| Publicado: |
Springer Nature
Dec2021
|
| Materias: | |
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=hlh&AN=152947521&site=ehost-live header: @attributes: shortDbName: hlh uiTerm: 152947521 longDbName: Humanities International Complete uiTag: AN controlInfo: bkinfo: jinfo: jid: 1574020X 179V jtl: Language Resources & Evaluation issn: 1574020X maglogo: N pubinfo: dt: Dec2021 vid: 55 iid: 4 pid: 237 pub: Springer Nature artinfo: ui: 152947521 10.1007/s10579-021-09551-7 ppf: 1047 ppct: 29 formats: fmt: – @attributes: type: T – @attributes: type: P size: 922KB tig: atl: LanguageCrawl: a generic tool for building language models upon common Crawl. aug: au: Roziewski, Szymon Kozłowski, Marek affil: National Information Processing Institute, Warsaw, Poland su: Natural language processing Virtual communities Websites Interference (Linguistics) sug: subj: Natural language processing Virtual communities Websites Interference (Linguistics) keyword: Common Crawl Language Models N-gram Polish Web Corpus Word2Vec ab: The exponential growth of the internet community has resulted in the production of a vast amount of unstructured data, including web pages, blogs and social media. Such a volume consisting of hundreds of billions of words is unlikely to be analyzed by humans. In this work we introduce the tool LanguageCrawl, which allows Natural Language Processing (NLP) researchers to easily build web-scale corpora using the Common Crawl Archive—an open repository of web crawl information, which contains petabytes of data. We present three use cases in the course of this work: filtering of Polish websites, the construction of n-gram corpora and the training of a continuous skipgram language model with hierarchical softmax. Each of them has been implemented within the LanguageCrawl toolkit, with the possibility to adjust specified language and n-gram ranks. This paper focuses particularly on high computing efficiency by applying highly concurrent multitasking. Our tool utilizes effective libraries and design. LanguageCrawl has been made publicly available to enrich the current set of NLP resources. We strongly believe that our work will facilitate further NLP research, especially in under-resourced languages, in which the lack of appropriately-sized corpora is a serious hindrance to applying data-intensive methods, such as deep neural networks. pubtype: Academic Journal doctype: Article src: R language: English refInfo: copyright: @attributes: flag: Y custom: Language Resources & Evaluation is a copyright of Springer, 2021. All Rights Reserved. item: Language Resources & Evaluation holder: Springer Nature dt: @attributes: year: 2021 holdings: @attributes: islocal: N |
|---|