List Issue - KB Assistant

Recently, there has been a multitude of advances in the field of natural language, especially since the release of the Transformer [1] architecture. Before Transformers, RNNs (Recurrent neural networks) [2] like LSTMs (Long short-term memory networks) [3], and GRUs (Gated recurrent units) [4] were prevalent due to their network architecture design that suited the domain of natural language in the form of texts. However, most of these older architectures struggled to model long-term sequential dependency and are often slow to train due to the inability of parallelization.

On the other hand, the Transformer solves this issue by being parallelizable during training and extensively applying the attention mechanism to better model long sequences of inputs. Several modern architectures were then derived from the original Transformer model, such as the OpenAI GPT (Generative Pre-Training) [5], BERT (Bidirectional Encoder Representations from Transformers) [6], and subsequently variants like GPT-2 [7], GPT-3 [8], and RoBERTa (Robustly Optimized BERT Pre-training Approach) [9].

For instance, the GPT architecture stacks decoder blocks of the original Transformer architecture, while BERT stacks encoder blocks of the original Transformer and modifies them to become naturally bidirectional by design. Due to this discrepancy, GPT-based models are normally pre-trained as causal/generative language models, whereas BERT-based models are pre-trained as masked language models. Once these models have been pre-trained, they can similarly be fine-tuned to downstream natural language understanding tasks. This scheme allowed for breakthroughs in tasks like text classification, question-answering, natural language inference, among many others.

However, most of these recent successes occur in the domain of high-resource languages like English, Chinese, and Spanish where data are abundant. Low-resource languages, conversely, have reaped little benefits despite the recent advances; attributing to the lack of data and existing work [10]. The best alternative is to fine-tune a multilingual model whose corpus contains that specific low-resource language. However, previous studies [11, 12] showed that multilingual models often perform poorly compared to monolingual models of low-resource languages.

To that, we propose the pre-training of various Transformer-based language models on a low-resource language, namely Sundanese. The Sundanese language is Indonesia’s third-most spoken language [13], with over 32 million speakers worldwide and is the 52nd most spoken language in the world [14]. It is the official language in larger regions of Banten and West Java, as well as a minority language in various parts of Jakarta, Lampung, Central Java, and other nearby provinces in Indonesia [15]. Linguists usually classify the language into as many as 8 dialects [16], with the Priangan dialect being the most prominent of them all.

Moreover, a previous study [17] showed that the Sundanese language remains the main communication tool for adults and parents living in those regions. In the same paper, the authors concluded that there is a desire to preserve the Sundanese language in the fast-growing modern technological world as a cultural identity of the people. Another study [18] similarly noted that a great deal of Indonesian social media activity in platforms like Twitter originates from West Java, where the Sundanese language is most often used. With that many speakers and a huge market, it would be beneficial to have Sundanese language models that could be applied to various business processes.

Contrarily, there is very limited Sundanese corpus to be used for pre-training. Hence, on top of the pre-existing multilingual corpora like OSCAR (Open Super-large Crawled Aggregated Corpus) [19], CC-100 [20, 21], and C4 [22], Sundanese documents from Wikipedia were also used during the pre-training of our language models to cater to the scarcity of data availability.

Having been pre-trained, these models were thus fine-tuned to a downstream task of classifying emotions from tweets and were subsequently benchmarked against existing traditional machine learning algorithms and multilingual Transformer models such as the multilingual BERT [6] and the RoBERTa variant of XLM (Cross-lingual Language Model Pre-training) [20].

Once these models have been fully pre-trained and fine-tuned, they are subsequently released in the HuggingFace Transformers Model Hub^Footnote1, in hopes of encouraging other researchers to work on the field of Sundanese language modeling with open access to our models.

In the following sections, we will continue by discussing the background of Sundanese language modeling in greater detail, followed by the methodology used and experiments performed during the pre-training of our models, and finally, various analyses, discussions, and conclusions thereafter.

Transformer architectures

Before the advent of deep learning [23] and Transformers [1] following thereafter, natural language is one domain that computers have always struggled to model. This underlying issue prevails due to the fact that the rules of natural languages are often complex and vary from one language to another. Language modeling by means of traditional methods like statistical or rule-based models would often crumble when new, unseen inputs are fed into these models, given their inability to generalize.

On the contrary, deep learning models need not be taught the rules of natural language explicitly. Instead, by using methods like supervised learning, these models could gain an understanding of sequences of texts given the ground ruth/label. A few of the earliest deep learning architectures developed to learn sequences of inputs include Recurrent neural networks [2], Long short-term memory networks [3], and Gated Recurrent Units [4]. Unlike feed-forward neural network architectures, a recurrent neural network has temporal information passed over time by considering the output of previous time steps.