Training a transformer to predict the gender of German nouns

Context

• Der Löffel (the spoon) - Masculine

• Die Gabel (the fork) - Feminine

• Das Messer (the knife) - Neuter

• How well can a model learn the German genders?

• Assuming the same dataset size and same model architecture:
• Are German genders more difficult to classify compared to other languages?

Gathering datasets

• https://frequencylists.blogspot.com/2015/12/the-2000-most-frequent-german-nouns.html

• Catalan: it is my native language, and I was really curious to see how the model would do on it. However, it only has two genders. My hypothesis for Catalan was that the gendering is very intuitive and “machine learnable”. I couldn’t find a similar dataset, and I thought it made sense to translate word by word using the Microsoft Translator API. I’d recommend using the Python library deep-translator.

• Croatian: it is a perfect comparison to German because it also has three genders. My girlfriend speaks it, and it really helped because I didn’t know that, but Croatian has genders but no articles (I went to store). The problem was that I was translating the German dataset to Croatian, and of course it would translate der Apfel → jabuka, so I would lose the gender 😢. Then my Croatian-native collaborator suggested the trick of translating the word along with their possessive pronoun, e.g., mein Apfel → moja jabuka. And I could extract the gender of the Croatian words like this: moj → masculine, moja → feminine, and moje → neutrum. Nice.

• German: 2539 unique nouns. Find it in data/german_words_processed.csv.

• Croatian: 2196 unique nouns. Find it in data/croatian_words_processed.csv.

• Catalan: 2250 unique nouns. Find it in data/catalan_words_processed.csv.

Training and evaluation setup

• German: 0.4013

• Croatian: 0.4800

• Catalan: 0.5633

Model architecture

import torch
import torch.nn as nn

class TransformerClassifier(nn.Module):
    def __init__(
        self,
        vocab_size: int,
        max_length: int,
        num_classes: int,
        embed_dim: int = 128,
        num_heads: int = 8,
        num_layers: int = 2,
    ) -> None:
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.positional_encoding = nn.Parameter(
            torch.zeros(1, max_length, embed_dim)
        )
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim, nhead=num_heads, batch_first=True
        )
        self.transformer_encoder = nn.TransformerEncoder(
            encoder_layer, num_layers=num_layers
        )
        self.fc = nn.Linear(embed_dim, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.embedding(x) + self.positional_encoding[:, : x.size(1), :]
        x = self.transformer_encoder(x)
        x = x.mean(dim=1)  # Global average pooling
        return self.fc(x)

1. Embedding Layer: nn.Embedding(vocab_size, embed_dim) converts input token indices into dense vector representations of dimension embed_dim. This layer helps in mapping discrete token indices to continuous embeddings.

2. Positional Encoding: nn.Parameter(torch.zeros(1, max_length, embed_dim)) adds positional information to the token embeddings to capture the order of tokens in the input sequence. Positional encodings are necessary because the transformer architecture does not inherently incorporate token position information.

3. Transformer Encoder:
• nn.TransformerEncoderLayer(d_model=embed_dim, nhead=num_heads, batch_first=True) defines a single encoder layer that applies multi-head self-attention and a position-wise fully connected feed-forward network to the input embeddings. Here, d_model specifies the dimension of the input embeddings, and nhead denotes the number of attention heads.
• nn.TransformerEncoder(encoder_layer, num_layers=num_layers) stacks multiple TransformerEncoderLayer instances to create a deeper, more expressive model. This allows for capturing complex dependencies in the input sequence.

4. Fully Connected (Linear) Layer: nn.Linear(embed_dim, num_classes) maps the final representation of the sequence to output class scores. This layer acts as the classifier in the model, translating the aggregated sequence representation into the desired number of output classes.

1. Embedding Lookup: Convert input token indices into dense embeddings using self.embedding(x). The input x is a tensor of token indices with shape (batch_size, sequence_length).

2. Add Positional Encoding: Incorporate positional information into the embeddings by adding the positional encoding: x = self.embedding(x) + self.positional_encoding[:, : x.size(1), :]. This operation ensures that the model is aware of the token positions within the sequence.

3. Transformer Encoding: Pass the positionally encoded embeddings through the transformer encoder: x = self.transformer_encoder(x). The encoder processes the sequence to produce a contextualized representation of each token.

4. Global Average Pooling: Aggregate token representations by averaging over the sequence length: x = x.mean(dim=1). This operation reduces the sequence dimension, resulting in a fixed-size vector representation of shape (batch_size, embed_dim).

5. Classification: Transform the pooled representation into class scores using the fully connected layer: x = self.fc(x). The final output is a tensor of shape (batch_size, num_classes), representing the predicted class scores for each input sequence.

• Loss Function: nn.CrossEntropyLoss() is the most natural choice

• Optimizer: optim.Adam(model.parameters(), lr=config.lr) is also widely used.

• Learning Rate Scheduler: optim.lr_scheduler.StepLR(optimizer, step_size=config.step_size, gamma=config.gamma) adjusts the learning rate at regular intervals (step_size) by a factor of gamma. This helps in fine-tuning the learning rate during training to improve convergence.

DEFAULT_SETTINGS = {
    "batch_size": 32,
    "lr": 0.001,
    "step_size": 10,
    "gamma": 0.1,
    "val_size": 0.2,
    "early_stop": None,
}

This model architecture leverages the power of transformer-based self-attention mechanisms to capture long-range dependencies and interactions between tokens in a sequence, making it well-suited for various natural language processing tasks such as text classification.

General results

Accuracy:
- german: 0.5860
- croatian: 0.6240
- catalan: 0.6540
F1 Score:
- german: 0.5559
- croatian: 0.6239
- catalan: 0.6271

Results - German:

Detailed scores and confusion matrix

Model 20240730210843 (german)
- Accuracy: 0.5860
- Precision: 0.5619
- Recall: 0.5860
- F1 Score: 0.5559

1. Predicting “der” when the correct article is “das”

2. Predicting “der” when the correct article is “die”

3. Predicting “die” when the correct article is “der”

der raum:       predicted der -> correct
der saum:       predicted der -> correct
das wachstum:   predicted die -> wrong
das individuum: predicted das -> correct
das stipendium: predicted das -> correct

Results - Croatian:

Detailed scores and confusion matrix

Model 20240730213637 (croatian)
- Accuracy: 0.6240
- Precision: 0.6239
- Recall: 0.6240
- F1 Score: 0.6239

Results - Catalan:

Detailed scores and confusion matrix

Model 20240730214109 (catalan)
- Accuracy: 0.6540
- Precision: 0.6677
- Recall: 0.6540
- F1 Score: 0.6271

Lessons learned

• Hatch is handy: use it as your python project and environment manager! I also used an automatic code fromatter and commit hooks, and I am sure I will always do it from now on.

• If you have an idea and a bit of time, just get your laptop and work on it! This was what happened with this project. But don’t forget to add it to your GitHub with a nice Readme and to find a way to include in your portfolio.

• Using translation tools like Microsoft Translator API and libraries like deep-translator can facilitate dataset creation for languages where pre-existing datasets are unavailable. You can get something like 4M words to translate for free per month.