Computation for Linguists

Working with non-English data

Dr. Andrew M. Byrd

2025-11-12

Plan for the Day

• Review
• Analyzing Languages with Non-Latin Scripts
• Transcribing Languages
• Using WordNet

Review

• What did you learn last time?

Recap from Last Time

spaCy provides language-specific pipelines trained for each language.

Language	Model Name	Example
English	en_core_web_sm	“The students studied hard.”
French	fr_core_news_sm	“Les étudiants ont étudié dur.”
Spanish	es_core_news_sm	“Los estudiantes estudiaron mucho.”
German	de_core_news_sm	“Die Studenten haben fleißig gelernt.”

Recap from Last Time

import spacy

nlp_en = spacy.load("en_core_web_sm")
nlp_fr = spacy.load("fr_core_news_sm")
nlp_es = spacy.load("es_core_news_sm")
nlp_de = spacy.load("de_core_news_sm")

text_en = "The students have analyzed Proto-Indo-European roots."
text_fr = "Les étudiants ont analysé les racines proto-indo-européennes."
text_es = "Los estudiantes analizaron las raíces protoindoeuropeas."
text_de = "Die Studenten analysierten die indogermanischen Wurzeln."

for lang, nlp, text in [("English", nlp_en, text_en), ("French", nlp_fr, text_fr), ("Spanish", nlp_es, text_es), ("German", nlp_de, text_de)]:
    doc = nlp(text)
    print(f"\n {lang} Tokens:")
    print([t.text for t in doc])

Recap from Last Time

for lang, nlp, text in [("English", nlp_en, text_en), ("French", nlp_fr, text_fr), ("Spanish", nlp_es, text_es), ("German", nlp_de, text_de)]:
    doc = nlp(text)
    print(f"\n {lang} Lemmas:")
    for token in doc:
        print(token.text, "→", token.lemma_, token.pos_)

Recap from Last Time

• Let’s use our Counter function to compare how common certain words are.
• Before we begin, let’s load up two texts in English & Spanish.

text_en = "Whereas recognition of the inherent dignity and of the equal and inalienable rights of all members of the human family is the foundation of freedom, justice and peace in the world, the peoples of the United Nations have reaffirmed their faith in fundamental human rights and in the dignity and worth of the human person. They have resolved to promote social progress and better standards of life in larger freedom."

text_es = "Considerando que el reconocimiento de la dignidad intrínseca y de los derechos iguales e inalienables de todos los miembros de la familia humana constituye la base de la libertad, la justicia y la paz en el mundo, los pueblos de las Naciones Unidas han reafirmado su fe en los derechos humanos fundamentales y en la dignidad y el valor de la persona humana. Han decidido promover el progreso social y elevar el nivel de vida dentro de una libertad más amplia."

Recap from Last Time

from collections import Counter
import spacy

nlp_en = spacy.load("en_core_web_sm")
nlp_es = spacy.load("es_core_news_sm")

def is_preposition(tok):
  # True for tokens that are prepositions or postpositions
  return tok.pos_ == "ADP"

def prep_stats(nlp, text, lang_label):
  doc = nlp(text)
  # word-like tokens only for the denominator
  word_tokens = [t for t in doc if t.is_alpha]
  total = len(word_tokens)
  preps = [t.text for t in doc if is_preposition(t)]
  n = len(preps)
  rate = (n / total) if total else 0.0
  print(f"{lang_label}: {n} prepositions / {total} word tokens ({rate:.2%})")
  print("Top forms:", Counter(w.lower() for w in preps).most_common())

# Run on English and Spanish samples
prep_stats(nlp_en, text_en, "English")
prep_stats(nlp_es, text_es, "Spanish")

Review Activity

• Copy the below texts and calculate how many prepositions are in German & French.

text_fr = "Considérant que la reconnaissance de la dignité inhérente et des droits égaux et inaliénables de tous les membres de la famille humaine constitue le fondement de la liberté, de la justice et de la paix dans le monde, les peuples des Nations Unies ont réaffirmé leur foi dans les droits fondamentaux de l’homme, dans la dignité et la valeur de la personne humaine. Ils se sont engagés à favoriser le progrès social et à élever le niveau de vie dans une liberté plus grande."

text_de = "Da die Anerkennung der angeborenen Würde und der gleichen und unveräußerlichen Rechte aller Mitglieder der menschlichen Familie die Grundlage der Freiheit, der Gerechtigkeit und des Friedens in der Welt bildet, haben die Völker der Vereinten Nationen ihren Glauben an die grundlegenden Menschenrechte sowie an die Würde und den Wert der menschlichen Person erneut bekräftigt. Sie haben beschlossen, sozialen Fortschritt zu fördern und den Lebensstandard in größerer Freiheit zu erhöhen."

Analyzing Languages with Non-Latin Scripts

Non-Latin Examples (Greek, Chinese, Japanese)

el_text = "Ο σκύλος τρέχει γρήγορα στον κήπο. Οι σκύλοι είναι πιστά ζώα."
zh_text = "狗在花园里跑得很快。狗是忠诚的动物。"
ja_text = "犬は庭で速く走ります。犬は忠実な動物です。"

• Each line means something like: “The dog runs quickly in the garden. Dogs are loyal animals.”

Tokenizing

import spacy

nlp_el = spacy.load("el_core_news_sm")
nlp_zh = spacy.load("zh_core_web_sm")
nlp_ja = spacy.load("ja_core_news_sm")

for lang, nlp, text in [
    ("Greek", nlp_el, el_text),
    ("Chinese", nlp_zh, zh_text),
    ("Japanese", nlp_ja, ja_text)
]:
    doc = nlp(text)
    print(f"\n{lang} tokens:")
    print([t.text for t in doc])

Lemmatizing

import spacy

nlp_el = spacy.load("el_core_news_sm")
nlp_zh = spacy.load("zh_core_web_sm")
nlp_ja = spacy.load("ja_core_news_sm")

for lang, nlp, text in [
    ("Greek", nlp_el, el_text),
    ("Chinese", nlp_zh, zh_text),
    ("Japanese", nlp_ja, ja_text)
]:
    doc = nlp(text)
    print(f"\n{lang} lemmas:")
    print([t.lemma_ for t in doc])

Activity: Tokenize & Lemmatize

• Copy the below sentences, and create lemmatized lists for each language.

cat_el = "Η γάτα κοιμάται στον καναπέ."
cat_zh = "猫在沙发上睡觉。"
cat_ja = "猫はソファで寝ます。"

Transcribing Languages

Transcribing Greek

# pip install Unidecode
from unidecode import unidecode

el_text = "Ο σκύλος τρέχει γρήγορα στον κήπο. Οι σκύλοι είναι πιστά ζώα."
print(unidecode(el_text))

Transcribing Chinese

# pip install pypinyin
from pypinyin import pinyin, Style

zh_text = "狗在花园里跑得很快。狗是忠诚的动物。"
pinyin_list = pinyin(zh_text, style=Style.TONE3)
print(" ".join([syll[0] for syll in pinyin_list]))

Transcribing Japanese

# pip install pykakasi
import pykakasi

text = "犬は庭で速く走ります。犬は忠実な動物です。"

kks = pykakasi.kakasi()
result = kks.convert(text)
romaji = " ".join([item['hepburn'] for item in result])
print(romaji)

Using WordNet

WordNet in Other Languages

• Last time: Open Multilingual Wordnet (OMW) links many languages to the Princeton WordNet synsets.
• Re-run this code and identify the ISO codes for Modern Greek, Mandarin Chinese, and Japanese

import pycountry
from nltk.corpus import wordnet as wn
# List languages available via OMW (IDs)
codes = sorted(wn.langs())

for code in codes:
    lang = pycountry.languages.get(alpha_3=code)
    if lang:
        print(f"{code} → {lang.name}")
    else:
        print(f"{code} → (not found)")

Looking up lemmas by code

• Let’s modify this script to look up the word for ‘dog’ in our three languages

ss = wn.synset('dog.n.01')
[lem.name() for lem in ss.lemmas(lang='spa')]

Activity

• Look up the lemmas in WordNet linked to the basic English synset for cat
• Then, transcribe each word into Latin characters.