Decipherment

Day 11: Frequency

Dr. Andrew M. Byrd

Review

Review

With your neighbor, briefly define:

  • Alphabet
  • Abjad
  • Alphasyllabary (abugida)
  • Syllabary
  • Logographic system

Which inventories have the smallest number of characters? Which ones have the largest?

Identifying the Writing System

Examine the following writing systems. Identify (i) the type of writing system and (ii) its linear orientation.

Language A

Text

ᒥᓯᐌ ᐃᓂᓂᐤ ᑎᐯᓂᒥᑎᓱᐎᓂᐠ ᐁᔑ ᓂᑕᐎᑭᐟ ᓀᐢᑕ ᐯᔭᑾᐣ ᑭᒋ ᐃᔑ ᑲᓇᐗᐸᒥᑯᐎᓯᐟ ᑭᐢᑌᓂᒥᑎᓱᐎᓂᐠ ᓀᐢᑕ ᒥᓂᑯᐎᓯᐎᓇ᙮ ᐁ ᐸᑭᑎᓇᒪᒋᐠ ᑲᑫᑕᐌᓂᑕᒧᐎᓂᓂᐤ ᓀᐢᑕ ᒥᑐᓀᓂᒋᑲᓂᓂᐤ ᓀᐢᑕ ᐎᒋᑴᓯᑐᐎᓂᐠ ᑭᒋ ᐃᔑ ᑲᓇᐗᐸᒥᑐᒋᐠ᙮

Transcription

Misiwe ininiw tipênimitisowinik êshi nitawikit nêsta pêyakwan kici ishi kanawapamikowisit kistênimitisowinik nêsta minikowisiwina. Ê pakitinamacik kakêtawenitamowininiw nêsta mitonênicikaniniw nêsta.

Language B

Text

Transcription

Motun inkanun thayenal ttaypwuthe cayulowumye ku conemkwa kwelliey isse tongtunghata. inkanun chenpwucekulo isengkwa yangsimul pwuyepatassumye selo hyengceyayuy cengsinulo hayngtonghayeya hanta.

Language C

Goals for Today:

  • Use reasoned judgment to identify a type of writing system, even without transcriptions/explanations/context, using:
    • symbol inventories
    • frequency
  • Learn the difference between tokens & types

Frequency

How Many Words are in this Passage?

In the beginning God created the heaven and the earth. And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters. (Genesis 1:1-2)

Tokens: 39

In the beginning God created the heaven and the earth. And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters. (Genesis 1:1-2)

Types: 20

In the beginning God created the heaven and the earth. And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters. (Genesis 1:1-2)

Types

  • ‘the’: 9
  • ‘and’: 4
  • ‘of’, ‘God’, ‘earth’: 2
  • others: 1

Token frequency data

Note the Shape of the Curve

Shape of Graph = Zipfian distribution

Zipf’s Law

The most common word occurs approximately twice as often as the next common one, and so on.

Activity: How Many Tokens & Types are in this Passage?

And God said, Let there be light: and there was light. And God saw the light, that it was good: and God divided the light from the darkness. (Genesis 1:3-4)

Activity Discussion

Does the distribution follow Zipf’s Law?

Why No Zipf?

No Zipf?

  • Corpus might be too small!
  • Let’s look at the entire King James Bible

Looking at a Larger Corpus (Top 20)

Total word tokens: 795227
Total word types:  12870

Looking at a Larger Corpus (Top 100)

Total word tokens: 795227
Total word types:  12870

Looking at a Larger Corpus (Top 500)

Total word tokens: 795227
Total word types:  12870

Zipf’s Law & Writing System Type

Where Would We Expect Zipf’s Law to Appear Most Prominently?

  • Alphabet
  • Abjad
  • Alphasyllabary (abugida)
  • Syllabary
  • Logographic system

Zipf’s Law & Writing System Type

Where Would We Expect Zipf’s Law to Appear Most Prominently?

  • Alphabet
  • Abjad
  • Alphasyllabary (abugida)
  • Syllabary
  • Logographic system

Frequency and Phonographic Systems

Think back to Cryptography…

Was there a letter in English that was the most common? How about the least?

How Many Letters are in this Passage?

In the beginning God created the heaven and the earth. (Genesis 1:1)

Tokens? Types?

Tokens: 44

In the beginning God created the heaven and the earth. (Genesis 1:1)

Types: 13

In the beginning God created the heaven and the earth. (Genesis 1:1)

Letter Frequency

Letter Frequency: The Entire KJB

Letter Frequency: die Bibel (German)

Letter Frequency: First 4 Books of the Filipino New Testament

How about a Syllabary?

The Takeaway: Logographic vs. Syllabary vs. Alphabet

  • Number of Characters
  • Frequency of Characters

Activity

Activity Instructions

  • We’ll split the room up into three groups. Each group will take a text.
    • Close your laptops; put away your phones!
    • In your group, divvy up labor as you see fit.
  • Examine the symbols in your text, and identify:
    • How many tokens?
    • Which tokens are the most common? The least?
    • What type of writing system is it? What is its linear orientation?

Frequency: Text 1

Frequency: Text 2

Frequency: Text 3

Total tokens: 157
Total types: 83
Top 50 symbols:
✏   11
✐   10
✝   8
✛   7
✑   5
✧   5
✫   5
✯   5
✊   4
✍   3
✇   3
✗   3
✂   3
✙   3
✡   3
✌   2
✓   2
✔   2
✕   2
✠   2
✢   2
✤   2
✮   2
✰   2
✲   2
✹   2
✎   1
✃   1
✒   1
✖   1
✘   1
✚   1
✄   1
✜   1
✞   1
✟   1
✋   1
✉   1
✣   1
✥   1
✦   1
✁   1
✨   1
✩   1
✪   1
✬   1
✆   1
✭   1
✈   1
✱   1

Frequency: Text 3