Letter Frequency: The Distribution That Cracked Caesar Ciphers and Built QWERTY

In the 9th century, Arab philosopher Abu Yusuf Yaqub ibn Ishaq al-Kindi wrote the earliest known work on frequency analysis. Al-Kindi observed that in any language, certain letters appear more frequently than others, and that this statistical imbalance can break substitution ciphers. His insight — that the most common letter in an encrypted message corresponds to the most common letter in plaintext — became foundational to cryptanalysis. Over 1,200 years later, letter frequency data shapes keyboard layouts, autocomplete algorithms, and natural language processing. {#deck}

1. What Letter Frequency Actually Measures
2. The Historical Origins of Frequency Analysis
3. The English Letter Distribution: ETAOIN SHRDLU Explained
4. How Letter Frequency Is Used in Cryptanalysis and NLP
5. Language-Specific Letter Distributions and Variations
6. Frequently Asked Questions

What Letter Frequency Actually Measures

Letter frequency measures the count or percentage of each letter's occurrence in a given text, relative to the total number of alphabetic characters. It is distinct from word frequency or general character frequency, which include spaces, digits, and punctuation. By focusing exclusively on A-Z characters, letter frequency isolates the linguistic signature of a language's phonological and orthographic structure.

The most famous statistical law governing letter frequency is Zipf's law, formulated in 1935 by American linguist George Kingsley Zipf. Zipf observed that the frequency of any word is inversely proportional to its rank in the frequency table. The same principle applies to letters: E occurs roughly 1.4 times more often than T, which occurs about 1.2 times more often than A, and so on in a power-law distribution. This regularity is remarkably stable across large corpora of natural language text, suggesting deep structural properties of human language.

However, letter frequency is not static across contexts. Several factors introduce variation. Text genre plays a role — scientific papers use technical vocabulary with different letter distributions than literary novels. Authorship introduces stylistic quirks; some writers favor longer words or particular suffixes that shift frequency profiles. Historical period matters because English spelling has evolved significantly — Old English used thorn (þ) and eth (ð), while Middle English spelling varied wildly before standardization. Even medium influences distribution: Twitter posts, constrained by character limits, use more abbreviations and fewer articles than formal prose.

Short texts exhibit high variance. A 50-character text might not contain a single E, while a 1,000-character text will closely approximate the expected distribution. For this reason, reliable frequency analysis requires sufficient sample size. The Brown Corpus (1961, 1 million words), the Corpus of Contemporary American English (COCA, 1990–2019, over 1 billion words), and the Oxford English Corpus (2 billion words) all provide stable reference distributions that differ slightly in exact percentages but converge on the same ranking.

Modern applications depend on these statistical regularities. Spam filters use letter frequency as a feature in classification pipelines. Optical character recognition software applies frequency distributions as Bayesian priors to resolve ambiguous characters. Language detection algorithms compare observed frequencies against known language profiles, achieving over 99% accuracy for texts exceeding 200 characters.

The Historical Origins of Frequency Analysis

Frequency analysis was born in the Abbasid Caliphate during the Islamic Golden Age. Around 855 CE, the Arab polymath Abu Yusuf Yaqub ibn Ishaq al-Kindi (c. 801–873 CE) wrote "A Manuscript on Deciphering Cryptographic Messages" (Risalah fi Istikhraj al-Mu'amma), the earliest known treatise on cryptanalysis. In this work, al-Kindi described the core insight of frequency analysis: "One way to solve an encrypted message, if we know its language, is to find a different plaintext of the same language long enough to fill one sheet or so, and then we count the occurrences of each letter. If one letter occurs more often than others, we assume it corresponds to the most common letter in the plaintext language." This observation, made over 1,100 years ago, remains the foundation of classical cryptanalysis.

The technique was independently rediscovered in Renaissance Europe. In 1466–1467, the Italian architect, artist, and polymath Leon Battista Alberti wrote "De Cifris," a treatise describing both the first polyalphabetic cipher (the Alberti cipher disk) and the method of frequency analysis. Alberti's frequency table for Latin was among the first produced in Europe. Ironically, his invention of the polyalphabetic cipher was a direct response to the vulnerability of simple substitution ciphers to frequency analysis.

Frequency analysis played a decisive role in one of the most consequential espionage cases in British history. In 1586, Mary Queen of Scots used a substitution cipher to communicate with Catholic conspirators plotting to assassinate Queen Elizabeth I in what became known as the Babington Plot. Sir Francis Walsingham's cryptanalyst, Thomas Phelippes, applied frequency analysis to intercept and decrypt Mary's correspondence. The decrypted letters provided irrefutable evidence of Mary's complicity, leading to her trial and execution at Fotheringhay Castle in 1587. This case remains the most famous early example of frequency analysis altering the course of political history.

In 1641, English clergyman and polymath John Wilkins published "Mercury, or the Secret and Swift Messenger," which included detailed frequency tables for English and systematic methods for cryptanalysis. Wilkins categorized ciphers by their resistance to frequency analysis and recommended techniques for making ciphers more secure, such as using homophones (multiple symbols for common letters) and null characters (meaningless symbols interspersed in ciphertext).

By the 19th century, frequency analysis was standard equipment for every cryptanalyst. Edgar Allan Poe, an amateur cryptographer, famously challenged readers of Graham's Magazine in 1841 to submit ciphers he could not break — and systematically solved them using frequency analysis. In his 1843 essay "A Few Words on Secret Writing," Poe described the method in detail, bringing cryptanalysis to a popular audience.

The English Letter Distribution: ETAOIN SHRDLU Explained

ETAOIN SHRDLU is the approximate ordering of the twelve most frequent letters in English, and it originates from a surprising source: the Linotype hot metal typesetting machine. Invented by German-American engineer Ottmar Mergenthaler in 1886, the Linotype revolutionized printing by casting entire lines of type as solid metal slugs. Its 90-character keyboard arranged letters in descending frequency order across vertical columns. The first (leftmost) column contained E-T-A-O-I-N, and the second column contained S-H-R-D-L-U. When a Linotype operator made a composing error, they would run a finger down these columns to produce a visible slug reading "ETAOIN SHRDLU" — which printers used to indicate a mistake that would be discarded.

English Letter Frequency Ranking by Percentage (Corpus-Based)

The dominance of E is dramatic. At 12.7%, E occurs nearly 40% more often than T (9.1%), its nearest competitor. The top four letters — E, T, A, O — together account for 37.5% of all letters in standard English text. By rank 10, cumulative frequency reaches nearly 74%, meaning just ten letters cover almost three-quarters of all alphabetic characters in any typical English document.

These percentages vary slightly depending on the corpus. The Brown Corpus (1961, Henry Kučera and W. Nelson Francis, 1 million words across 15 American English genres) produced the 12.7% figure for E. The COCA corpus, which samples from 1990 to 2019, reports E at 12.5%. The Oxford English Corpus, drawing from global English, shows E at 12.4%. The differences are minor and reflect genre balance, regional spelling variation, and temporal shifts in vocabulary. Despite these small fluctuations, the ranking order is remarkably stable across all major corpora — E always first, T and A always in the top three.

The cultural impact of ETAOIN SHRDLU extends far beyond printing. The sequence appeared in comic strips (Walt Kelly's "Pogo" featured a character named Etaoin Shrdlu), in novels (James Thurber referenced it), and in puzzle culture. It became a nostalgic symbol of the hot metal era, and the term persists in typography circles today as a mnemonic for English letter frequency.

How Letter Frequency Is Used in Cryptanalysis and NLP

Frequency analysis is the most direct method for breaking classical substitution ciphers. For a simple substitution cipher, where each plaintext letter is consistently replaced by a fixed ciphertext symbol, the cryptanalyst counts the frequency of each ciphertext character, sorts them in descending order, and maps the most frequent ciphertext character to E, the second most frequent to T, and so on. With 100 or more characters of ciphertext, this method reliably recovers the plaintext. With 500+ characters, automated frequency matching achieves over 95% accuracy for English plaintext.

For Caesar ciphers, where each letter is shifted by a fixed number of positions, the chi-squared test provides a rigorous statistical approach:

χ² = Σ (Oᵢ − Eᵢ)² / Eᵢ

In this formula, Oᵢ represents the observed count of each letter in the ciphertext, and Eᵢ represents the expected count based on the English frequency distribution. The cryptanalyst computes χ² for each of the 25 possible shifts (excluding the identity shift, which produces plaintext). The shift yielding the lowest chi-squared value is the most likely key. This statistical method is far more reliable than brute-force trial decryption because it quantifies goodness-of-fit rather than relying on human judgment of whether output "looks like" English.

Frequency analysis faces limitations with more sophisticated encryption. Homophonic ciphers assign multiple symbols to common letters, flattening the frequency distribution. Polyalphabetic ciphers, like the Vigenère cipher (invented by Giovan Battista Bellaso in 1553 and misattributed to Blaise de Vigenère), change the substitution at each position, producing a distribution that resists naive frequency matching. Breaking Vigenère requires Kasiski examination or Friedman's index of coincidence — techniques that analyze patterns at the level of repeated sequences rather than single-letter counts. Modern encryption (AES, ChaCha20, RSA) produces ciphertext that is statistically indistinguishable from random, making frequency analysis entirely ineffective.

In natural language processing, letter frequency serves as a foundational feature for several tasks. Language identification algorithms store reference frequency profiles for dozens of languages and compare unknown text against these profiles using Kullback-Leibler divergence or cosine similarity. This technique works even for languages the system was not explicitly trained on, as long as reference frequency tables are available.

Authorship attribution, a subfield of stylometry, has used letter frequency since the 1960s. The landmark 1964 study by Frederick Mosteller and David Wallace at Cornell University applied statistical analysis of function-word frequencies to determine authorship of the disputed Federalist Papers. Their analysis conclusively attributed twelve disputed papers to James Madison rather than Alexander Hamilton. Modern stylometry combines letter frequency with n-gram analysis, syntactic patterns, and machine learning.

OCR systems use letter frequency as a Bayesian prior during character recognition. When a character is ambiguous — for example, distinguishing between 'r' and 'n' in degraded print — the frequency prior biases the decision toward the statistically more common letter in the given language context.

Language-Specific Letter Distributions and Variations

Letter frequency distributions differ markedly across languages, reflecting each language's phonological inventory, orthographic conventions, and historical development. These differences make frequency analysis a powerful tool for language identification and a critical consideration for multilingual text processing.

French distributes vowels differently from English: E reaches approximately 14%, while A stands at 7.5%, S at 7.2%, and I at 6.8%. Accented vowels (é, è, ê, ë) complicate analysis because their treatment varies — some systems collapse them to their base letter, while others treat them as distinct characters. German features the highest E frequency of any major European language at approximately 16.4%, driven by common suffixes such as "-en" (infinitives and past participles) and "-er" (comparatives and agent nouns). German also uses the sharp s (ß), which is sometimes counted separately and sometimes merged with "ss."

Spanish has E at 13.7% and A at 12.5%, with the vowel-heavy distribution reflecting the Romance language family's phonological structure. Italian shows E at 11.8%, A at 11.7%, and I at 10.1% — an unusually even distribution among the top three vowels. The Slavic languages shift the pattern dramatically.

Most Common Letter by Language

Russian is particularly striking: O is more common than E, a reversal of the pattern in nearly every Western European language. This reflects the phonological prominence of unstressed /o/ in Russian and the vowel-reduction patterns of East Slavic languages. O accounts for approximately 10.9% of Russian text, with E at 8.8% and A at 7.6%.

These cross-linguistic differences have practical importance. Multilingual spam filters must maintain separate frequency profiles for each supported language. Search engines use letter frequency as one signal in language detection pipelines, which is essential for serving language-appropriate results. Machine translation systems use source-language identification — often based on letter frequency — as a preprocessing step when encoding metadata is absent or unreliable.

For mixed-language corpora, sliding-window frequency analysis can segment text by language with reasonable accuracy. A window of 100–200 characters slides across the document, and at each position the observed frequency distribution is compared against language reference profiles. Language boundaries are identified where the best-matching language switches. This technique is widely used in preprocessing pipelines for computational linguistics and digital humanities research.

Frequently Asked Questions

Why is E the most common letter in English?

E holds its top position for several converging linguistic reasons. It is the only letter appearing in the definite article "the," the single most common word in English. E dominates verb morphology as the past tense marker "-ed" and the plural or third-person singular marker "-es." As the most frequently used vowel in English orthography, E appears in nearly every common word — from basic function words like "he," "she," "we," and "be" to the core of the lexicon. The widely cited 12.7% figure originates from the Brown Corpus, a landmark linguistic resource compiled in 1961 by Henry Kučera and W. Nelson Francis at Brown University. This corpus analyzed one million words of American English across 15 genres — from press reportage and fiction to scientific writing — establishing the standard reference for English letter frequency that remains the default benchmark today.

What does ETAOIN SHRDLU mean?

ETAOIN SHRDLU is not a word or acronym — it represents the approximate descending frequency order of letters in English, specifically the sequence found in the two leftmost columns of Linotype hot metal typesetting machine keyboards. The Linotype, invented by Ottmar Mergenthaler in 1886, revolutionized printing by casting complete lines of type as single metal slugs. Its keyboard arranged the twelve most common English letters into the first two columns: E-T-A-O-I-N in the leftmost and S-H-R-D-L-U in the second. When operators made errors that could not be salvaged, they would produce a slug reading "ETAOIN SHRDLU" as a visible placeholder indicating a mistake to be discarded. The term migrated into popular culture, appearing in newspaper comics, in James Thurber's writings, and in crossword puzzles throughout the 20th century as a nostalgic touchstone of the hot metal printing era.

How accurate is letter frequency analysis for breaking codes?

For simple substitution and Caesar ciphers, frequency analysis is highly effective when sufficient ciphertext is available. With 100 or more characters, automated frequency matching reliably decrypts the majority of plaintext. With 500 or more characters, success rates exceed 95% for English text. However, accuracy drops sharply with homophonic ciphers, which assign multiple symbols to common letters to flatten the frequency distribution. Polyalphabetic ciphers like the Vigenère cipher, which change the substitution at each position, render direct frequency matching useless — the cryptanalyst must first determine the key length using techniques like Kasiski examination. For short ciphertexts under 50 characters, random sampling variation makes frequency counts unreliable; pattern-based approaches and manual cryptanalysis become more important than pure statistics. Modern encryption algorithms (AES, ChaCha20, RSA) produce ciphertext with statistically uniform distributions, making frequency analysis entirely ineffective.

Does letter frequency differ between British and American English?

Minor but measurable differences exist between the two major English dialects. British English retains the letter 'u' in words derived from French — "colour," "labour," "favour" — slightly elevating U's frequency compared to American English, which standardized to "color," "labor," "favor." British English also uses 's' where American English uses 'z' in verbs such as "analyse/analyze," "organise/organize," and "criticise/criticize." American English doubles the final 'l' less frequently in past tenses (traveled vs. travelled). However, these differences are small — typically less than 0.2% variation per letter. For cryptanalysis, the distributions are functionally interchangeable. For stylistic analysis or authorship attribution, these subtle variations can serve as one signal among many in identifying a writer's national origin, particularly when combined with vocabulary choice and spelling patterns.

How do computers count letter frequency efficiently?

Every modern implementation follows the same O(n) algorithm: allocate an array of 26 integer counters initialized to zero, iterate through the text one character at a time, check whether each character is in the range A-Z after uppercasing, increment the corresponding counter, and finally divide each count by the total number of letters to obtain normalized frequencies. This simple approach processes millions of characters per second on commodity hardware. For Unicode texts spanning multiple scripts, implementations maintain separate counter sets for each script — Latin, Cyrillic, Arabic, Devanagari, and so on — using Unicode General Category properties (Lu, Ll, Lt) to identify alphabetic characters across writing systems. Optimized implementations use SIMD (Single Instruction, Multiple Data) instructions to classify and count 16 to 32 characters in parallel, achieving throughput exceeding 100 million characters per second. For web-based tools, JavaScript implementations using Typed Arrays (Uint32Array) provide near-native performance in the browser.

Related topics:

• Word Frequency Counter
• Character Counter
• Syllable Counter
• Caesar Cipher Shifter
• Hash Generator