Improved Decipherment of Homophonic Ciphers

Abstract

In this paper, we present two improvements to the beam search approach for solving homophonic substitution ciphers presented in Nuhn et al. (2013): An improved rest cost estimation together with an optimized strategy for obtaining the order in which the symbols of the cipher are deciphered reduces the beam size needed to successfully decipher the Zodiac-408 cipher from several million down to less than one hundred: The search effort is reduced from several hours of computation time to just a few seconds on a single CPU. These improvements allow us to successfully decipher the second part of the famous Beale cipher (see (Ward et al., 1885) and e.g. (King, 1993)): Having 182 different cipher symbols while having a length of just 762 symbols, the decipherment is way more challenging than the decipherment of the previously deciphered Zodiac-408 cipher (length 408, 54 different symbols). To the best of our knowledge, this cipher has not been deciphered automatically before.

Decipherment of Substitution Ciphers with Neural Language Models

“…Our neural LM model with global rest cost estimation and frequency matching heuristic with a beam size of 1M has SER of 1.2% compared to the beam search algorithm (Nuhn et al, 2013) with beam size of 10M with a 6-gram LM which gives an SER of 2%. The improved beam search (Nuhn et al, 2014) with an 8-gram LM, however, gets 52 out of 54 mappings correct on the Zodiac-408 cipher.…”

“…2.2 Beam Search Algorithm 1 is the beam search algorithm (Nuhn et al, 2013(Nuhn et al, , 2014 for solving substitution ciphers. It monitors all partial hypotheses in lists H s and H t based on their quality.…”

“…Score estimation evaluates the quality of the partial hypotheses φ. Using the example from Nuhn et al (2014), consider the vocabularies V e = {a, b, c, d} and V f = {A, B, C, D}, extension order (B, A, C, D), and ciphertext $ ABDDCABCDADCABDC $. Let φ = {(a, A), (b, B))} be the partial hypothesis.…”

“…As a heuristic, n-grams which still consist of unsolved cipher-symbols are assigned a trivial estimate of probability 1. An improved version of rest cost es-timation (Nuhn et al, 2014) consults lower order n-grams to score each position.…”

“…Nuhn et al (2013) produce better results in faster time compared to ILP and EM-based decipherment methods by employing a higher order language model and an iterative beam search algorithm. Nuhn et al (2014) present various improvements to the beam search algorithm in Nuhn et al (2013) including improved rest cost estimation and an optimized strategy for ordering decipherment of the cipher symbols. Hauer et al (2014) propose a novel approach for solving mono-alphabetic substitution ciphers which combines character-level and word-level language model.…”