The Venona Papers: How cryptologists broke cold war encryption

In July 1995, the congressional U.S. Commission on Government Secrecy ordered the U.S. government to declassify and release what is known as the Venona files.

The Venona project was a counterintelligence program that started during World War II by the United States Army’s Signal Intelligence Service (an early iteration of the National Security Agency). The program ran for roughly 37 years, from 1943 to 1980. The goal of the program was to decrypt the messages being transmitted by the Soviet Union’s three intelligence agencies (the NKVD, the KGB, and the GRU). Ironically, the Venona project was initiated when the Soviet Union and the U.S. were allies, but continued through the Cold War, until late 1980, when it was discontinued.

Throughout the Venona project, the Signal Intelligence Service was able to decrypt and translate around 3000 Russian messages. Those intercepts enabled the discovery of Soviet espionage of the Manhattan Project in the U.S. (a nuclear weapons development program) and the Cambridge Five espionage ring in the United Kingdom, to name a few.

In this post, we’re going to provide a brief overview of the Venona project, and look at the encryption that was used and how it was deciphered, because… well because we like encryption and it’s kind of cool to get a glimpse of decades-old, yet remarkably sophisticated spycraft. We’ll also look at some of the historical aftermath of the Venona project.

How the Venona Project began

On February 1, 1943, Colonel Carter Clarke, the chief of the U.S. Army’s Special Branch, a sub-department of the War Department’s Military Intelligence Division, tasked the U.S. Army Signal Intelligence Service with a secret mission: to intercept and decipher cables being sent and received to and from the U.S. by the Soviet Union. This was the genesis of the Venona project.

At the time, Clarke’s main reason for initiating the Venona project was to find out if the rumors that the Soviets and the Germans were negotiating a separate peace were true. He ordered his small team of code-breakers, working from their headquarters in Arlington Hall, to read all Soviet diplomatic messages being sent from the United States to Moscow. And they did. For months they worked on thousands of Russian diplomatic cables they were able to pick up over the wires. And they eventually succeeded in cracking the code. But to their surprise, they didn’t discover evidence of a Russian-German peace deal. Instead, they uncovered a large-scale, well-organized Soviet espionage ring operating at the highest levels of the U.S. government.

From there, the Venona project set out to uncover as much information as possible on Soviet spying activities in the U.S. and some of its close allies (namely, the U.K., Canada, and Australia). The majority of the messages Venona deciphered were sent between 1942 and 1945. In 1945, however, the Soviets became aware of the fact that the U.S. could at least partially decrypt some of their messages. They made some changes to their encryption scheme while continuing to use the “broken” one to send certain messages in an attempt to keep the Americans from the fact that they knew what was going on.

In 1949, the Venona project made a significant breakthrough in its deciphering efforts when it was able to decrypt a large enough portion of a Soviet message. It was able to identify the message as the text of a telegram, from 1945, between Winston Churchill and Harry Truman. And from there, they were able to able to unmask many double agents within the highest spheres of the U.S. government, and prompted the beginning of the anti-communist hysteria that became McCarthyism.

The cipher

What was encryption like in the 1940s, and how was it cracked?

The encryption scheme used by the Soviets at the time should have been unbreakable. It used a method referred to as “one-time pads”. First, the words were converted into numbered sequences. After that, a “one-time pad”, literally a pad of paper with random characters on it, was added to the mix, meaning the random characters on the one-time pad were added to the enciphered words to obfuscate and add complexity to the encryption scheme. In modern cryptography, this is called salting.

Once those random characters were added and the message was sent, the paper would be torn off and destroyed. There were only supposed to be two copies of each pad before their destruction: one for the operative and one for Moscow headquarters. The one-time pads were what guaranteed the integrity of the encryption – with all emphasis on the “one-time” part of the expression, as we’re about to see.

Breaking the cipher

At first, it really was unbreakable. Initially, all the Venona code-breakers could do was sort the intercepted traffic based on its point of origin, which could reveal whether the message was related to trade or diplomacy. As the end of the war drew closer, the Venona team grew, and more experienced cryptanalysts were assigned to the project. That’s when Lt. Richard Hallock, who has studied archeology and had translated texts from Babylonian dialects to English, was brought into the project.

Though there shouldn’t have been any repetitions in the number sequences, Hallock decided to troll through the traffic methodically to try and find such repetitions, just in case the Soviets had gotten sloppy at one point. After all, human error will always be your best bet for getting at human secrets. And his assumption was correct. Hallock ended up finding repetitions in the code. He only found a handful of them. But it was enough to determine that the Soviets had used some one-time pads more than once.

The reuse of one-time pads by the Soviets apparently started happening around 1941, when the Soviet capital was under siege by German forces. Moscow started distributing extra copies of the “one-time” pads, for easier communication between more parties. And the Americans were further helped after the war had ended in 1945 when U.S. military intelligence in Germany managed to get its hands on a partially burned Soviet codebook, obtained by the Nazis. So with the compromised one-time pads in one hand and the partially recovered Soviet codebook in the other, the Venona project, which was now bigger than ever, churned away at their IBM punch-card computers in an effort to decipher more Soviet messages.

In 1946, Meredith Gardner, a linguistics professor turned codebreaker after being recruited by the Venona project, made a huge breakthrough just a few months after he was transferred there.

What Gardner discovered was this: When the Soviets communicated in Russian only, their code was unbreakable. But they sometimes needed to transmit an unfamiliar name or concept. What they would do in those cases was to spell it out, one coded letter at a time. But they also needed a way to let the recipient know what they were doing. So they would prefix the spelled-out word or expression with “spell” and follow it with “endspell”.

So Gardner would look for the “spell/endspell” prompts within the intercepted Soviet cables. And by focusing his attention on the coded text in between, Gardner timidly started to decipher the code and begin to understand the Soviet codebook.

We also know, from the Venona files, that the code breakers developed some home brew tools to assist them and increase the number of code breaks. One such tool was a key device referred to as a “window index.” It worked as follows:

Each time a word or phrase was deciphered, it was indexed everywhere else it appeared in the intercepted traffic. The “window index” was used in a variety of ways. One of the ways it was used was by placing two unsolved groups of cipher text on each side of a decrypted word or phrase (an index). By continually repeating this process, these window indexes eventually led to repetitions, in which different words that had been previously deciphered were followed by the same unsolved sequence of cipher text. The repetition sometimes yielded enough collateral information to begin deciphering the unbroken cipher text, and in turn augmenting the window indexes.

Another technique used was called “dragging.” Whenever an unknown “Spell/Endspell” sequence came up, the “spelled-out” cipher text sequences would be cross-referenced against the rest of the intercepts (the “dragging”), using a computer, which would then output a list of all the repeats. The code breakers would then work to decipher the text on either side of the matches in the hope of eventually deciphering the “Spell/Endspell” sequence once the surrounding text was readable.

It was a tedious and time-consuming effort to be sure. And with the help of the FBI, who was tasked with providing collateral information that could provide context to the partially decoded messages in the hopes of deciphering more of them, Gardner was eventually able to decipher the contents of an entire message. It was a message sent in 1944, in which the Soviets predicted that Franklin Roosevelt would win the popular vote in the upcoming election, but that he would nonetheless lose in the electoral college.

The code was broken.

After the Venona Project

Shortly after that, in 1946, Igor Gouzenko, a cipher clerk working at the Soviet embassy in Ottawa, Canada, defected from the USSR. He revealed to the West a massive Soviet espionage operation against its allies, particularly the United States, that aimed to gain insights into American nuclear secrets. This gave a major push to Venona efforts and the project started identifying (with sometimes debatable certainty but with unquestionable zeal) and prosecuting Soviet spies that had infiltrated the U.S. government and scientific community.

Many of the names of those unmasked by Venona are well-known to history. Here are some of the most talked about:

Julius and Ethel Rosenberg were convicted of espionage in 1951 and executed in 1953. Julius Rosenberg was an engineer inspector for the Army Signal Corps who was accused of supplying the Soviets with information on radar, sonar, jet propulsion engines, and nuclear weapon designs. Ethel Rosenberg, charged as an accessory to her husband’s spying activities, worked as a secretary for a shipping company. Both were members of the Communist party. The Rosenbergs maintained their innocence until their deaths.

David Greenglass was the brother of Ethel Rosenberg and, like his sister, was also a member of the Communist party. He worked as an engineer at the Las Alamos Laboratory in New Mexico as part of the Manhattan Project. He was accused of Soviet espionage, like his sister and brother-in-law, but made a plea deal and provided testimony to help convict the Rosenbergs. He served a nine and a half year prison term.

Klaus Fuchs was a German theoretical physicist, also working out of Las Alamos as part of the Manhattan Project. He was accused of supplying the Soviet Union with information on the American, British, and Canadian Manhattan Project. He was convicted of espionage in 1950. He was jailed for 9 years in the U.K. before being released and returning to Germany.

Undoubtedly, a lot of those convicted after being uncovered by Venona were actual Soviet spies. But while some of the Venona papers make a clear case against many of those prosecuted (including the Rosenbergs), their convictions nonetheless raised some controversy because of the emerging anti-communist zeal that was going into overdrive at the time. And in retrospect, the incredibly small number of intercepts that Venona was actually able to decrypt also casts a shadow of doubt on some of these convictions.

Here are the percentages of messages deciphered by the Venona project, based on the year in which they were sent:

• 1942 1.8%
• 1943 15.0%
• 1944 49.0%
• 1945 1.5%

It’s not very much, but it’s enough to paint a picture – even if incomplete. An incomplete picture that morphed into a witch hunt (communist witches, to be precise). And while nobody celebrates McCarthyism (and rightfully so), it’s nonetheless interesting to trace back part of its origins to a little-known Cold War code-breaking program. I think it can also shed some light on the current state of relations between Russia and the U.S.: distrust, interference, and election-meddling… And it can also perhaps partly explain the U.S. government’s historical distaste for encryption. Nobody knew it at the time, but this was the beginning of the Crypto Wars.

See also: Beginner’s guide to cryptography