My work

[Quicktrader]

@Quicktrader,

The way I see it is that the solver uses another n-gram list for the n-grams you wish to mark. You supply this list yourself so you can do whatever you want. For instance I make a secondary 5-gram list with entries: ZODIAC, OFTHE, ABCDE. The solver will select a 5-gram from that list randomly to the 5-grams you marked in the cipher everytime it makes a change. So eventually it'll end up with the best fitting 5-gram for that piece of ciphertext. Is that okay?

Hi Jarlve,

that's absolutely perfect..

I currently do the logical way via Excel - and it's simply too slow. The procedure, however, is quite simple and efficient, imo:

- Setting the IoFBc 5-gram plus a + letter (e.g. 'L') delivers some letters
- Adding the w symbol plus the circle with the horizontal line adds some more (each A-Z)
- Choosing finally the reverse L symbol gives another one

This is what I can come up with, so far (line 17 & 18, here together in one line):

SAMPLE.jpg

The colored ones I got from the 5-gram plus the + symbol. All others are not colored. Nevertheless, by defining the 5-grams (plus +) as well as only three (!) other symbols, I get a variety of potential cleartext results of 12 letters (orange line). It should be considered that the circle with the horizontal line shall, in case of + being an 'L' - be no other alphabetical letter than AEIOURWY, thus most likely is a vowel (for + being an 'L').

Even if the IoFBc section is not a double frequent trigram, it may well be expected that it is amongst the 20 if not 100 most frequent 5-grams (consisting of frequent trigrams). I got some solid statistics about that and will post it below.

This leads me to approximately half a billion varieties. The procedure can be repeated for a + being a 'T' or 'E'.

Currently I do get loads of results, those I select should contain either longer words (e.g. >5 letters) or two words (e.g. both >4 letters). This is guessing. The results are astonishing: '..AGENTCALL..' and similar results are not rare at all. Actually there are plenty of them, however it's about finding the correct one somehow.. Changing the final letter currently takes me 15 minutes of Excel's calculation time (each).

Still collecting results, so nothing in detail so far. But I write this because a 5-gram function indeed would support your superquick tool..thanks for your efforts.

QT

[Quicktrader]

This is the complete list of 5-grams ('IoFBc') consisting of frequent 3-grams 'IoF' and 'FBc'.

Those 5-grams consisting of BOTH 3-grams which an expectation to appear >1 in the 340, have a control number '1'.

Those with an expectation between 0.2 and 1 to appear in the 340 are shown with a control number '0'.

All other 5-grams are extreme outliers, either because the 5-gram has not even appeared 2,000 times in the Gutenberg project (source), or because they consist of at least one 3-gram with an expectation of less than 0.2 to appear in the 340. Both trigrams should actually be expected to appear at least 1.5 times, instead (as they are present two times - possibly even more are shuffled behind different homophones).

Statistically it can actually be said that the 'IoFBc' 5-gram is amongst the ones with the control value '1'.

The large number shows how frequent the 5-grams actually are for themselves, as a 5-gram. Some of them appear even 4.27m times, so it should be quite ok to ignore those with a frequency of 2,000 or less.

list.xlsx

file has only values, no formulas

QT

[Jarlve]

Thanks for the information about your GPU solver daikon, sucks that it didn't came through.

Quicktrader, you'll have to excuse me for a while.

[Jarlve]

I'm happy to share a new set of letter n-grams (weeks of work) compiled from the complete Reddit 2007-2015 comment corpus from: https://archive.org/details/2015_reddit_comments_corpus

The Reddit dataset is (about) 150GB compressed and 1TB uncompressed. From that I extracted the comments which ended up being 264GB. I then divided everything into sentences, and removed Reddit specific formatting, messages, weblinks and etc.

Only the best sentences were kept with a minimum of 2 words. Sentences containing numerical information have been discarded entirely. Next up is a sample of sentences starting with the letter "B", the general quality of the sentences is really as high as this.

Code: Select all

Better we interpret the utterings of two- and three-year-olds for true enlightenment.
But it wouldn't be research if it were oh-so-easy.
Bottom line is that the guy must have known, **absolutely**, that what he was doing was totally wrong, and would get him in deep shit.
Bearing in mind, as I said, that his treatment seems to have been over the top.
But then I get over it.
But if I pretend I don't get frustrated then I'd be deceiving you and myself.
Better even, as you wouldn't be locked with a Zune.
But yeah, I think at some point we have to realize that we are not the best custodians of ourselves.
BTW does anyone believe that the presidential election is one round, one vote per se?
Big fucking surprise, down modded by a bunch of ill-informed dipshits.
Being a pornographer is the only way a person can lawfully approach random girls on the street and offer to pay them to have sex.
But of course, if your notion of programming is sing trial and error to hack around bugs in crappy browsers, I'm sure things like this might be a little too high level.
Because the FBI's job is *not* to enforce private resource center's terms of use.


The resulting file is 149GB of sentences only containing letters "ialsowonderwhatthedifferencesare". From that files I compiled the letter n-grams (sorted by frequency) without crossing sentence boundaries. My first general impression of these n-grams are that they are very good for (not sure how to call it) non-formal language. Getting improved results for most ciphers but a few may be worse (daikon3).

Compressed with 7-zip format.

Download 1-grams.
Download 2-grams.
Download 3-grams.
Download 4-grams.
Download 5-grams. (21MB download, 76.7MB unzipped)
Download 6-grams. (153MB download, 654MB unzipped)
Download 7-grams. (790MB download, 3.13GB unzipped)

I'm now working on a new iteration of AZdecrypt which uses some of these new n-grams and I'm also planning some kind of comparetive test to see what sets of n-grams different ciphers prefer.

[daikon]

Wow, that's impressive! I'm definitely downloading. I think I've came across this corpus, but thought it would be too large to process. The corpus I'm using right now is "only" 28Gb (Usenet corpus + a bunch of books from Project Gutenberg). I might combine your N-grams with mine. Reddit comments should be a good match for the type of language used by Zodiac in his writings. Might not fit all ciphers though.

Did you run an IoC filter on the sentences? It helps weed out comments like "hahahahahahahahahahahaha, that's funny", etc., which surprisingly there is a fairly good amount of, at least in the Usenet postings. Check the N-grams for obvious noise. For example, 5 and more of the same letter repeating ("AAAAA".."ZZZZZ"). I've also done a 3-gram filter on all the sentences to weed out binary data (uuencoded/base64), but that might not be necessary for Reddit comments.

[Jarlve]

Thanks daikon,

It's not a problem to process. The files are split up per month. I didn't run an IoC or n-gram filter on the corpus so there's certainly stuff like "hahahaha" and "zzzzzzzz" in there (I think it's okay). Though I've looked over these sentence files for days and can safely say that the general quality is very high. Though there appear to be not many numerals, and ciphers that contain them have problems.

I looked for "aaaaa", "zzzzz", "hahaha" and these are some of the sentences found. Almost every time it's part of a sentence which I feel is genuine. There's also quite a bit of redundancy in there, it's the Reddit corpus after all.

Code: Select all

Very interested in seeing this but maaaaaaaaaaaaaybe keeping Jessie Jackson's corrupt ass off of there would've been a good idea.
Vivianaaaaaaaaaaaaa where's my red bull.
Value aaaaaaaaaaaall day long.
Vote to change it, it's the perfect scheme muahahahahaha.
Vroom vroom and zzzzzzzzz.
Volume on chinese exchanges is like zzZzZzzZ.
Video plzzzzz.


Try this corpus, very high quality texts, needs almost no processing at all: http://ebiquity.umbc.edu/redirect/to/re ... ase-corpus (12.8GB download)

The wikipedia corpus is also out there but I really disliked it.

[glurk]

Just a brief comment, but I think if you are using a GB+ sized corpus, a few "hahaha" and etc. probably won't matter that much.
I personally got much better results in ZKD, though, when I log-weighted the scores, very simply, like:

VALUE=(int)10*log((double)VALUE);

-glurk

[Jarlve]

I copied the log thing from ZKD. When memory allows I prefer a higher multiplication factor though.

This is a solve for the first 7 rows of the 408 using the Reddit n-grams.

Code: Select all

Score: 24586 Ioc: 615 M: 428 C: 119 S: 51

ilikekillingpeopl
ebecauseitissomuc
hbutitismorefunth
ankillingwildgamb
itthebiggestbecau
semanisthemoatran
gertubanamalofall

9%P/Z/UB%kOR=pX=B
WV+eGYF69HP@K!qYe
MJY^UIk7qTtNQYD5)
S(/9#BPORAU%fRlqE
k^LMZJdr\pFHVWe8Y
@+qGD9KI)6qX85zS(
RNtIYElO8qGBTQS#B


[Jarlve]

This surprised me, here is a 6 row 408 solve.

Code: Select all

Score: 26465 Ioc: 607 M: 500 C: 102 S: 51

ilikekillingpeopl
ebecauseitissomuc
hfunitismorefunth
atkillingwildgame
intheforbestbecau
semanisthemoatwat

9%P/Z/UB%kOR=pX=B
WV+eGYF69HP@K!qYe
MJY^UIk7qTtNQYD5)
S(/9#BPORAU%fRlqE
k^LMZJdr\pFHVWe8Y
@+qGD9KI)6qX85zS(


[doranchak]

Impressive work, Jarlve! Especially considering the multiplicity of those 7 lines is about 0.42. The multiplicity of the first 6 is 0.50. Very nice!

Thanks for taking the time to do this.

I've been wanting to rule out Dan Olson's hypothesis on splicing together lines 1-3 and 11-13 of Z340. But the multiplicity is 0.52. Perhaps you can unleash your improved solver on it.

I'm going to produce all permutations of those 6 lines for you with some further thoughts.