Unigram distance curiosity

[smokie treats]

Are you sure about the above rearrangement of rows? Because I tried it and got very slightly lower cycle score overall.

[doranchak]

Thanks for checking. I think the reason the modified cipher scores highly for me is because its cycles are more statistically significant, rather than simply being more numerous.

Here's my methodology:
1) Generate stats for a bunch of shuffles, for all cycles of length 2.
2) Stats are collected for the "runs" of cycles. Example: "AAA AB AB AB BAB" has a max run length of 3 because of [AB] [AB] [AB].
3) In the shuffles, average and standard deviation of max runs are computed for every pair of symbols.
4) Then, actual measurements of a target cipher's cycles are compared to the shuffle stats.
5) The target cipher will have observed runs for each pair of symbols. The observed runs are compared to the mean during shuffles. Then I count how many standard deviations (sigma) the observed measurement is from the average runs observed during shuffles.
6) For each symbol pair, I compute sigma. So, this gives a relative sense of the significance of each observed cycle.
7) Then I compute the average of all the observed sigma values (over all symbol pairs).

I believe using standard deviation from the shuffle stats has a normalization effect on the cycle measurements. This helps filter out some of the noise of false cycles.

Computed this way, the row-swapped Z340 has about double the average sigma as the unmodified Z340. You can see this looking at individual cycles, where the overall relative probabilities are, on average, less than those of the Z340. Here is the raw output of cycles of both ciphers for comparison:

https://docs.google.com/spreadsheets/d/ ... sp=sharing

It shows all L2 cycles detected for both ciphers, side by side, in decreasing order by estimated probability. When you scroll down, you'll notice an emerging trend for cycles to become more improbable in the row-swapped Z340. Look at the "How much more improbable" column. Positive values indicate an increase in improbability. There are also 172 extra cycles in the row-swapped Z340 compared to the original.

So, I guess this boils down to a philosophical question: Can a cipher be considered "more" homophonic if it doesn't really show that many more long cycles BUT has more improbable cycles overall?

All I know is that the "average sigma" measurement scores very high for Z408 compared to Z340 (0.68 vs 0.21).

[Jarlve]

Good luck with your talk next week doranchak!

Thank you for the test, the results are very clear. Rearrangement of rows is a possible cause for the high unigram distance in the 340 as it could either decrease or increase the value. Your test suggests that the value of the 340 is however quite rare and therefore more unlikely to have been caused by rearrangement of rows, right? Your example cipher decreases the unique sequence length 17 repeats from 26 to 18 and since we know that is a very significant observation perhaps you could use it to filter out false positives.

Here are my 2 most likely hypotheses for the high unigram distance in the 340:

1. The high unigram distance of the 340 is related to the group of symbols that do not appear in the middle 7 rows (as marie said).
2. A long key (as Largo said) is used in the homophonic substitution process and some of the most frequently occuring symbols are for whatever reason not part it or are wildcards. Likely hypotheses for the symbols that were not taken up in the homophonic substitution or are wildcards are that these are 1:1 substitutes, plaintext nulls or wildcards (as smokie said). I would like to go as wide as possible with the interpretation of wildcards.

In your test my cipher jarlve2 very closely matches the 340 which is hypothesis 2. Some things to look in may be exotic cycling types (again) such as palindromic cycling which also increases unigram distance and not trying to repeat symbols in a certain view window as opposed to actively cycling homophones.

[smokie treats]

Thanks for the explanation about how you score the cycles.

I re-drafted it into 5 columns, and you can really see it. Tonight I will re-draft into different column counts to look for a pattern. Then I will explore in other ways.

unigram distance phenomenon.1.png

[Mr lowe]

hi smokie nice work. maybe mark up the pivots in a separate spreadsheet for any visual patterns as well

[doranchak]

That's interesting, smokie. It reminds me of the "multiples of 5" harmonic that occurs when normalizing columnar IoC measurements:

http://www.zodiackillersite.com/viewtop ... 442#p48442



I am not sure what, if any, significance there is to this phenomenon.

[doranchak]

Good luck with your talk next week doranchak!

Thank you!

Thank you for the test, the results are very clear. Rearrangement of rows is a possible cause for the high unigram distance in the 340 as it could either decrease or increase the value. Your test suggests that the value of the 340 is however quite rare and therefore more unlikely to have been caused by rearrangement of rows, right?

I really don't know. I have become very reluctant to make any conclusions based solely on the rareness suggested by shuffle tests, because there are other considerations that must be made. For example, throwing a golf ball into a field of grass will hit a blade of grass. From the blade's perspective, it was a rare one-in-a-billion event. But from the golf ball's perspective, it was almost certainly going to hit a blade of grass. So when I see a rare phenomenon, I have to wonder: Are there many other blades of grass I'm not considering?

Your example cipher decreases the unique sequence length 17 repeats from 26 to 18 and since we know that is a very significant observation perhaps you could use it to filter out false positives.

That might be an effective method, but I also wonder if it is a side effect of the procedure that weakened the cycles and/or bigrams.

Here are my 2 most likely hypotheses for the high unigram distance in the 340:
1. The high unigram distance of the 340 is related to the group of symbols that do not appear in the middle 7 rows (as marie said).
2. A long key (as Largo said) is used in the homophonic substitution process and some of the most frequently occuring symbols are for whatever reason not part it or are wildcards. Likely hypotheses for the symbols that were not taken up in the homophonic substitution or are wildcards are that these are 1:1 substitutes, plaintext nulls or wildcards (as smokie said). I would like to go as wide as possible with the interpretation of wildcards.
In your test my cipher jarlve2 very closely matches the 340 which is hypothesis 2. Some things to look in may be exotic cycling types (again) such as palindromic cycling which also increases unigram distance and not trying to repeat symbols in a certain view window as opposed to actively cycling homophones.

Interesting ideas - these sound worth exploring. I wonder if Zodiac would have bothered to hide the cycling effect of symbol assignments. He may not have been aware of that particular weakness of homophonic substitution (i.e. brute force searches of cycles), since Z408 was broken mostly with identification of other repeating patterns and trial and error guesses of certain words and phrases.

[smokie treats]

I re-drafted into 4 to 85 columns and didn't see any recognizable patterns.

I think that if you slid a lot of plaintext through a 17 x 20 grid you would easily find low frequency letters that appear at the top and bottom of the message, but not in the middle. If you encoded a lot of plaintext in chunks of 340 they would show through. But... packing 63 symbols into a key to get similar symbol count distributions maybe they wouldn't show through in pairs like this. I noticed that most of the same symbols appear at both the top and the bottom of the message. It's not like some symbols appear at the top and some at the bottom but with the same overall high score. There is a bi-regional bias.

Maybe the score should reflect whether the symbol appears in both the top and bottom half of the message ( if it doesn't already ). The test could be: 1. Does the symbol appear in both top and bottom half? 2. If yes, then calculate the distances.

We tried palindromic cycling and determined that it is not. But maybe there is some other creative cycling method.

I think that it could be evidence of the cipher. I skim or read through writings about cryptography hoping to trigger recognition or ideas. I am reading through Mysterious Messages by Gary Blackwood right now. I found it in the children's section of the library. It's not advanced mathematics or cryptanalyst stuff, but tells a lot of interesting stories about types and uses of cryptography since the days of the ancient Greeks. People have created a lot of different types of ciphers on their own, to be used and sometimes forgotten for centuries.

I think that there is a good possibility that Zodiac just combined some classical ciphers. Homophonic was used in the Renaissance era. Leon Battista Alberti also invented his disk for Pope Paul II in 1466. It was used to make polyalphabetic cryptograms, but not necessarily by rotating the disk at each plaintext. It could be rotated after an agreed count of plaintext or words.

I wonder if it could be Alberti + homophonic that creates a phantom spike in period x repeats. Or transposition + Alberti + homophonic, but just only a few rotations of the disc. I can imagine Zodiac making a cardboard Alberti disk. I wonder if such a cipher could make one symbol unlike the others ( the + ). And then there is the regional bias of the period 39 repeats to consider.

Speaking of the +, one time a couple of years ago I generated route transposition + homophonic messages, then randomly plopped 24 of one symbol into the message. There weren't very many bigram repeats at the route period that had the + included. But with the 340, there are.

EDIT: If Zodiac had a book about cryptography that discussed ciphers in chronological order, then Alberti would have been right next to homophonic. Just saying.

[smokie treats]

I encoded some messages from the library all perfect cycles but with one polyphone and got scores higher than 15034. It was pretty easy.

Message 92 score 15411

46 40 42 24 32 18 20 1 9 7 36 30 11 50 37 51 21
12 5 2 43 38 33 59 39 16 8 3 34 52 35 13 57 19
44 4 47 48 58 11 56 14 10 6 15 60 36 32 9 53 22
11 49 23 12 13 41 17 37 28 10 11 1 33 9 50 20 14
16 29 38 7 26 46 17 2 25 45 28 59 42 3 34 51 39
52 21 15 47 22 11 4 43 48 12 11 18 13 44 53 36 5
14 16 45 15 11 37 17 50 23 12 24 42 57 38 39 29 60
58 25 35 51 13 43 8 36 1 52 49 53 20 14 11 50 44
15 2 31 46 47 3 32 19 37 16 30 11 28 51 24 33 18
48 34 38 57 4 35 10 21 25 19 22 7 29 39 54 9 49
18 45 11 61 12 10 52 23 13 40 1 11 53 55 42 14 46
36 17 50 20 15 47 27 59 37 32 28 60 2 33 11 3 19
29 12 51 43 38 54 6 28 13 9 52 21 14 48 41 44 24
34 18 53 25 31 15 11 4 49 12 39 16 40 11 11 50 55
45 13 5 42 36 37 26 1 35 10 46 27 59 24 51 43 38
9 14 52 22 15 58 2 44 30 3 25 45 47 11 12 26 24
32 19 41 42 13 60 4 33 10 57 23 14 34 25 53 48 58
24 35 18 49 8 43 39 11 46 15 9 50 20 11 47 54 32
51 21 12 59 7 11 48 52 1 31 13 33 2 8 25 34 19
49 22 3 10 36 57 37 35 53 23 14 29 4 32 9 28 55

Message 9 score 15432

15 23 43 47 51 52 19 24 32 17 16 36 44 53 20 11 30
37 54 21 12 45 11 38 9 39 55 33 10 46 13 48 49 51
22 14 5 1 6 61 2 34 9 40 56 52 19 25 31 53 11
5 11 10 26 35 3 42 55 23 12 54 43 36 37 30 4 32
9 41 28 1 7 13 2 33 24 8 14 6 3 18 38 34 20
25 50 21 11 4 10 39 44 59 45 26 35 17 7 29 11 11
22 47 36 56 51 37 15 23 8 12 60 1 52 13 46 38 43
57 14 44 62 7 39 28 9 59 2 53 11 45 3 32 10 40
29 4 8 12 54 19 13 31 11 33 5 1 6 61 48 20 14
2 9 3 34 10 7 21 4 35 18 11 36 16 11 12 32 51
37 27 13 14 41 52 22 11 30 8 38 28 9 60 1 46 31
53 19 12 15 13 14 54 59 24 11 20 2 21 39 51 60 3
52 11 43 5 4 17 25 16 53 22 12 10 11 7 54 36 44
8 1 33 34 37 11 6 13 40 45 14 49 11 35 51 50 38
39 32 18 26 58 12 5 2 6 62 3 30 55 47 52 4 46
9 15 11 36 53 5 1 54 19 23 33 6 13 10 56 48 14
11 59 37 51 2 5 29 11 49 41 38 39 34 16 55 28 50
52 11 3 17 4 29 28 36 35 37 15 60 1 53 12 43 47
38 31 13 2 9 57 24 48 14 49 54 44 39 32 18 11 45
25 16 11 20 12 7 11 33 58 56 29 50 26 36 34 47 3

Message 56 score 15775

48 21 12 59 1 49 62 41 56 37 19 2 38 10 5 13 3
57 52 25 17 56 31 6 57 53 39 42 60 50 22 14 26 51
7 4 54 55 15 46 16 11 12 40 10 11 12 1 10 48 23
13 27 49 37 43 12 52 24 14 18 28 47 50 53 2 38 11
51 21 15 59 25 32 33 39 44 54 8 16 55 22 12 34 3
48 12 26 31 4 63 12 60 1 29 13 40 27 20 23 52 49
53 24 28 37 30 25 38 19 2 5 41 56 54 35 62 39 14
61 55 58 26 9 12 27 36 35 3 63 6 15 50 21 16 59
28 32 33 7 12 52 22 13 8 14 4 57 53 25 17 56 34
5 31 42 40 10 54 23 12 55 6 1 7 62 51 26 12 48
37 15 2 46 52 24 16 32 27 53 54 33 12 49 55 43 47
13 3 38 11 60 4 34 29 50 10 44 59 39 12 21 14 11
12 46 30 1 31 32 15 63 16 2 9 22 12 58 13 40 28
37 20 3 8 41 57 52 51 14 58 15 38 42 47 36 4 62
5 16 48 23 12 60 25 33 34 6 13 53 24 14 49 21 12
45 15 31 63 7 32 56 16 12 62 13 10 8 46 43 59 39
14 54 55 12 22 1 52 50 2 26 11 40 44 60 23 15 37
27 3 51 29 16 10 24 12 47 18 41 46 4 11 12 53 13
28 38 21 25 19 22 48 9 23 42 43 33 5 57 54 35 1
63 6 14 26 55 59 27 34 31 39 44 12 7 15 16 28 52

In fact, as I increased the randomization of my symbol selection, the scores were lower and it was more difficult to get 15034. You might want to check that.

I think that the bi-regional bias is what we should be looking at. There really is one and it does show up in the pictures that emphasize the 15034 score. The absence of some symbols in the middle third of the message, roughly, but presence of the same symbols in the top and bottom third, roughly, seems more important to me. There are a lot of them.

Here is the 340, with symbols that only appear in the top and bottom six rows.

unigram distance phenomenon 2.png

I made up a new score, which may in the future turn out to be trash. For each symbol, I find the average distance from the center of the message to the position(s). Thus, if symbol A appears at positions 37 and 244, the distances, rounded, are 134 and 73. The average is 103. Then, I add up the average distance from center for each symbol that appears exclusively in the top and bottom six rows. It's sort of arbitrary, but the picture in the first post shows symbols in the top and bottom six rows so I went with that.

A symbol that appears in row 6 and row 15 will have a lower average than a symbol that appears in rows 1 and 15, which would be lower than a symbol that appears in rows 1 and 20. It takes into account both quantity and distance.

The 340 has a score of 4710.

With perfect cycles, I can easily match the 15034 score, but not get even close to the 4710 score. With 25% randomization, which is what the 340 is, roughly over all, I still can't get close.

The green symbols are the ones that are exclusive to the middle eight rows. Maybe not score so much, but region is important. Please check.

Oh, by the way, thanks for giving the speech doranchak in advance. Maybe someone in the audience will be crazy enough to get involved with this. Try to generate some enthusiasm.

[Jarlve]

I encoded some messages from the library all perfect cycles but with one polyphone and got scores higher than 15034. It was pretty easy.

Yes, it is what Largo referred to as a long key in this thread, or a key where the frequency distribution is as flat as possible. Your messages have a lower ioc than the 340 which makes it easier to attain higher unigram distance values.

I think that the bi-regional bias is what we should be looking at. There really is one and it does show up in the pictures that emphasize the 15034 score. The absence of some symbols in the middle third of the message, roughly, but presence of the same symbols in the top and bottom third, roughly, seems more important to me. There are a lot of them.

I agree that it seems to be the more likely cause of the high unigram distance. What is the cause of a reasonably large group of symbols not appearing in the middle 3rd of the cipher?

Here is the 340, with symbols that only appear in the top and bottom six rows.

Thank you for the clear picture.