Route Transposition and Phenomenon

[Chaucer]

Here's a real life example of "columnar transposition with nulls":

http://scienceblogs.de/klausis-krypto-k ... odebreaker

Part of the story there is Jim Gillogly used a hillclimber to try to crack a columnar transposition but it was still messed up,
until he recognized that two columns of meaningless vowels had been added to the message.

Have you guys ruled out that possibility already?

The article further describes how an Australian codebreaker discovered that the remaining unsolved cipher couldn't be solved
until it was recognized that a cipher text letter was missing, throwing off the columnar transcription.

Very cool how he managed to figure that out!

I know nothing about codes, but I am endlessly fascinated by the work you guys do. Tremendous stuff.

I have long believed that Zodiac was merely an amateur cryptologist because of the rather easy substitution cipher he used in the 408 and the numerous mistakes he made. As a result, I think the 340 will never be solved because I think he tried to make it harder and ended screwing it up. It would be cool to see if anyone can identify the critical mistake he might have made in order to solve the code.

[smokie treats]

I have a new matrix that makes as many pivots as possible. I repeat little sections of a message that is 205 letters long like this:

pivot matrix.png

I got 35 pivots in the matrix, all going in the same direction like in the 340. Now I can diffuse 1000 messages with as many symbols as I want, and use different randomization patterns and then see how many pivots are left over.

So far, it looks like perfect cycles greatly reduces pivots, and the more random the more pivots. A 15% / 35% with 63 symbols including 1 polyphone with count as close to 24 as possible and the sum of the squares of the differences between 340 frequencies and message frequencies is minimized, allows for about 1.7 pivots per message, but I have to do more.

I really like the idea of non message letters written out in some kind of pattern, typewriter art, or a short message written over and over again in a pattern.

176 052 053 055 016 017 018 166 021 023 025 001 002 003 163 006 008
052 177 054 056 058 019 020 021 167 024 026 028 004 005 006 164 009
053 054 178 057 059 061 022 023 024 168 027 029 031 007 008 009 165
055 056 057 179 060 062 064 025 026 027 169 030 032 034 010 011 012
103 058 059 060 180 063 065 067 028 029 030 170 033 035 037 013 014
104 106 061 062 063 181 066 068 070 031 032 033 171 036 038 040 015
105 107 109 064 065 066 182 069 071 073 034 035 036 172 039 041 043
193 108 110 112 067 068 069 183 072 074 076 037 038 039 173 042 044
108 194 111 113 115 070 071 072 184 075 077 079 040 041 042 174 045
110 111 195 114 116 118 073 074 075 185 078 080 082 043 044 045 175
112 113 114 196 117 119 121 076 077 078 186 081 083 085 046 047 048
142 115 116 117 197 120 122 124 079 080 081 187 084 086 088 049 050
143 145 118 119 120 198 123 125 127 082 083 084 188 087 089 091 051
144 146 148 121 122 123 199 126 128 130 085 086 087 189 090 092 094
206 147 149 151 124 125 126 200 129 131 133 088 089 090 190 093 095
147 207 150 152 154 127 128 129 201 132 134 136 091 092 093 191 096
149 150 208 153 155 157 130 131 132 202 135 137 139 094 095 096 192
151 152 153 209 156 158 160 133 134 135 203 138 140 142 097 098 099
160 154 155 156 210 159 161 163 136 137 138 204 141 143 145 100 101
161 162 157 158 159 211 162 164 165 139 140 141 205 144 146 147 102

[Largo]

Your matrix is cool, smokie. I just did three series of tests with it. Here are the results:

The first 340 characters of z408 transposed with your matrix and substituted with 25% random cycles, 100000 test ciphers:

Number of ciphers with bigram peak on P19: 120
Number of ciphers with exactly 2 pivots (without direction): 18398
Number of ciphers with P19 peak AND 2 pivots: 3

One of these three ciphers had two pivots in the same direction:

Code: Select all

adAhY:x3bSycuCPjU
DLj:590ePohcj+jAu
7jwURliv1sNKXCuUb
5u:aEhkxcLCo;eCAJ
EhRJnuE;hoKKSFrYx
y=lCuwZ+7X;v0IRs=
GSdBEDp5NPbpIQhc;
XgR;;aCiFIVrq5xZ=
3rdZX7L2R=tMwciqn
qZt:Iu0IXEPs0;=V9
;dUGDuXnGxg9bSaIr
N=IiXLjuow4sZ1;AN
p;U:Le5:=xedw3DCF
yXw=j5jK;wvM+hRo2
LiwVU:1K3pA;dq7sP
iF2cu=;g=CJ5hMNE;
LpDUcJwF5IEQo2a;T
t5:EqxYREJI3;sXiw
Yu5qc1y0CQ0lPp;HZ
gnEaKGV7bM;xcx=Zq


The first 170 letters of Z408, two times in a row, transposed with your matrix and substituted with 25% random cycles, 100000 test ciphers

Number of ciphers with bigram peak on P19: 0
Number of ciphers with exactly 2 pivots (without direction): 25668
Number of ciphers with P19 peak AND 2 pivots: 0

Continuous repetition of the phrase "THISISTHEZODIACSPEAKING" transposed with your matrix and substituted with 25% random cycles. Key contains only symbols for T, H, I, S, E, Z, O, D, A, P, C, K, N, G. 100000 test ciphers

Number of ciphers with bigram peak on P19: 0
Number of ciphers with exactly 2 pivots (without direction): 2187
Number of ciphers with P19 peak AND 2 pivots: 0

Your matrix generates a lot of P18 bigram peaks. I wonder how it could be rebuilt to produce P19 peaks.

[smokie treats]

I encoded 1000 messages from American Short Stories each starting with 35 pivots totalling 35,000 pivots. After 63 diffusion, here is how many pivots were left:

987 with perfect cycles ( 0.99 ) pivots per message
1544 with 15% / 35% randomization top / bottom ( 1.54 pivots per message )
2544 with 100% randomization ( 2.54 pivots per message )

More randomization = more pivots. Perfect cycles kill pivots because the cycles must be perfectly complete to make pivots.

[smokie treats]

Thanks for your work, Largo. The matrix wasn't intended to make P19, but perhaps future ones will. I would like to think that there is still a plaintext in the 340, and think that the idea of a shorter plaintext written over and over again in some kind of pattern or path, a creative inscription, is plausible. it would explain a lot.

Perhaps something simple. I will try to think of something simple that explains what we see. Maybe the same plaintext over and over again diagonally so that the same plaintext letters keep lining up with each other vertically and horizontally, and in P19 and P39. The length of the plaintext would be important, and I wonder about experimenting with different plaintext lengths to achieve the phenomenon.

[smokie treats]

Jarlve ~

If there was a plaintext written LRTB in a homophonic 63 cryptogram, and it was shorter than 340 but written over and over again, how would that affect the performance of your solver? Is there a message shortness or longness that makes a difference between solve ability or not? I am not saying this is what I think the Z340 is, but just wondering about length, repetition, and solve ability. If the plaintext length was say, 34 and over and over again, could it solve? What about 68, or 136, etc.?

[smokie treats]

If the same plaintext was written over and over again in a pattern, then there should be Kasiski spikes if we read the message in the right direction, or rearranged the message somehow and read the message in the right direction, or read in the right path. Has anyone tried this?

[doranchak]

My guess is that Jarlve's solver will punish plaintexts that are overly repetitive, due to them having higher IoC than "normal" messages.

[Jarlve]

Jarlve ~

If there was a plaintext written LRTB in a homophonic 63 cryptogram, and it was shorter than 340 but written over and over again, how would that affect the performance of your solver? Is there a message shortness or longness that makes a difference between solve ability or not? I am not saying this is what I think the Z340 is, but just wondering about length, repetition, and solve ability. If the plaintext length was say, 34 and over and over again, could it solve? What about 68, or 136, etc.?

Imagine a plain text with a length of 17 with a multiplicity of 0.5. It solves into gibberish, now, will repeating the plain text over and over again make it easier to solve? No, because the solver also scores the same n-grams over and again. So, more or less, the difficulty is equal to the multiplicity of the single plain text string. But if the multiplicity of the resulting cipher is higher than the multiplicity of the single plain text string, then the difficulty is equal to the multiplicity of the cipher instead.

My guess is that length 34 or greater will usually solve. Something to take in mind is that any cipher with a short string repeated over and over again will have a boatload of repeats. And as you say Kasiski and other Vigenère tests will have it show up.

[Jarlve]

My guess is that Jarlve's solver will punish plaintexts that are overly repetitive, due to them having higher IoC than "normal" messages.

SInce 1.15 AZdecrypt internally uses entropy instead of IOC.

Code: Select all

thisisthezodiacspeaking


IOC: 0.05138339920948617
Entropy: 3.621175542919471 <---

Code: Select all

thisisthezodiacspeaking
thisisthezodiacspeaking
thisisthezodiacspeaking
thisisthezodiacspeaking
thisisthezodiacspeaking


IOC: 0.08466819221967964
Entropy: 3.621175542919471 <---

Entropy remains the same. If the frequency distribution is "narrow" the entropy will be lower and that is now punished, but not overly so.