Cycle types

[Jarlve]

Here are the 3-symbol cycle ngrams. Thank you smokie for this wonderful idea, I hope you do not mind my take on it?

https://drive.google.com/open?id=1mANVn ... XAaqQb9nfS

EDIT: added missing ngrams

For the 340 one ngram stood out to me:

ABCABAB: 14.25
ABACABA: 12.53
ABCABCB: 12.49
ABCABAC: 12.13
ABCAABC: 11.63
ABCACAC: 11.39
BCABABC: 10.67
ABCBAAC: 10.46
ABCACBC: 10.39
ABCBAAB: 10.02

It looks like this for the 408:

ABCABCA: 37.43
BCABCAB: 31.12
ABCABCB: 28.62
CABCABC: 25.49
ABCABCC: 23.98
ABCACBA: 22.20
ABCABAB: 20.44
BCACBAC: 19.75
ABCABAC: 19.73
BCABCAC: 19.41

Questions. Why would ABCABAB almost be 2 sigma higher than all the other ngrams? ABCABCA is not even in the top 10 for the 340?

[doranchak]

Questions. Why would ABCABAB almost be 2 sigma higher than all the other ngrams? ABCABCA is not even in the top 10 for the 340?

I think it's because the real cycles, if they do exist, are perturbed by some step in the encipherment. The sigmas you got for Z408 are so much higher than the top ones for Z340. It makes me think that we are seeing a lot of noise in Z340's cycles. The cycle isomorphism test produces very many samples, so we do expect to see many outliers since we're getting many samples from the tail ends of the distribution.

Another point of evidence towards the "perturbed cycles" hypothesis might be the fact that in cycle shuffling experiments, I can easily increase the mean sigma of all normal cycles (ABAB, ABCABC, etc) by exchanging rows of Z340. The "best" cycles in the rearrangement don't look better, but the rearrangement produces many more improved cycles overall. Can you run your L=2 and L=3 shuffles on this and see how it compares to the unmodified Z340?

Code: Select all

HER>pl^VPk|1LTG2d
Np+B(#O%DWY.<*Kf)
2<clRJ|*5T4M.+&BF
z69Sy#+N|5FBc(;8R
lGFN^f524b.cV4t++
|FkdW<7tB_YOB*-Cc
>MDHNpkSzZO8A|K;+
(G2Jfj#O+_NYz+@L9
d<M+b+ZR2FBcyA64K
-zlUV+^J+Op7<FBy-
U+R/5tE|DYBpbTMKO
By:cM+UZGW()L#zHJ
Spp7^l8*V3pO++RK2
_9M+ztjd|5FP+&4k/
yBX1*:49CE>VUZ5-+
|c.3zBK(Op^.fMqG2
RcT+L16C<+FlWB|)L
++)WCzWcPOSHT/()p
p8R^FlO-*dCkF>2D(
#5+Kq%;2UcXGV.zL|

z340:
jarlve nonrepeat 1: 4462
jarlve nonrepeat 2: 1599
pcs2: 247.85
pcs3: 62.36
jarlve m_2s_cycles: 2150.72
mean l2 sigma: 0.21
z340_best_l2:
jarlve nonrepeat 1: 4282
jarlve nonrepeat 2: 1600
pcs2: 271.57
pcs3: 100.25
jarlve m_2s_cycles: 2259.03
mean l2 sigma: 0.43


Here is the illustration of the row swaps to produce the modified cipher: http://zodiackillerciphers.com/images/z ... ved-L2.jpg

It may be that the average sigma will increase, but the tail ends of the distribution will still look the same. I think that's what I'm seeing when looking at specific L=2 cycles.

This whole business with cycles is really maddening.

[smokie treats]

Thank you smokie for this wonderful idea, I hope you do not mind my take on it?
Questions. Why would ABCABAB almost be 2 sigma higher than all the other ngrams? ABCABCA is not even in the top 10 for the 340?

You are welcome, I don't mind, and I am rather enjoying watching you work and working on my own. But you are much faster at getting results. At this point I have no idea why ABCABAB is significant, but will continue my work over here. I am still working on my L = 2 spreadsheet. I am working on different options, and will soon start making some test messages with different encoding patterns, including 100% random symbol selection, to see what happens. We are definitely detecting something.

[Jarlve]

Questions. Why would ABCABAB almost be 2 sigma higher than all the other ngrams? ABCABCA is not even in the top 10 for the 340?

You will have to excuse me because some ngrams are missing from the 3-symbol lists. Such as ABC and ABA. I do not understand why since these ngrams are not missing from the randomizations and the sorting algorithm is working correctly.

Another point of evidence towards the "perturbed cycles" hypothesis might be the fact that in cycle shuffling experiments, I can easily increase the mean sigma of all normal cycles (ABAB, ABCABC, etc) by exchanging rows of Z340. The "best" cycles in the rearrangement don't look better, but the rearrangement produces many more improved cycles overall. Can you run your L=2 and L=3 shuffles on this and see how it compares to the unmodified Z340?

By the way, I have changed my n-symbol cycles routine to not include symbols that occur only once.

Normal 340:
- 2-symbol cycle score: 2136
- 3-symbol cycle score: 5922

Your 340:
- 2-symbol cycle score: 2253
- 3-symbol cycle score: 6692

First 340 characters of the 408:
- 2-symbol cycle score: 2856
- 3-symbol cycle score: 12048

The thing with cycles is that they are multiplicative, cycles start to cycle with eachother so to say. If you do not compensate for that in your measurement the score blows up very easily. The calculation of the measurement may not be to exponential. The cycles are not isolated outside of the cipher.

[doranchak]

Thanks Jarlve. So my modified 340 has slightly improved scores. Seems to be very easy to improve any kind of cycling score for Z340 with simple manipulations.

That's a good point about the multiplicative effect of cycle scores.

[smokie treats]

I got my L=2 cycle chunks spreadsheet done. I can compare all patterns regardless of length, or set an ngram length and the start position.

340

For ngram length of 6 and start position 1:

158 ABAABA
120 ABABAA
107 ABABAB
106 AABAAA
99 ABABBB
94 ABAAAB
93 ABBAAB
89 ABAAAA
87 ABBABB
87 AABAAB

If I try ngram length of 3 and start position 7:

214 ABA
183 AAB
181 BAB
161 BAA
157 AAA
144 BBB
141 BBA
100 ABB

EDIT: I did about 30 shuffles, and ABA was always somewhere on the list, but never on the top. AAA most often on the top.

This is still consistent with the ABAABA pattern, but if I go with length 9 start position 1, ABAABAABA is way down there:

25 AABAAAABA
24 ABAABABBA
22 ABABAAAAB
20 AABAABAAB
19 ABBABBBBB
19 AABAAABAA
17 ABAABAAAB
17 ABAABABAB
16 ABBABBBBA
16 ABBAABBBB
15 ABABBBBBA
15 ABAABAABA

If I go with length 4 and start position 7, it is still consistent with ABAABA:

89 ABAA
86 AABA
85 ABAB
84 BABA
80 AAAA
70 BBBB
69 BAAB
64 BBBA
64 BAAA
61 BABB

If I go with length 5 and start position 7, the pattern is down on the list:

49 AAAAA
43 ABAAA
42 ABABA
42 AABAA
38 ABAAB
37 BBBBB
32 BAAAA
32 BABBB
31 AAABA
31 BABAB

[Jarlve]

I fixed the bug with the missing ngrams, all results can be found here: https://drive.google.com/open?id=1mANVn ... XAaqQb9nfS

Why are our counts so different smokie? I am excluding cycles with symbols that occur only once.

AAAAAA: 398
ABAABA: 381
ABAAAA: 377
AABAAA: 374
AAABAA: 325
ABABAB: 316
ABABAA: 307
AAAABA: 292
ABAAAB: 279
AABAAB: 277

[Jarlve]

Thanks Jarlve. So my modified 340 has slightly improved scores. Seems to be very easy to improve any kind of cycling score for Z340 with simple manipulations.

It must mainly depend on the initial randomness of the cycles in the cipher. It will be more easy to improve the state of the cipher by manipulations if its cycles are more random.

Could you test this cipher (randomized plaintext) versus your manipulations and how it performs versus the 340?

Code: Select all

+E7'D*!$3*.)HSF1$
M&^*%Y6[ZU=($VQ,*
$R+>)*'/KG$(*8V$B
ITC#X3>E14#]+H\LX
-3*=$)*@II^1HQ$[>
!YP^<R\U:T,<?$IO;
#QK*5A"$9I0&CE*^.
+0D]M/JV$!K'*QR46
)-4Z"MI$8H1?9'?E&
7^A?(!-B)\V?Z+55A
*Q_M9KX+G_/$2-4%E
]R!;SFL)_<*8"1I3&
#N=#HL';Y$O0Z*F@[
S!X.V"':$*-^V\:DK
3UF,*E!8AB"TQ$7K>
[4=*["^\H#$XME*T3
C49+[]PG>-;%5#X*\
&N)T/RE1@?N3>2S\I
Q#]P$P'[JU*;O*:1@
@K$*#$U:?RW2,Y*9E


[smokie treats]

I fixed the bug with the missing ngrams, all results can be found here: https://drive.google.com/open?id=1mANVn ... XAaqQb9nfS

Why are our counts so different smokie? I am excluding cycles with symbols that occur only once.

AAAAAA: 398
ABAABA: 381
ABAAAA: 377
AABAAA: 374
AAABAA: 325
ABABAB: 316
ABABAA: 307
AAAABA: 292
ABAAAB: 279
AABAAB: 277

I don't know. I am going like this for my symbols, with only 1,953 possible combinations ( I am using numbers for symbols ):

1 2
1 3
1 4
1 5
1 6
1 7
1 8
1 9
1 10
1 11
1 12
1 13
1 14
1 15
1 16
1 17
1 18
1 19
1 20
1 21
1 22
1 23
1 24
1 25
1 26
1 27
1 28
1 29
1 30
1 31
1 32
1 33
1 34
1 35
1 36
1 37
1 38
1 39
1 40
1 41
1 42
1 43
1 44
1 45
1 46
1 47
1 48
1 49
1 50
1 51
1 52
1 53
1 54
1 55
1 56
1 57
1 58
1 59
1 60
1 61
1 62
1 63
2 3
2 4
2 5
2 6
2 7

Etc. In other words, if I have two symbols, 1 and 2, I don't repeat with 2 and 1. I just start with 2 and 3. I could have a problem, and will look at in the morning if you don't know.

[smokie treats]

I fixed the bug with the missing ngrams, all results can be found here: https://drive.google.com/open?id=1mANVn ... XAaqQb9nfS

Why are our counts so different smokie? I am excluding cycles with symbols that occur only once.

AAAAAA: 398
ABAABA: 381
ABAAAA: 377
AABAAA: 374
AAABAA: 325
ABABAB: 316
ABABAA: 307
AAAABA: 292
ABAAAB: 279
AABAAB: 277

I am not sliding through my patterns, only taking chunks like the first 6, etc. Maybe that is why.