Cycle types

[smokie treats]

And here is the 408.

ABABAB: 777
BABABA: 643
ABAABA: 495
ABABBA: 491
AABABA: 461
ABABAA: 455
ABBABA: 415
BABBAB: 389
BABAAB: 374
BABABB: 352

Interesting that consecutive alternations shows up at spot 1 and 2, which is exactly what he did. The test worked here.

Okay,

Here are the 2-symbol cycle ngram frequencies from 2 to 7 for the 340. I could have posted more but prefer to keep things manageable and investigate the ABAABA repeats. Will now work to get the sigma of each repeat versus randomizations.

AAAAAA: 398
ABAABA: 381
ABAAAA: 377
AABAAA: 374
AAABAA: 325
ABABAB: 316
ABABAA: 307
AAAABA: 292
ABAAAB: 279
AABAAB: 277

Here the top spot only has one symbol, so ABAABA actually comes out on top. Interesting that there are no BABBAB near the top. You guys are really tearing this up, hope you are having fun.

[Jarlve]

Will now work to get the sigma of each repeat versus randomizations.

Here they are for the 340.

To clarify, the ngrams are compared to 5000 full cipher randomizations. And as smokie mentioned, A and B are determined by order of appearance. And a positive sigma means that the ngram is observed more versus randomizations and a negative sigma means it is observed less.

ABABAB: 7.22
ABAABA: 7.01
ABABBB: 4.34
ABABAA: 4.17
AAABBA: -3.95
BABABA: 3.45
ABABBA: 3.34
ABAAAB: 3.33
BBABAA: -3.29
AAAABB: -3.25

Code: Select all

AZdecrypt cycle ngram stats for: 340.txt
--------------------------------------------------------

2-symbol cycles, 3-grams, sigma:
--------------------------------------------------------
ABA: 6.79
AAA: -5.93
BAB: 3.19
BBA: -2.19
BAA: -1.51
ABB: 1.07
BBB: -0.52
AAB: -0.41

2-symbol cycles, 4-grams, sigma:
--------------------------------------------------------
ABAB: 6.78
ABAA: 4.62
AAAA: -4.08
AAAB: -3.76
AABB: -3.53
BAAA: -3.11
BBAA: -2.82
BABA: 2.52
AABA: 2.27
BAAB: 1.40
ABBA: 1.37
BABB: 1.36
ABBB: 1.33
BBBA: -1.03
BBAB: 0.98
BBBB: 0.26

2-symbol cycles, 5-grams, sigma:
--------------------------------------------------------
ABABA: 6.23
ABAAB: 5.77
AAABB: -4.47
ABABB: 4.19
BABAB: 3.94
BBAAA: -3.59
AAAAA: -3.51
ABBAB: 3.07
AABAA: 2.98
AABBA: -2.70
ABAAA: 2.50
BABBB: 2.47
AAAAB: -2.44
BBBAA: -2.20
AABBB: -1.93
BAABA: 1.86
BAAAB: -1.65
BAAAA: -1.63
ABBBB: 1.56
BAABB: -1.39
BBBAB: 1.04
ABBBA: 0.79
BABAA: -0.75
BBAAB: -0.65
BBBBB: 0.64
BBABB: 0.51
AAABA: -0.45
BABBA: 0.41
AABAB: 0.15
BBABA: 0.11
ABBAA: 0.03
BBBBA: -0

2-symbol cycles, 6-grams, sigma:
--------------------------------------------------------
ABABAB: 7.22
ABAABA: 7.01
ABABBB: 4.34
ABABAA: 4.17
AAABBA: -3.95
BABABA: 3.45
ABABBA: 3.34
ABAAAB: 3.33
BBABAA: -3.29
AAAABB: -3.25
BBAAAB: -3.25
ABBBAB: 3.12
AABAAB: 3.04
ABBAAB: 2.84
AAAAAA: -2.84
BAAABB: -2.78
BAABAB: 2.73
ABBABB: 2.72
BBBAAA: -2.68
AAABBB: -2.43
AAAAAB: -2.31
ABAAAA: 2.27
BBAAAA: -2.26
AABAAA: 2.20
AAABAB: -2.19
BAAAAA: -2.14
ABBABA: 2.02
BBABAB: 2
BBAABA: -1.93
BBBAAB: -1.91
BABBAB: 1.90
AABBAB: -1.77
AABBAA: -1.75
BABBAA: -1.74
BABAAA: -1.74
BABBBB: 1.72
BBABBB: 1.56
BABBBA: 1.53
AABABA: 1.52
BABABB: 1.51
AABBBA: -1.47
ABBAAA: -1.46
AABABB: -1.40
ABBBAA: -1.37
ABBBBB: 1.23
ABBBBA: 1.20
BAABBB: -1.17
BAABBA: -1.07
BABAAB: 1.07
BBBBBB: 0.94
AABBBB: -0.91
BAABAA: 0.75
BBBBAA: -0.74
AAABAA: 0.67
BBBABA: 0.67
BBBBAB: 0.64
BBBBBA: 0.46
AAAABA: -0.35
ABAABB: -0.31
BBABBA: -0.17
BBBABB: 0.13
BBAABB: 0.09
BAAABA: 0.07
BAAAAB: 0.04

2-symbol cycles, 7-grams, sigma:
--------------------------------------------------------
ABAABAB: 6.98
ABABABA: 6.63
ABAABAA: 5.57
ABAAABA: 5.32
ABABAAB: 5.15
BABABAB: 4.71
ABBABBB: 4.30
ABABABB: 4.22
ABABBAB: 3.72
ABAAAAB: 3.70
ABABBBA: 3.60
ABBABAB: 3.32
BBABAAA: -3.26
AAAABBA: -3.17
AAABBAA: -3.09
AABAABA: 3.07
BBAAABA: -3.06
BBBAAAB: -3
BABBBAB: 2.89
AABBABB: -2.86
ABABBBB: 2.77
BABBAAA: -2.76
BBBAABA: -2.70
BBBABAA: -2.69
BBABBAA: -2.68
ABBAABB: 2.65
AAABBAB: -2.63
AAAAABB: -2.60
AABABAB: 2.47
AAAAAAA: -2.45
AABAAAB: 2.45
BAABABA: 2.38
BBAABAA: -2.36
ABABAAA: 2.34
BAAAABB: -2.33
BAAABBA: -2.32
AABAAAA: 2.32
BBAAAAA: -2.32
BAABAAB: 2.19
ABBBABB: 2.16
ABBBABA: 2.15
ABBBAAA: -2.14
BBBABAB: 2.09
AAABABB: -2.03
AAABBBA: -2
AAAABAB: -2
BABBABB: 1.86
AAAAAAB: -1.86
AABBAAA: -1.79
BBAAABB: -1.77
AAABBBB: -1.74
ABABBAA: 1.70
BABBBBB: 1.63
BAAABBB: -1.62
AAAABBB: -1.61
BBBAAAA: -1.58
BABABBB: 1.55
BAAAAAA: -1.53
BBABBAB: 1.50
BBABAAB: -1.50
ABBBBAB: 1.47
BBBBAAA: -1.43
ABBAABA: 1.41
ABBBBBB: 1.40
BBAAAAB: -1.36
AABBBAB: -1.31
BBABBBB: 1.31
BABABBA: 1.29
ABAAABB: -1.27
BABAAAB: -1.27
AABBBBB: -1.25
BBBBBBB: 1.24
BBBBAAB: -1.23
BBABABA: 1.21
AAABAAA: 1.17
BABAABA: 1.16
AABABAA: 1.12
BAABBAA: -1.11
BAAAABA: 1.06
ABBBBBA: 1.05
BBBBBAB: 1.05
BABAABB: -1.04
ABBABAA: -1.03
AABBBAA: -1
BAABBBA: -1
BBBBBBA: 0.98
BABAAAA: -0.97
BABBBAA: -0.96
BAABABB: 0.95
BBABBBA: 0.91
BAAAAAB: -0.86
ABBAAAB: -0.86
BBBAABB: -0.85
BBBBABA: 0.85
AAABABA: -0.85
ABAAAAA: 0.83
ABBABBA: 0.82
ABBAAAA: -0.82
AAAABAA: 0.79
BABBBBA: 0.74
AABABBB: -0.73
BABABAA: 0.73
BBBABBA: 0.66
BAABAAA: -0.64
BABBABA: 0.59
AABBAAB: -0.58
BABBAAB: 0.51
BBAABAB: -0.49
AABAABB: -0.44
BBBBABB: -0.41
AAAAABA: -0.39
AABABBA: -0.38
AABBABA: 0.37
ABAABBB: -0.30
ABBBAAB: -0.29
BAAABAB: -0.25
ABBBBAA: 0.25
BBABABB: 0.22
BBBBBAA: -0.19
AABBBBA: -0.18
BBBABBB: -0.16
ABAABBA: 0.14
AAABAAB: 0.11
BBAABBB: -0.11
BAAABAA: -0.10
BBAABBA: -0.09
BAABBBB: -0.06
BAABBAB: -0.06


[Jarlve]

And for the 408.

ABABAB: 15.93
BABABA: 11.97
ABABBA: 9.94
BABBAB: 8.94
BABABB: 7.89
ABBABA: 7
ABAABA: 6.65
AABABA: 6.19
ABABAA: 5.93
BABAAB: 5.18

Code: Select all

AZdecrypt cycle ngram stats for: 408.txt
--------------------------------------------------------

2-symbol cycles, 3-grams, sigma:
--------------------------------------------------------
ABA: 9.89
BAB: 9.51
AAA: -6.91
BAA: -4.77
BBB: -4.01
BBA: -3.04
AAB: -2.94
ABB: 0.60

2-symbol cycles, 4-grams, sigma:
--------------------------------------------------------
ABAB: 12.76
BABA: 9.30
BAAA: -6.56
AAAB: -6.02
BABB: 5.48
BBAA: -5.38
AAAA: -4.79
BBBA: -4.44
AABB: -3.87
ABAA: 3.17
ABBB: -3.07
ABBA: 2.89
BBAB: 2.82
BBBB: -2.54
AABA: 1.75
BAAB: 0.32

2-symbol cycles, 5-grams, sigma:
--------------------------------------------------------
ABABA: 13.04
BABAB: 11.77
ABABB: 8.23
BABBA: 6.70
ABBAB: 6.07
BBAAA: -5.28
ABAAB: 5.18
BAAAA: -5.13
AAAAB: -5
BBBAA: -4.93
AABAB: 4.61
AAABB: -4.04
AABBB: -3.82
AAAAA: -3.55
BBBBA: -3.27
BBAAB: -3.12
BBABA: 3.07
BABAA: 2.92
BAAAB: -2.70
ABBBB: -2.59
AAABA: -2.42
BAABA: 2.15
AABBA: -2.11
BBABB: 1.54
ABBAA: -1.51
BAABB: -1.50
BBBBB: -1.44
ABBBA: -1.39
AABAA: -0.89
BBBAB: -0.60
ABAAA: -0.45
BABBB: 0.07

2-symbol cycles, 6-grams, sigma:
--------------------------------------------------------
ABABAB: 15.93
BABABA: 11.97
ABABBA: 9.94
BABBAB: 8.94
BABABB: 7.89
ABBABA: 7
ABAABA: 6.65
AABABA: 6.19
ABABAA: 5.93
BABAAB: 5.18
BBABAB: 4.65
BBAAAA: -4.44
BBBAAB: -4.30
BAAAAB: -3.92
BBBAAA: -3.83
BAABAB: 3.80
AAAABB: -3.71
BBAAAB: -3.61
AAAAAB: -3.59
BBBBAA: -3.49
BAAAAA: -3.37
AAABBB: -3.25
ABBAAA: -3.17
ABBBAA: -3.13
AAAABA: -2.85
AABBBB: -2.81
ABBABB: 2.74
AABBBA: -2.68
AAABBA: -2.67
AAAAAA: -2.67
ABBBBA: -2.66
BBAABA: -2.66
AABBAA: -2.57
BAABBB: -2.50
AAABAA: -2.33
BAAABB: -2.27
BBAABB: -2.15
BBABBA: 1.89
BABBBA: 1.80
ABBBAB: 1.74
BBBBBA: -1.72
ABAAAA: -1.62
BABBBB: -1.54
ABAAAB: 1.44
BAAABA: -1.44
BBBABA: -1.37
ABAABB: 1.36
BBBBAB: -1.24
ABBBBB: -1.21
AABBAB: -1.14
ABABBB: 1.04
BABAAA: -1.02
BBBABB: 0.94
AABAAA: -0.81
BBBBBB: -0.80
BAABBA: -0.75
AAABAB: -0.65
BABBAA: 0.48
BBABBB: 0.42
AABABB: 0.37
AABAAB: -0.20
BBABAA: -0.19
BAABAA: 0.18
ABBAAB: 0.06

2-symbol cycles, 7-grams, sigma:
--------------------------------------------------------
ABABABA: 18.01
BABABAB: 15.54
ABABBAB: 11.69
ABABABB: 9.93
BABABBA: 9.14
BABBABA: 8.96
ABABAAB: 8.69
ABAABAB: 7.79
ABBABAB: 7.69
BABAABA: 6.54
BABBABB: 5.84
AABABAB: 5.49
BABABAA: 5.31
BAABABA: 5.28
BBABBAB: 4.06
BBBAABA: -4.05
BBAAAAB: -3.91
BBAAABA: -3.69
ABAABAA: 3.64
BBABABB: 3.62
BBBBAAB: -3.36
ABABBAA: 3.32
BABBBAB: 3.20
BBBAAAA: -3.18
AAAABBB: -3.10
ABBAAAA: -3.08
AABBBAA: -3.06
BAAAABA: -3.05
AAAAABB: -3.05
BBBAAAB: -2.97
AABBAAA: -2.97
ABABBBA: 2.96
ABBBBAA: -2.94
BBAAAAA: -2.93
ABBABAA: 2.93
ABBBAAA: -2.81
ABBABBA: 2.78
BBBBAAA: -2.77
BAAAAAB: -2.76
AAABBAB: -2.69
AABABAA: 2.65
AAABBBA: -2.64
AAAAAAB: -2.62
BBABABA: 2.61
BAAAAAA: -2.60
ABAAABA: 2.58
AAAABAA: -2.45
BBAABAA: -2.39
ABBBABB: 2.30
BBBAABB: -2.30
BABBAAB: 2.29
AAAABBA: -2.28
AAABBBB: -2.28
ABBBAAB: -2.27
AABBBBA: -2.25
AAAAAAA: -2.25
BBAABBB: -2.21
AABBAAB: -2.21
AAAAABA: -2.20
AAABBAA: -2.19
ABAABBA: 2.18
BABBBBA: -2.17
BAAAABB: -2.13
BAAABAA: -2.13
BAABBBB: -2.13
BBBBBAA: -2.13
BAAABBA: -2.03
BBAABBA: -2.03
BBAAABB: -2
BBBABAA: -1.98
BABAAAA: -1.96
BAAABBB: -1.83
AABBBBB: -1.79
AAABABB: -1.78
AABBABB: -1.75
ABBAAAB: -1.70
BAABBAA: -1.70
AAABAAA: -1.69
BAABBBA: -1.68
BABABBB: 1.68
AAABAAB: -1.64
BBBBABA: -1.60
AABABBA: 1.48
ABAAAAA: -1.42
ABBABBB: 1.42
AAAABAB: -1.39
ABABBBB: -1.38
BABBAAA: -1.32
BBABBAA: -1.30
BBBABBA: 1.29
ABBBBBA: -1.26
BABAABB: 1.23
AABAABA: 1.18
AABAAAA: -1.11
BBBBBBA: -1.06
BBABBBA: 0.97
AABAABB: -0.93
BBAABAB: -0.88
BABBBAA: -0.86
AABABBB: -0.86
AAABABA: 0.80
ABBAABA: 0.79
AABBBAB: -0.77
BABAAAB: 0.76
BAABABB: 0.71
ABBBABA: 0.65
ABAAAAB: -0.60
ABAABBB: -0.60
ABBBBAB: -0.59
AABBABA: 0.59
ABBAABB: -0.58
BBBBBBB: -0.58
BBBBBAB: -0.57
BABBBBB: -0.56
BBABAAB: 0.55
BBABAAA: -0.50
BAABBAB: 0.49
BAABAAB: 0.47
ABBBBBB: -0.45
ABAAABB: -0.41
ABABAAA: 0.38
BAAABAB: 0.26
AABAAAB: 0.25
BBBABAB: 0.24
BBBABBB: 0.17
BBBBABB: 0.16
BBABBBB: -0.16
BAABAAA: 0.11


[Jarlve]

One thing that I realized this morning. I assigned A or B depending on the symbol's order of appearance. But ABAABA for 121121 is the same thing as BABBAB for 323323.

I do this to. Would it be a problem? Perhaps each cycle string could be inverted also.

[smokie treats]

One thing that I realized this morning. I assigned A or B depending on the symbol's order of appearance. But ABAABA for 121121 is the same thing as BABBAB for 323323.

I do this to. Would it be a problem? Perhaps each cycle string could be inverted also.

Simple example.

Let's say the message has two of ABAABA and only one of BABBAB. You would have to flip both of them, then add them all up, and then remove duplicate stats.

ABAABA
ABAABA
BABBAB

Then flip.

BABBAB
BABBAB
ABAABA

Add them all up.

ABAABA 3
BABBAB 3

Since the numbers equal each other, and one is the flip of the other, then just remove one.

ABAABA 3

[doranchak]

I kicked off an L=3 shuffle test for isomorphic sequences, limited to those of lengths between 6 and 15. It will take a few hours. My test has a variation which is to consider all substrings of a given sequence. For example, if it finds a sequence AAAAAAAAAABCABCAAAAAAAAAA, it will consider sequences AAAAAA as well as ABCABC, in addition to all other substrings of lengths between 6 and 15. I did that because I figured Z408's cycles have a lot of imperfections.

I think we are still faced with the problem of separating true cycles from false ones. I'm wondering what other input is needed from the ciphertext stats which could help make the distinction, if any such input actually exists. Perhaps it is a problem that could be solved with machine learning (i.e., give an algorithm many examples of true and false cycles to train with). Such algorithms are capable of deriving the inputs needed to make the decision.

[Jarlve]

Let's say the message has two of ABAABA and only one of BABBAB. You would have to flip both of them, then add them all up, and then remove duplicate stats.

Need to think about it.

I think we are still faced with the problem of separating true cycles from false ones. I'm wondering what other input is needed from the ciphertext stats which could help make the distinction, if any such input actually exists. Perhaps it is a problem that could be solved with machine learning (i.e., give an algorithm many examples of true and false cycles to train with). Such algorithms are capable of deriving the inputs needed to make the decision.

One thing I noted is that you either say a cycle is true or false while it actually is more fuzzy. A false cycle by your definition could have 3 out of 4 symbols correct. I wonder how many cycles that you have at the top of your list that are completely false. Though I also recognize this problem and my solution was to come up with a cycle merging hill climber (the one in AZdecrypt). True cycles will more likely be part of the best fit total merge. Other clues could be derived from the properties of sequential homophonic substitution. And we could be looking for the wrong cycle type.

[Jarlve]

Talking about cycle types, I have done allot of work on it the last week. It seems that the 340 just as the 408 is suffering from increasingly random cycles. My routine has detection for random, offset, palindromic, shortened, lengthened, anti and pattern cycles and these do not look interesting. This detection works on all the test ciphers in the main post. No detection yet for regional cycles. It is still a question for me where the extra randomness in the 340 comes from but could it be as simple that the 340 just starts out more randomly?

340:

2-symbol cycles:
Cycles: 6.37 sigma
Increasingly random cycles: 2.17 sigma
Decreasingly random cycles: -1.36 sigma

3-symbol cycles:
Cycles: 6.28 sigma
Increasingly random cycles: 4.40 sigma
Decreasingly random cycles: 0.02 sigma

4-symbol cycles:
Cycles: 7.03 sigma
Increasingly random cycles: 6.21 sigma
Decreasingly random cycles: 0.77 sigma

5-symbol cycles:
Cycles: 7.43 sigma
Increasingly random cycles: 6.95 sigma
Decreasingly random cycles: 1.13 sigma

408:

2-symbol cycles:
Cycles: 9.74 sigma
Increasingly random cycles: 4.13 sigma
Decreasingly random cycles: -1.79 sigma

3-symbol cycles:
Cycles: 13.30 sigma
Increasingly random cycles: 9.40 sigma
Decreasingly random cycles: -0.44 sigma

4-symbol cycles:
Cycles: 18.98 sigma
Increasingly random cycles: 16.74 sigma
Decreasingly random cycles: 0.59 sigma

5-symbol cycles:
Cycles: 26.70 sigma
Increasingly random cycles: 27.12 sigma
Decreasingly random cycles: 1.33 sigma

Example of a increasingly random 5-symbol cycle in the 340: MUJ_9MUJ_9MUJUMJM99UM_M

[Largo]

Thank you for your detailed explanations, smokie and Jarlve! Maybe I also have some ideas and can contribute something as soon as I find some time.

[doranchak]

One thing I noted is that you either say a cycle is true or false while it actually is more fuzzy. A false cycle by your definition could have 3 out of 4 symbols correct.

That's a great point. I should alter my measurement to permit partial credit. It does seem like many of the "false" cycles are simply multiple true cycles that have gotten mixed together.