Page 2 of 7

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Tue Jul 31, 2018 9:45 am
by Boo!
I long thought that the strange symbol he does in the Halloween card could be two keys, they kind of look like keys of two different types to me. There is also that photo that was sent in years later of two keys, though I am not sure that is a confirmed Zodiac correspondence?

Then there are four dots, perhaps suggesting that the 340 is in four parts of 5 lines each, only jumbled up in the current wrong order. The first cipher he did in 3 parts so it's possible he had a similar line of thinking.

So to make the 340 harder than the 408 he -

Uses more symbols
Perhaps uses a key(s) somehow to change cipher encryption type
Splits the message into four parts instead of 3, and arranges them on paper out of order

I could see him doing something relatively simple like this to make the cipher harder to crack than the last one, and evolve his method. Picking up ideas out of his 'code breaking for noobs' book he got from the library.

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Tue Jul 31, 2018 10:20 am
by Largo
Jarlve wrote:As far as cycle examination goes I was able to identify the use of most cycle types in the thread:


It's hard to find an idea you or smokie didn't have before ;)
I can still think of two things:

- Have you ever examined the cycles separately for the upper and lower half? The MUJ_9MUJ_9MUJUMJM99UM_M example looks as if it were interrupted approximately in the middle. Some time ago I experimented with the perfect-cycle feature of AZDecrypt, but I had the feeling that 170 characters (one half of the cipher) are simply not enough analysis material.

- Would it make sense to use cycles instead of ngram scores to evaluate your transposition solver? So that it' looking for transpositions that result in better cycles? In this case, however, only minimally invasive transpositions should be used, i.e. simple region-based flips and mirrors.

However, both ideas would probably come into conflict with the following:


Boo! wrote:Uses more symbols
Perhaps uses a key(s) somehow to change cipher encryption type
Splits the message into four parts instead of 3, and arranges them on paper out of order

I could see him doing something relatively simple like this to make the cipher harder to crack than the last one, and evolve his method. Picking up ideas out of his 'code breaking for noobs' book he got from the library.


That would be a realistic possibility and something similar has already been discussed frequently. I agree with you that this could be a good explanation for the "next step" from z408 to z340. However, there is a problem: z340 has statistically very significant peaks at period 19 and at mirror/period 15.
This is the biggest indicator that z340 is based on transposition. I think that these peaks also occur when a plain text is transposed and then encrypted with several keys. That's something to test. In this case, however, it is extremely difficult to find a solution, as the key and plain text do not correlate.

I really can't think right now. The heat wave here is just annoying. In the evening it is still 38 degrees. Just now two Hobbits have thrown a ring through my window and muttered something about the "cracks of doom".

Translated with http://www.DeepL.com/Translator

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Tue Jul 31, 2018 2:22 pm
by Jarlve
Largo wrote:1) Have you ever examined the cycles separately for the upper and lower half? The MUJ_9MUJ_9MUJUMJM99UM_M example looks as if it were interrupted approximately in the middle. Some time ago I experimented with the perfect-cycle feature of AZDecrypt, but I had the feeling that 170 characters (one half of the cipher) are simply not enough analysis material.

2) Would it make sense to use cycles instead of ngram scores to evaluate your transposition solver? So that it' looking for transpositions that result in better cycles? In this case, however, only minimally invasive transpositions should be used, i.e. simple region-based flips and mirrors.

1) Yeah, but nothing too extensive. That's how I discovered the MUJ_9 cycle initially. It is indeed interrupted in the middle and it goes along really well with period 19! Perfect cycles are more prone to outliers so it is better to use the regular n-symbol cycles measurements instead.

2) In that case a 2-symbol cycles measurement would be better suited. So far I have found only a few anomalies: the 340 has a fairly high period 2 transposed cycle score and also some of the diagonal readings are fairly high, I think these are correlated. It is a 2 sigma observation so it is nothing to write home about. A high period 2 transposition cycle score can indicate 2 different substitution keys for the top and bottom half, though I consider that hypothesis to be unlikely. But please, when you get the n-symbol cycles score system up and running consider your doing your own analysis. I am very skeptic towards transposition after or during substitution but maybe you can find something I have not.

Good luck with the heat. We had the 38 degrees here in Belgium some days ago but since then it has gotten allot better. I recommend to eat lightly.

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Tue Jul 31, 2018 6:32 pm
by smokie treats
Yeah we have explored the cycles quite extensively. From all four corners reading in all directions, top half versus bottom half, rearranging rows to increase cycles, detecting areas of the message and positions where cycles are disrupted and comparing those with the pivots, compared transposition + homophonic cycles versus homophonic cycles + transposition, isomorphic cycle patterns, etc. etc. The top half is somewhat more cyclic than the bottom half. We have discussed the mechanics of cyclic encoding with pen and paper, and how a person could start with cycles and gradually become more random. As far as we can tell the message was transposed before encoding, if it was transposed, and encoded LRTB. L=2 cycles are easier to detect, and we found that there are alternation cycles like ABABABAB and there is also a spike at ABAABAABA which does not appear in test messages with random cycling or with alternation cycles. Oh, and we explored pandromic EDIT: palindromic cycles. Doranchak did a lot of work with us on most of the above, and Moonrock gave us some ideas about cycle patterns. I probably left something out, as I do not have the memory that Jarlve has.

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Wed Aug 01, 2018 9:08 am
by Largo
Jarlve wrote:I am very skeptic towards transposition after or during substitution but maybe you can find something I have not.

I agree with you in principle, after all we have enough indications against transposition after / during substitution. For example, the fact that there are 9 rows without repeated symbols. Nevertheless, there are possible transpositions that are compatible with the observations. Let's say z340 would be a simple homophonic substitution in which the lower half was rotated 180 degrees. Let's also say that P19 has nothing to do with transposition, but comes from a strongly repeating plain text. All this just to give an example. In this case, rotating the lower half would neither affect P19 nor the "rows without repeated symbols" statistics. What would become apparent in this case, however, would be a cycle sequence interrupted in the middle. Sure, if z340 were built that way, it would have been solved long ago. I just want to show that simple transpositions are quite possible. However, this cannot be the only reason why z340 has not yet been solved.
At the moment I am unfortunately very busy and with the actuel heat wave I don't really want to program in the evening. But I will do some cycle analyses soon, maybe I really notice something new.

smokie treats wrote:Yeah we have explored the cycles quite extensively.
...
We have discussed the mechanics of cyclic encoding with pen and paper, and how a person could start with cycles and gradually become more random.
...
I probably left something out, as I do not have the memory that Jarlve has.

Yes, I remember that. At that time, I had the following idea:
viewtopic.php?f=81&t=3337
It may sound arrogant, but for me it was the most logical explanation for the interrupted cycles in z408 and thus also in z340 at that time, so I hadn't studied this topic intensively. But in the meantime I see it a little different and would like to take another look at the strange cycles of z340. Maybe it's a good thing that I haven't read all the threads on the topic yet, so I can approach the matter in an unbiased way.


Translated with http://www.DeepL.com/Translator

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Wed Aug 01, 2018 11:37 am
by Jarlve
Largo wrote:Nevertheless, there are possible transpositions that are compatible with the observations. Let's say z340 would be a simple homophonic substitution in which the lower half was rotated 180 degrees. Let's also say that P19 has nothing to do with transposition, but comes from a strongly repeating plain text. All this just to give an example. In this case, rotating the lower half would neither affect P19 nor the "rows without repeated symbols" statistics. What would become apparent in this case, however, would be a cycle sequence interrupted in the middle. Sure, if z340 were built that way, it would have been solved long ago. I just want to show that simple transpositions are quite possible.

Let me illustrate something. The 340 has 26 length 17 unique sequences/strings. And after mirroring the lower half as you suggest this observation drops to 15 length 17 unique sequences. doranchak calculated the odds of 26 length 17 appearing randomly to about 1 in 600,000,000. It narrows down the transposition options even further.

Here is a picture. The red line are the unique sequence length frequencies. The other colors are different directions and green is the 340 fully mirrored. To interpret it differently, the red "mountain" is right shifted from the green "mountain". And the more the mountain is right shifted the better it looks as considering the direction as the actual sequential homophonic substitution direction. Mirroring the lower half left shifts the mountain. This sharp peak at 17 with a steep drop off is actually a bit weird. Though it would be weird if it would be normal, it is the 340 after all.

Image

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Sat Aug 04, 2018 10:47 am
by Largo
Hi,

I implemented the scoring shown by Jarlve and did some experiments. I remembered that sometimes shifting rows or the whole cipher leads to better period n peaks. By shifts of rows I mean the following:

Code: Select all
original row:
HERabcdVPeIfLTGgh

row shifted by 3:
GghHERabcdVPeIfLT


So why not test whether the 2-cycle-score also increases significantly? I wrote a little function that tries different row shifts. Here is the best result so far:

Shifting amounts (rows 1-20):
3, 15, 15, 1, 11, 4, 11, 7, 16, 0, 16, 5, 15, 13, 2, 0, 2, 16, 0, 6

Resulting Cipher:
Code: Select all
GghHERabcdVPeIfLT
+BjkOlDWYmnoKpqNb
stM+UZGWjqLkuHJBr
gSbbvdcwoVxbO++RK
2hI7FP+34e5yzM+u1
agDjbwRdFcO-ohCeF
8gUtXGVmuLIk7+KQl
NYu+9LzjGgJp2kO+y
nM+0+ZRgFBtrA#4Kh
-ucUV+dJ+ObvnFBr-
+R571EIDYBb0TMKOU
m+3BFgntcRJIo7T4M
zSrk+NI7FBtj8wRu#
dp7g40mtV41++cGFN
-+rBXfos4zCEaVUZ7
ItmxuBKjObdmpMQGg
qLRtT+Lf#Cn+FcWBI
+qWCuWtPOSHT5jqb+
IFehWnv1ByYOBo-Ct
wAIK8+aMDHNbeSuZO


The 2-Cycle score is 2433, an increase of 13.9% over the original z340. Is that significant? I honestly have no idea.
Compared to z340, the perfect n-cycles have also increased (obviously).

However, I have not yet made a comparison with other ciphers. Perhaps these observations are based on quite normal behavior.
I'll try to include the unique sequences length stuff shown in the last chart, so I can make such a comparison too.

Code: Select all
Perfect 4-symbol cycles:
PlsxPlsxP (30)
fsxQfsxQf (30)
lsxQlsxQ (20)
l23Xl23X (20)
l23Ql23Q (20)
l2XQl2XQ (20)
l3XQl3XQ (20)
sxQAsxQA (20)
v3XQv3XQv (30)
v8XQv8XQv8 (42)
23XQ23XQ (20)
3XQA3XQA (20)
58XQ58XQ58 (42)
y8XQy8XQy8 (42)
8XQA8XQA8 (30)

Perfect 5-symbol cycles:
l23XQl23XQ (30)


shift experiments 2.png

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Sun Aug 05, 2018 2:16 am
by Jarlve
Largo wrote:The 2-Cycle score is 2433, an increase of 13.9% over the original z340. Is that significant? I honestly have no idea.
Compared to z340, the perfect n-cycles have also increased (obviously).

However, I have not yet made a comparison with other ciphers. Perhaps these observations are based on quite normal behavior.
I'll try to include the unique sequences length stuff shown in the last chart, so I can make such a comparison too.

Each row has 17 unique offsets which give 4,064,231,406,647,572,522,401,601 possibilities. I do not think 2433 is significant in such a space. When comparing to other ciphers make sure to test some that are about as cyclic as the 340 since it is easier to improve upon cycles that are more randomized.

You can test the 340 versus itself for such a scheme (row offset shifts). Make random row shifts and count the number of times the 2-symbol cycle score improved. If the improvement ratio is like 20% or greater then that may be something. This improvement ratio idea (borrowed from smokie) only works when the space is sufficiently large since it is prone to outliers if for instance there are only 10 possibilities.

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Mon Aug 06, 2018 9:29 am
by doranchak
I have the impression that many operations will produce large increases in cycle scores.

Jarlve once found that deleting row 14 dramatically increases the 4-cycle score: viewtopic.php?p=54990#p54990

I looked for row swaps that improve cycle scores too. If you allow arbitrary re-arrangement of rows, you can get a lot of cycle improvements. I found a simpler rearrangement that improves cycles but I can't remember how much improved they are:

Re: Some thoughts on Z340 (dual-alphabet?)

PostPosted: Tue Aug 07, 2018 10:50 am
by Largo
Thank you Jarlve and Doranchak for the explanations.

My idea to search for better cycles with shifts was probably not well planned and a bit thoughtless. But you never know...
I did the shifting with some test ciphers (25% random encoding). The result is similar to z340: The cycles are improved quite quickly. Nothing special indeed.