Page 8 of 144
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 7:21 am
by Jarlve
smokie treats wrote:Regroup and refresh. Sometimes a break has a really good positive effect because you have the time to quietly mull things over for a while and come up with good new ideas.
Totally agree!

Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 8:31 am
by smokie treats
This morning I realized why I over did it on the masking. I was trying to get close to 340 numbers, where the list of bigram repeats is count 46. I started with a list of 104 count bigram repeats, and masked 42 bigrams. But since the new list didn't show the unmasked sister bigram repeats, I reduced the list by twice as much as I had intended.
The question is, if I had only used a wildcard count necessary to get the bigram repeat list down to about count 46, would the message still have been unsolvable? To get from 104 to 46, I would have only had to use count 29 wildcards.
I wonder how many bigram repeats Zodiac initially had to start with before alleged masking. Note that a lot of Zodiac's bigram repeats after alleged masking include several suspected wildcards. So perhaps Zodiac did more masking than the minimum necessary to get to count 46, but because he didn't carefully track how many bigram repeats he was actually creating when masking, the list of repeated bigrams after masking was longer than he intended. On the other hand, the alleged wildcards in the 340 bigram repeat list are high count, making them more likely to cause bigram repeats.
Final bigram repeat list count = initial bigram repeat list count - ( wildcard count used in bigram masking step * 2 ) + bigram repeats created by bigram masking step.
Makes me think about the + with count of 24 and the count of other alleged wildcards. So this is probably obvious to everyone in the room except me, but I have to write it down at least for myself.
340 factors include:
1. The message;
2. The key, which distributes 63 symbols among high, medium and low frequency letters in a way that causes a certain number of bigram repeats;
3. The randomization of the cycles, including which cycles, how much, and where;
4. The bigram masking step, including how many unique wildcard symbols and how many of each unique wildcard symbol; and
5. Mistakes
Smokie
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 12:11 pm
by daikon
Jarlve wrote:I decided to take a break for a while. Everything is okay, just proctecting myself from overextension. May the doge be with you.
Sorry to hear! Don't leave us for too long, it won't be the same without you. :)
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 12:17 pm
by daikon
smokie treats wrote:I should probably hold my horses on the "prospective sister bigram cycle reconstruction" idea. There are just so many uncertainties.
I think you idea about sister bigrams is good. But I'm not sure how far you can take it, as you said, there are too many uncertainties. Another thing that occurred to me. The '+' symbol in Z340 is the prime suspect of being a wildcard, correct? Why does it appear right next to itself (i.e. double '+') three times?! If the wildcards were used to reduce bigram repeats, here he creates a very obvious bigram repeat. He could've easily used one of the other wildcards. It would've reduced the overall number of '+'s as well. I can't explain this.
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 12:33 pm
by daikon
smokie treats wrote:The question is, if I had only used a wildcard count necessary to get the bigram repeat list down to about count 46, would the message still have been unsolvable? To get from 104 to 46, I would have only had to use count 29 wildcards.
That is definitely worth exploring! The more tests we have/run, the more reliable the empirical conclusion we can make based on the results. I would also use a different plaintext, as your un-wildcarded cipher is already pretty hard to solve, so adding even a small number of wildcards will likely render it unsolvable. Something from
Z's letters?
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 1:05 pm
by smokie treats
I am going to try to operate within tighter constraints. Try to make a message with initially high bigram repeats, so that after masking there are about 46 left over. Yet there have to be enough wildcards to make the message unsolvable. The last one was easy because I didn't have a constraint on the final number of bigram repeats. I don't know much about that subject, but I am going to start with a key that makes a symbol distribution as flat as possible. Then adjust the key. I am guessing that fewer symbols mapped to high frequency letters and more symbols mapped to lower frequency letters might do the trick. But I don't know yet. Try to do some randomizing of the cycles as well, so that I can match 340 stats. If the cipher will fit into a neat little box, then we may have something. If I can't make an unsolvable message with 46 bigram repeats, then then we may not have something. I just finished a spreadsheet that makes messages and allows me to select different cycles for randomization or not, and different percentages of randomization depending on where in the message you are.
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 6:37 pm
by smokie treats
I have a question.
Method 1: I have a message, and randomly place X number of wildcards on top of X number of symbols.
Method 2: I use the same message from Method 1, and use X number of wildcards to mask X number of bigrams.
On average, will the altered message from Method 2 be more difficult to solve than the altered message from Method 1?
If nobody answers, no problem, I will keep going anyway and eventually figure it out. But if anyone knows the answer, it would be greatly appreciated.
Smokie
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 7:00 pm
by daikon
If I had to guess, and this is purely a guess at this point, but I would say Method #2, where you mask bigram repeats, will be harder to solve. Reason being, the auto-solvers use the intrinsic order (in the sense of the opposite of randomness) in the cipher to converge on the correct solution. Basically they tweak the key (which letter each symbol in the cipher maps to) little by little until the solution makes the most sense. It helps a lot for different parts of the cipher to have the same patterns. It's kind of like assembling a jigsaw puzzle, where different pieces are linked together, so that when you place one piece (assign a letter to a symbol) all other linked pieces (all occurrences of that symbol in the cipher) have to fit as well (i.e. form a text that looks like normal English sentences). A bigram, a two-letter sequence, being the simplest form of a pattern, is bound to repeat the most, so it is the main building block of all repeating patterns in the cipher. And if you target bigram repeats, and destroy them specifically, you are more likely to arrive at a hard to solve cipher because there will be less order, and more randomness, in the cipher.
Hmm, now that I put the whole reasoning into words, I'm convinced that Z could've followed the same logic, and did specifically try to mask as many bigram repeats as he can spot. There were no known automatic software solvers at the time of course, but Z might have thought that FBI had some powerful mainframes that were capable of that task at the time.
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 7:38 pm
by smokie treats
Thanks for the answer. I think that you are probably right, but had to ask. The reason for the question is because I am trying to make a message that has enough bigram repeats so that after I mask, I get to about 47 remaining bigram repeats. Most of my messages have about +/- 100 bigram repeats to start with. With one I got 109. If I try to randomize the cycles, the number goes down. I am wondering if I start with about 100 bigram repeats and mask about 24 bigrams, will that do the job? If I can find a way to distribute about 60 symbols amongst the letters in the key to make that work, then no problem.
Still working on it, but the exercise is instructive. There are several high count (about 10) symbols in the 340. Some cycle well with each other, and some do not. I suspect that Zodiac may have lowered the count of symbols for high frequency letters, and raised the count for low frequency letters. He may have done that thinking that it could make frequency analysis more difficult. But it raised the number of bigrams a bit.
I also think that he may have masked the traditionally high frequency bigrams, targeting them on purpose instead of just masking at random. The reason I am saying this is because with the list of bigram repeats, there are several symbols which I have always suspected as wildcards because they are high count, and don't particularly cycle well with others. But, if he could have randomized a bit, those symbols may not be wildcards at all. Not sure yet. But if he targeted traditionally high frequency bigrams, then those may be the ones that he didn't target.
Also, if the message has at least a few low frequency letters, like B, K, or Z, that make a big difference. I can assign more symbols to those letters and fewer symbols to the high frequency letters. All in all, the distribution of symbols in the key does not look like it would if making the distribution of symbols as flat as possible. It's usually a more uniform or flat looking key. That would explain why there are so many low count symbols in the 340. He may have assigned them to low frequency letters, resulting in low count.
I've just been messing around with that, and have not decided on any message to mask yet. I have been trying some of Zodiac's letters. And some other stuff too, which gives me a few more initial bigram repeats.
Thanks again.
Another question for anyone out there. Has anyone ever tried to do this before? Has anyone ever tried to figure out a way that Zodiac could have masked bigrams with wildcards in a fairly cyclic message? I mean, if someone has already tried to do this, I would prefer to just read about it instead of doing it. Just wondering if I am going down a rabbit hole that someone else has already gone down.
Smokie
Re: Homophonic substitution

Posted:
Sun Aug 09, 2015 9:26 pm
by smokie treats
Anyway, I've done my second masking exercise. It was very educational. I made a message with perfect cycles. Then randomized them a little bit toward the end of the message. Then I started with 110 bigram repeats, masked 32 of them with wildcards, leaving about 46 bigram repeats left. I have to double check that. ZKD is working on it right now, stuck at about 32k after fifteen minutes. I found out that bigrams sometimes sit next to each other. During the masking process, I doubled up some of the wildcards. I didn't seek out to do that, but when given a choice between two bigrams to mask, and masking one already caused another bigram to be created, it was easier to mask the bigram that caused a double wildcard symbol. Going through the process really helped me understand a lot. I encourage people to try it. I would like to see if I can make an unsolvable message with only 24 wildcard symbols. That would have been a lot easier, because I could have used a key with mountains and valleys instead of trying to level out the symbols across the letters so that there would be more initial bigrams, then shifting symbols around. I have to get some sleep now. If anyone wants to try to solve it, here it is. Symbols 61, 62 and 63 are the wildcards. The cycles are randomized 2% to 6% incrementally as you go through Rows 9-20.
17 25 1 15 27 62 9 18 26 35 16 10 13 31 28 37 58
7 36 15 9 32 10 22 17 39 9 8 2 16 59 3 4 18
61 25 27 61 56 26 60 33 35 44 49 17 31 36 45 41 10
35 62 28 21 9 11 18 22 61 61 57 42 17 36 16 35 61
9 10 58 40 34 59 12 53 27 32 23 51 61 25 7 2 26
28 61 46 55 52 24 9 61 21 58 27 31 44 10 36 56 8
32 45 47 60 31 9 33 1 25 49 55 16 28 22 10 54 61
35 15 26 59 35 63 17 58 14 18 25 17 35 36 27 34 18
35 57 28 41 26 59 58 27 32 61 28 9 2 35 17 36 42
56 51 60 15 59 48 3 18 61 16 27 21 10 1 61 40 36
15 2 61 23 9 56 26 52 5 28 23 11 59 31 36 17 35
16 61 7 1 29 10 32 12 9 6 36 22 44 31 27 38 58
8 49 61 59 32 21 18 19 10 2 30 27 31 61 15 28 22
9 29 2 17 25 36 10 57 13 32 9 62 26 53 62 63 16
60 33 15 17 58 45 55 9 21 62 59 54 4 31 1 34 51
20 25 27 47 18 26 36 62 10 9 43 2 5 35 24 17 40
7 21 61 36 61 61 8 63 59 32 27 30 61 58 9 49 28
25 35 59 56 36 37 3 10 52 61 60 29 9 57 15 1 22
21 22 18 19 61 61 35 38 26 58 9 21 56 39 10 31 44
6 27 50 11 59 32 36 60 3 22 9 35 37 25 26 10 21
Hope I didn't make any mistakes. Good night.
EDIT: I made some mistakes. I started with 113 bigram repeats, but made some too, ending with 56 total because of the double wildcards. But I have a little system down now, and can work on fewer wildcards and bigram repeats next time.