Re: Documentation of tested ideas

Posted:
Sun Jun 26, 2016 5:52 am
by doranchak
Thank you very much for posting these! Your ideas are very compelling, and it would be interesting to generate test ciphers (with known plaintexts) under some of these schemes to see how they behave statistically compared to Z340.
Re: Documentation of tested ideas

Posted:
Thu Jun 30, 2016 6:01 pm
by doranchak
Here is my post related to documenting tested ideas:
viewtopic.php?f=81&t=3108I eventually would like to incorporate some of your ideas into my test cipher generation process, since it is well suited at producing Z340-like ciphers under different schemes. It takes a long time, though!
Re: Documentation of tested ideas

Posted:
Sun Jul 03, 2016 5:25 am
by Largo
doranchak wrote:Here is my post related to documenting tested ideas: viewtopic.php?f=81&t=3108
Wow, your webtool is great! I like the fast and clear way to visualize statistics. The new article on your wiki page is very useful.
doranchak wrote:I eventually would like to incorporate some of your ideas into my test cipher generation process, since it is well suited at producing Z340-like ciphers under different schemes
That would be great. I have a tool for generating test ciphers too, but it is not so well suited as yours. It just takes a plaintext and generate a homophonic substition from a given set of cipher symbols. I am looking forward to see your results.
What do you think about a section which lists hypothesis which renders z340 probably unsolvable? I started a list a couple of weeks ago (also with the assumption that z340 is a homophonic substituion of an english text):
- Too much polyphones
- Double columnar transposition with long keywords
- One time pad (statistics indicate that this is not the case)
- Filler of arbitrary length after each word
- Sporadic fake parts whereas the valid parts are too short
- Crossword puzzle
- Different keys whereas the enciphered parts are too short
Edit: Added new documentation to this thread:- Skipping letters / chunks of letters)
- Way transpositions
PS: If my english is a bit confusing please feel free to ask what my gibberish means =)
Re: Documentation of tested ideas

Posted:
Wed Jul 13, 2016 2:39 pm
by Largo
Update:
Tried 75264 (!) new permutations by removing columns and rows. Refer
this posting, section "
Skipping letters, rows or columns - Skipping columns (different reading directions)" for more details.
Re: Documentation of tested ideas

Posted:
Sat Jul 16, 2016 3:03 am
by Quicktrader
Hi Largo..
welcome especially as you deal with the cipher AND Python.
There is already some info regarding some Python results (use search func) and have I written a working tool to check certain passages of the cipher with a dictionary etc. There also was some Bernoulli analysis which delivered L and S as a candidate for the + symbol. The tool uses an aho-corasick algorithm, too.
Personally I do not believe in any transposition or so..the complexity of the 340 is way higher than of the 408 (less letters, more symbols). Hard enough to crack it as it is..so no use for transposition, imo. Also Z had never used such a technique, he rather was the homophone substitution guy..besides that, transposition would require alphabetical letters, however Z had used symbols..so if you perform any kind of (even successful) transposition, a list of symbols is what you get.
You may find these threads interesting:
viewtopic.php?f=81&t=907&hilit=+PARTIALLYviewtopic.php?f=81&t=2856&hilit=BernoulliQT
Re: Documentation of tested ideas

Posted:
Sun Jul 24, 2016 6:23 am
by Largo
Quicktrader wrote:besides that, transposition would require alphabetical letters, however Z had used symbols.
Hm...why should a transposition should require alphabetical letters? Imagine the following text to be encrypted:
- Code: Select all
ILIKEGRILLINGINTHESUMMER
Now put that text into a grid:
- Code: Select all
ILIKE
GRILL
INGIN
THESU
MMER!
Now rearrange the columns:
- Code: Select all
KLIIE
LRIGL
INGIN
SHETU
RMEM!
Now apply a homophonic substitution and you get a transposed homophonic cipher. Another way to do this is to cut the grid which contains the plaintext and rearrange it in a different order before applying the homophonic substitution. There are many other ways to transpose the plaintext.
But I totally agree with you: It does not make things easier =)
EDIT:I have updated the documentation a bit. Currently I am testing an idea with skipped symbols (assumed as filler). A total amount of 164220 permutations are currently fed into AZDecrypt. Refer "Skipping Letters" section for more details
Re: Documentation of tested ideas

Posted:
Sun Jul 24, 2016 7:10 am
by smokie treats
Largo, I am really glad that you are interested in the subject of detecting transcription skips.
Here is an idea. Make a route transposition message, and skip a plaintext when transcribing the plaintext into the transcription rectangle. If you do that, and then untranspose the message into a 17x20 grid. There will be a vertical line somewhere between two columns where the plaintext are misaligned, basically cutting words in half and splicing them with other words cut in half. The same thing happens when you add a null when transcribing into the transcription rectangle.
I gather that you already understand the concept, so here is the idea.
To detect the possible location of a transcription skip, make 340 new messages, with nulls inserted at position 1, 2, 3, etc. Then count the number of period 19 bigram repeats for each message. The null positions showing the largest increase in period 19 repeats are more likely to be the actual location of the skip. You can detect a transcription null in the same way, by making 340 new messages, skip a symbol at each of the 340 positions, and shift the symbols after the skip one position.
Sometimes the method works extremely well, and can detect the skip or null position exactly. Sometimes the method detects the general area of the skip or null. And sometimes the method doesn't work at all because of the homophonic encoding or other variables.
The first four or so rows of the 340 have a high concentration of period 18 bigrams that share the same symbols as period 19 bigrams elsewhere in the message. That could indicate an incomplete inscription rectangle, or a few transcription skips.
The pivots are offset by 39 positions, and could be repeated words in an area where there are a couple of transcription nulls. If you delete the symbol at either position 165 or 166, and delete a symbol at either position 210 or 211, you can create a huge spike in the count of period 38 bigram repeats, which for a period 19 transposition scheme would be period 2 bigram repeats before transposition. You will get values at about where they should be.
So perhaps instead of trying to detect possible permutations of skip or null locations with the solver, try to detect them with increases in the period 19, 38 etc. bigram repeat counts. Then, take the permutations that cause the biggest increase in repeat counts, untranspose, and feed those to the solver. This may be more efficient.
A handful of transcription errors could cause a transposition message to be very difficult to solve, and is a very plausible explanation for our problems.
Thanks again for your interest in the subject.
Re: Documentation of tested ideas

Posted:
Thu Jul 28, 2016 3:12 pm
by Largo
smokie treats wrote:So perhaps instead of trying to detect possible permutations of skip or null locations with the solver, try to detect them with increases in the period 19, 38 etc. bigram repeat counts. Then, take the permutations that cause the biggest increase in repeat counts, untranspose, and feed those to the solver. This may be more efficient.
Thank you for this great explanation. I will implement this into my python library! But I also think that there are some ways to create letter skips which can not be detected by an analysis like that (and assuming that the period 19 bigram repeats and the pivots are coincidences). Maybe you want to analyze the following cipher with your tools? I have put in a couple of random letters but the positions of those fillers are not random. They have a specific order.
- Code: Select all
ND1IlDo8JXc7XQ0nw
F2+bXnjdlABjantTU
XrIALi5AK5CMDlbdY
sdIzqhiRnxwuuD7kF
5oUXCG0T2xlFdMbXS
GL5q5ohInzNYMtXkS
bfkUTYicKsXuDjqkL
pgGYnXBwSexkUT5ti
YI0RI1i5PkzLi7HYZ
7BH0uIQ0nzXpXnsFY
#3kxQIu4paUiTLDjx
TkdCsXXjeZuuXhp51
UXuG3TDqHHILldoiY
V+s+wTYPA73Tc8XXj
duuXsh2czUXhq0TTx
DFdGkNc9tXrEJpwCw
2XUfNXxUXuuTzFF+z
lXnxs2ZnsM2#ku5XQ
0UxqXGyQK0VlXvumk
Mpxu+dQyUD+9vw8pj
Largo
-> If my english sounds harsh or impolite, please forgive me. My native languages are German, C++ and Python.
Re: Documentation of tested ideas

Posted:
Fri Jul 29, 2016 7:21 pm
by smokie treats
Largo,
I checked your message to find out if it is a route transposition, and am not sure that it is. There are more period 1 bigram repeats than any other period, but I tried to solve it with the solver programs and it doesn't solve. Is it a route transposition?
Your English is fine. My language is English, and I am pretty good with Excel. I have not written a computer program in 30 years until this week, and am starting to learn JS.
Re: Documentation of tested ideas

Posted:
Sun Jul 31, 2016 11:54 am
by Largo
smokie treats wrote:I have not written a computer program in 30 years until this week, and am starting to learn JS.
This is a good idea. It is always useful and fun to learn a programming language. There is so much which can be done e.g. writing tools to make things easier. I hope you will enjoy learning js!
The cipher is not a route transposition. It is an english text written from left to right, top to bottom. The only thing I've done is putting some useless letters at specific positions. This prevents AZDecrypt from solving the cipher. Unfortunately I made a mistake in the previously posted cipher. The substitution was not cyclic, instead is was random (which is not a problem as long as the "nulls" are removed. AZDecrypt will solve it then).
Here is the cipher text with cyclic homophonic substition:
Cipher "Largo_1"- Code: Select all
sdoclizwEXCqXQanL
pFfbXtjDlAkjgRVTU
XrIGNZ57KcCMdleiY
8DI1qhZntxsuudAB2
5oUXcGyTpPlFizbXv
7wCqIKh5RMLY1VXHS
e+kUTYDcoNXuZjAB8
20GYnXHsrb3kUTCtd
YI#R5zicaBKwD7HYZ
qkBfuCQgVMXpXnLFY
xyHPQIu42+UdTNij0
TkD58XXjeZuuXhpc1
UXuA3TdGBHCslioDY
t#waLTYf7qgTINXXj
ZuuX8hF5zUXhAxTTy
d2iGksc9RXvJEpwCL
FXUPNX+UXuuTK2p0M
lXV38FDns12#BuIXQ
aUf7XqgQoxtlXSumH
zpyuPZQ+Ud04rwLFj
Now it may be a bit easier to analyze it. I have no idea how to find the positions of the "nulls" without knowing the pattern. I am curious if anyone can show me how to isolate them.
If you are interested in the solution please follow this link:
https://www.dropbox.com/sh/w0qrhsv9sx0a ... FSKga?dl=0