Page 52 of 143

Re: Route Transposition and Phenomenon

PostPosted: Mon May 29, 2017 3:58 am
by Jarlve
smokie treats wrote:This is really great, Jarlve. I am so happy that you did that. Even though the solve above isn't perfect, it is still a pretty good solve. You can read a lot of it. I say when you are ready we test it on various different types of transpositions, a variety of messages, plines and transpositions with various disturbances. Compare with untranspositions and the regular solver, solve independent chunks of messages. Before attacking the 340. Thanks.

You are welcome and thanks. I will finish the row solver and then update AZdecrypt so that you can give it a try. For the moment being and individually, I would prefer to do as little as possible though.

Re: Route Transposition and Phenomenon

PostPosted: Mon May 29, 2017 4:27 am
by smokie treats
O.k., that is fine.

Re: Route Transposition and Phenomenon

PostPosted: Mon May 29, 2017 7:06 am
by Largo
That pline idea looks very interesting. I have to catch up and read the previous posts since I worked on some own ideas the last few weeks but for now I have an interposed question:

Load the following plaintext into peek-a-boo:

Code: Select all
ASIWASGOINOVERTHE
CORKANDKERRYMOUNT
AINSISAWCAPTAINFA
RRELLANDHISMONEYH
EWASCOUNTINIFIRST
PRODUCEDMYPISTOLA
NDTHENPRODUCEDMYR
APIERISAIDSTANDAN
DDELIVEROHORTHEDE
VILHEMAYTAKEYAITO
OKALLOFHISMONEYAN
DITWASAPRETTYPENN
YITOOKALLOFHISMON
EYYEAHANDIBROUGHT
ITHOMETOMOLLYSHES
WORETHATSHEDLOVEM
ENONEVERWOULDSHEL
EAVEMEBUTTHEDEVIL
TAKETHATWOMANYEAH
FORYOUKNOWSHETRIC


Now resize it to 20 x 17 and apply a diagonal transposition (the default one in the diagonal transposition dialog). What you get is a transposed text that shows bigram peaks at period 19 and flipped + period 15. Nothing new so far.

Code: Select all
AISNTKRIPLEICTANEONN
SAIRRRAAENTUDRADTANO
WOEOETCRONDNYDEIYEMU
GVCKNWRMUOAMNHAEPSOL
OEDUAASORLDATYNYIROA
HNOSFICPOETREOTHBMHN
AMINHSTTCSOKMTFIOTED
YSIDASSUDHASEODTEMLE
NANWRIDIOTIRLNEREUMV
TAEIPOARYHPLAMOVOEEH
LHFYRSEAFAAHOWOWVDTN
YIMPIVMOSKAHSLRAEEAO
NDNRIELAOETEDEEHKMFK
EEELHLWOYIHEVLTAOHUS
HIELATTYTSHEETTWAOWT
PDIKIIEHYSNHULTEYOEI
DVODYNGLTOSBIAYRNHRC


Now resize it back to 17x20 and encode it with the default key that comes with peek-a-boo. The bigram peaks are now at period 38 and flipped + period 15. The „original“ bigram peaks vanish.

Code: Select all
dhoALjncY:aCbwDqx
X7R1i5GVtZd3A8rkn
DBsiq=lXy=+NeGX7H
RTk0hTP;vEQbjAlV;
S=Z;qFdfYzXu=gBrD
iKXtUHZLT7Tcn=d2R
XMICeY=awGxX8p9;F
AD;5q2osNb1=j;LIh
Xw3kTzcBiKMvHpZoy
=k8+;:07dRlVCB5Xs
htuAPnfS;QNDgcY=i
GTFYUZ;XQ=ax2:pIT
V13dIDiFXl=lQHLqT
C;Y5Q;XzjZ2Kutdy+
D=7kRnh0UiXPwfBga
pj;Ijx3y:Ful=Tc2+
QU8ZXprMFC0:dsNTL
o2Pfw8lD=lsYH5jhc
gpT1AFvuNaTXxCkQ=
BTqJULXz45iTG72Ve


This does not happen with all plaintexts tough. If you take the first 340 letters of z408 and do the transposition described above, the p19/15 peaks are preserved after the homophonic substitution. Obviously this depends on the plaintext and the key.
To me it looks possible that the p19/15 peaks in z340 could be a false positive and the „real“ period is a different one. AZDecrypt solves the cipher shown above within a minute (mode „Solve + transpose“). So a „false“ period peak could not be the only problem with z340.
Has this been discussed before?

Re: Route Transposition and Phenomenon

PostPosted: Mon May 29, 2017 8:29 am
by smokie treats
Largo wrote:This does not happen with all plaintexts tough. If you take the first 340 letters of z408 and do the transposition described above, the p19/15 peaks are preserved after the homophonic substitution. Obviously this depends on the plaintext and the key.
To me it looks possible that the p19/15 peaks in z340 could be a false positive and the „real“ period is a different one. AZDecrypt solves the cipher shown above within a minute (mode „Solve + transpose“). So a „false“ period peak could not be the only problem with z340.
Has this been discussed before?


Largo: Some thoughts. The 340 has a spike at p 15/19, but the bigram repeat count at period 1 is low. Evidence of transposition. If you do not transpose the plaintext, it is difficult to match these statistics. You can start with a plaintext that has a lower than average p1 and higher than average p19, then manipulate the key so that the p 1 bigrams are diffused more and the p 19 bigrams are diffused less. See this:

viewtopic.php?f=81&t=2617&p=43811&hilit=smokie18e#p43811

If you do not transpose the plaintext, then it is very difficult to match 340 p 15/19 stats. If you do transpose the plaintext, then it is much easier to match p 15/19 stats... and you can get phantom spikes at other periods depending on the plaintext and the key. I have been thinking about a small project, make one of these messages and then dissect it. If the 340 isn't a transposition at periods 15 or 19, then how do we know it is a transposition at all? If it was then Jarlve's program would have solved it anyway.

Pretty much everything you just said. A handful of null symbols, or skipped plaintext or symbols caused by transcription errors would cause misalignments in the untransposed message. I find it very plausible that someone could easily skip a few.

Check here for a list of possible transposition issues, and an attack plan that maybe could be used if there are disruptions or misalignments:

viewtopic.php?f=81&t=3196&start=330

That is why I am so excited about the independent row solver. I am hoping that the message can be solved with "sliding areas" plan and the independent row solver.

EDIT: Also consider scoring the bigram repeats according to probability. Here is my formula:

LN ( 1 / ( ( ( COUNT OF A / 340 ) * ( COUNT OF B / 340 ) ) ^ 2 ) )

Let's say you have only four of symbol A and only six of symbol B, and there are three occurrences of AB at p 19. That is pretty good evidence. Make a list of all bigram repeats, score them, sort them by score, and then graph the distribution. It is easier, by far, to match these stats at the correct period, but not at a phantom period, or with a randomly shuffled message.

Re: Route Transposition and Phenomenon

PostPosted: Tue May 30, 2017 2:43 am
by Jarlve
Jarlve wrote:I just checked p15 and 19 for the 340 and the best result came from p15 but does not seem to be a solve comparing it to 16671 score of the 408 piece.

Ignore these results, there was a significant bug. Will redo it at a later stage.

Re: Route Transposition and Phenomenon

PostPosted: Tue May 30, 2017 11:54 pm
by Jarlve
Here's another preview of your row solver concept smokie with added ngram scores in brackets.

Code: Select all
Adjusted multiplicity: 0.15525 Average ngram size: 4.8
Score: 23387.92 Ioc: 0.06542

ILIKEKILLINGPEOPL (1060)
EBECAUSEITISSOMUC (1129)
HFUNITIAMOREFUNTH (928)
ANKILLINGWILDGAME (988)
INTHEFORRESTBECAU (1080)
SEMANISTHEMOATDAN (915)
GERTUEANAMALOFALL (828)
TOKILLSOMETHINGGI (1103)
VESMETHEMOATTHRIL (933)
LINGEXPERENCEITIS (1015)
EVENBETTERTHANGET (1115)
TINGYOURROCKSOFFW (980)
ITHAGIRLTHEBESTPA (1001)
RTOFITIATHAEWHENI (865)
DIEIWILLBEREBORNI (944)
NPARADICEANDALLTH (1020)
EIHAVEKILLEDWILLB (1036)
ECOMEMYSLAVESIWIL (886)
LNOTGIVEYOUMYNAME (960)
BECAUSEYOUWILLTRY (1126)
TOSLOIDOWNORATOPM (788)
YCOLLECTINGOFSLAV (1002)
ESFORMYAFTERLIFEE (976)
BEORIETEMETHHPITI (698)

Re: Route Transposition and Phenomenon

PostPosted: Wed May 31, 2017 1:20 am
by Jarlve
Okay smokie, try this AZdecrypt demo update please: https://drive.google.com/open?id=0B5r0r ... FZSVlF0ZEE

To use the 5-gram solver of the substitution + by rows solver simply select it and click solve. And to use the 6-gram solver variant, go to file and then click load ngrams, select the reddit 6-grams file. After it has loaded select the solver and click solve. On first use it will load the smaller ngram sizes in memory so wait that out. Also, it is not a performance solver and you may want to wait a while longer with solutions to appear.

Let me know if there is anything wrong, or if you have any suggestions or questions. If it works well then I suggest you use it as you please because I might not make this update formal for a while.

Re: Route Transposition and Phenomenon

PostPosted: Wed May 31, 2017 3:15 am
by smokie treats
Hi Jarlve. Thanks a lot for all of your work. I just tried the row solver, but maybe doing something wrong. I made a message all homophonic, no transposition 17 x 20, then pasted only 15 x 20 into the solver cutting off two columns. The message seems to solve, but the output is one long row. It doesn't look like your output. Thanks.

Re: Route Transposition and Phenomenon

PostPosted: Wed May 31, 2017 3:44 am
by smokie treats
smokie treats wrote:Hi Jarlve. Thanks a lot for all of your work. I just tried the row solver, but maybe doing something wrong. I made a message all homophonic, no transposition 17 x 20, then pasted only 15 x 20 into the solver cutting off two columns. The message seems to solve, but the output is one long row. It doesn't look like your output. Thanks.


Nevermind, Jarlve. I put a period . at the end of each row and it worked awesome. You have contributed so much to this. Thanks a lot. This is a very useful tool.

Re: Route Transposition and Phenomenon

PostPosted: Wed May 31, 2017 3:49 am
by smokie treats
smokie treats wrote:
smokie treats wrote:Hi Jarlve. Thanks a lot for all of your work. I just tried the row solver, but maybe doing something wrong. I made a message all homophonic, no transposition 17 x 20, then pasted only 15 x 20 into the solver cutting off two columns. The message seems to solve, but the output is one long row. It doesn't look like your output. Thanks.


Nevermind, Jarlve. I put a period . at the end of each row and it worked awesome. You have contributed so much to this. Thanks a lot. This is a very useful tool.


It worked to stop the rows and show a score at the end of each row, but it added symbols, the rows have more symbols. It must be a paste from Excel issue, not sure. How should I manipulate the message or use the period . ? Thanks