Page 5 of 9

Re: Evidence of transposition misalignment in the 340

PostPosted: Sun Dec 11, 2016 6:00 pm
by smokie treats
Jarlve wrote:If you wish I could so some sort of scan for that, to your specifications.


Thanks a lot for the offer. For now I am fine, and I had to check out for the most part, for a little while.

I understand that moving column 17 to the column 1 position increases the period 15 / 19 bigram repeats. But I am not convinced that it means anything. Doranchak has a lot of cryptograms to compare with. I would suggest finding out if moving one column in any of those cryptograms would increase the bigram repeats as much to see if it rare or not.

Re: Evidence of transposition misalignment in the 340

PostPosted: Mon Dec 12, 2016 2:53 pm
by Largo
This might be a bit offtopic:

I have tried some transposition stuff which may not have been covered by now:

Imagine that the plaintext was transposed to chunks in which the order of the letters were shuffled. Example:

Code: Select all
Plaintext:
AVADAKEDAVRA

Chunk size 4, rearrangement order "3421"

Plaintext:
1234 1234 1234
AVAD AKED AVRA

Ciphertext:
4312 4312 4312
DAAV DEAK ARAV


I ran tests for all permutations of chunk sizes 2, 3, 4, 5, 6, 7, 8 and 9 with AZDecrypt (409112 test ciphers at all). Sometimes I got a pretty good scoring. The highest one was 20976 from a cipher with chunk size 9 and rearrangement pattern "7, 3, 4, 8, 1, 9, 5, 2, 6".

May you check the result with your statistic tools please? It would be interesting to know if this result is just some noise again:


Code: Select all
AZdecrypt 0.992 (Practical Cryptography 5-grams) Progressive, Index of coincidence

Score: 20975.64 Ioc: 0.07417056 Entropy: 3.867192 Chi-square: 39.93682 Characters: 333 Letters: 18

aoflinduseitiscis
twanedfornamhaets
taterfinanddoodad
itposedonhalloser
reriesinorschosta
tedgesahesfinisno
tewherlanthouetan
tstoespricenteror
medlorichangestew
asaparlortomanaed
astherecouncieame
dshopingsshorthel
atehoredactstheen
consthenalgacurri
ghtcalliaidugfors
ohanatandestantto
nithieraresiasmfo
raftnfrindiandint
osicsafeminanifhe
adicoeddes

Multiplicity: 0.1891892 Characters: 333 Symbols: 63

dRaVHPbEcgfLheNTI
GlBjDbWk+OqnoBYrK
mpGM+WsjUtZSkubqb
HLJxcwbvOodVzRKM+
+g+yF2hPu+I17Re5d
3Fb4we-oFcahOCKjk
Qgl7D+VUtm8uXgGpj
G2LkgIJ+yN9OLY+u+
nMZzR+h0#Bt4gKrFA
dcUJ-+Vu+rvn-OUFb
BI57D+Y1REO0TgBnM
bK7RJTt4Icou+3#Mz
BmF7k+FSBNrIG8wFt
NRjcm7gtdV4p0X++f
4or1BVzCUsZE4ax+I
u7Bt-mpObMKQdjmLR
tfG#Tg+B+FICqcnWu
+qWLtW+CjSHqPbTO5
veh1IBWFnCOBtyaoY
-SHNuMZbDe

Re: Evidence of transposition misalignment in the 340

PostPosted: Tue Dec 13, 2016 5:21 am
by Jarlve
Largo wrote:I have tried some transposition stuff which may not have been covered by now:

You basicly mean rearranging the columns in a set of dimensions. Me and daikon have done some work on it, nothing extensive.

Largo wrote:May you check the result with your statistic tools please? It would be interesting to know if this result is just some noise again:

You'll have to decide for yourself.

One thing you can do is look at the ioc and/or entropy. With the practical cryptography ngrams and typical AZdecrypt settings the ioc will typically inflate a bit if it can't find a legit solution. Solves that have an ioc under 0.07 and/or entropy of over 4 are more interesting. Another thing is to look at repeating fragments, more fit plaintext will have longer repeating fragments.

Here are the higher ngram repeats of a 340 character part of the 408:

Code: Select all
7-gram frequencies > 1:
--------------------------------------------------
BECAUSE: 3
KILLING: 2
EBECAUS: 2
THEMOAT: 2

8-gram frequencies > 1:
--------------------------------------------------
EBECAUSE: 2

And here is your solve:

Code: Select all
4-gram frequencies > 1:
--------------------------------------------------
sthe: 3
stat: 2
tate: 2
inan: 2
tant: 2

5-gram frequencies > 1:
--------------------------------------------------
state: 2

You can also inspect the plaintext in AZdecrypt 1.0 with stats -> plaintext direction 1b and 2. And compare scores with other plaintext.

Re: Evidence of transposition misalignment in the 340

PostPosted: Tue Dec 13, 2016 9:02 am
by smokie treats
Largo wrote:This might be a bit offtopic:

I have tried some transposition stuff which may not have been covered by now:

Imagine that the plaintext was transposed to chunks in which the order of the letters were shuffled. Example:

Code: Select all
Plaintext:
AVADAKEDAVRA

Chunk size 4, rearrangement order "3421"

Plaintext:
1234 1234 1234
AVAD AKED AVRA

Ciphertext:
4312 4312 4312
DAAV DEAK ARAV


I ran tests for all permutations of chunk sizes 2, 3, 4, 5, 6, 7, 8 and 9 with AZDecrypt (409112 test ciphers at all). Sometimes I got a pretty good scoring. The highest one was 20976 from a cipher with chunk size 9 and rearrangement pattern "7, 3, 4, 8, 1, 9, 5, 2, 6".

May you check the result with your statistic tools please? It would be interesting to know if this result is just some noise again:


Code: Select all
AZdecrypt 0.992 (Practical Cryptography 5-grams) Progressive, Index of coincidence

Score: 20975.64 Ioc: 0.07417056 Entropy: 3.867192 Chi-square: 39.93682 Characters: 333 Letters: 18

aoflinduseitiscis
twanedfornamhaets
taterfinanddoodad
itposedonhalloser
reriesinorschosta
tedgesahesfinisno
tewherlanthouetan
tstoespricenteror
medlorichangestew
asaparlortomanaed
astherecouncieame
dshopingsshorthel
atehoredactstheen
consthenalgacurri
ghtcalliaidugfors
ohanatandestantto
nithieraresiasmfo
raftnfrindiandint
osicsafeminanifhe
adicoeddes

Multiplicity: 0.1891892 Characters: 333 Symbols: 63

dRaVHPbEcgfLheNTI
GlBjDbWk+OqnoBYrK
mpGM+WsjUtZSkubqb
HLJxcwbvOodVzRKM+
+g+yF2hPu+I17Re5d
3Fb4we-oFcahOCKjk
Qgl7D+VUtm8uXgGpj
G2LkgIJ+yN9OLY+u+
nMZzR+h0#Bt4gKrFA
dcUJ-+Vu+rvn-OUFb
BI57D+Y1REO0TgBnM
bK7RJTt4Icou+3#Mz
BmF7k+FSBNrIG8wFt
NRjcm7gtdV4p0X++f
4or1BVzCUsZE4ax+I
u7Bt-mpObMKQdjmLR
tfG#Tg+B+FICqcnWu
+qWLtW+CjSHqPbTO5
veh1IBWFnCOBtyaoY
-SHNuMZbDe


I agree that the message could have been transposed in chunks. However, they are not small chunks. If you make a rail fence cipher with 30 columns and 2 rows, that is a chunk of 30 which will create period 15 repeats. You could make 11 of them to fit into the 340, but you will have a very difficult time getting 340 period 15 stats because each individual chunk is independent of every other chunk. Period 15 repeats are created within chunks, but not between chunks, except by coincidence.

Likewise, a bifid works with chunks. You can create a lot of period 15 repeats by working with chunks of 30 plaintext. But you get the same problem. The period 15 repeats are created within the chunks, and not between or across them except by coincidence. So you will have a difficult time matching 340 period 15 stats.

Grille cipher anagrams in chunks, but I could not get very many period 15 repeats within chunks of 64.

However, you can get 340 stats period15 stats with multiple inscription rectangles. The bigger the rectangles, the more you get.

There may be a way to detect this. Draft the message into different numbers of columns. Divisors of 15, multipliers of divisors of 15, and multipliers of 15. And starting at 340 different positions. Make a list of period 15 bigrams within each row but not from row to row. Then count the number of repeats. The arrangement that results in the highest count of repeats may show the chunk size. For example, if the result is 75 columns, then the chunk size may be 75, a multiplier of 15. The first symbol of each chunk and the last symbol of each preceding chunk may also be a period 1 bigram that has symbols that match other period 15 bigrams in the arrangement. They may appear on a horizontal number line, at increments of the chunk size, similar to your coincidence count spike.

Re: Evidence of transposition misalignment in the 340

PostPosted: Tue Dec 13, 2016 5:43 pm
by Jarlve
This could be fun, can anyone find the chunksize I used?

Code: Select all
;6MOX?&!R9=XFUHA3
P56&N$K*4#_@Z(8Q:
X#HWC[2+HI>*UK7J/
?%^M@\XE'&#5@91;P
=Y&DHA[78DZ\^EFZX
$(<RB$3LY?KXQ'.B:
+CD#PG@1[U7H9>[T5
^Q$Q:7XD0#RJE%&@Y
CMXEL%+N=K!]F28ZN
9F>@X')"*!#&$WQ=0
@A(:X#1+B^HKT%CE0
LN@O!RYX2$=T?!0AU
@X\7DM&96HJ#1K>@[
V%7=L;?.TF'Z)QS5'
O8-NWB>C%2F8\XM$#
@0XEX,=3%P[+D6>UO
97QNY!+8TFN'#?%>@
=F1XARJ#N^5@(:;'X
KP_9Z=U^YN#@2:QRT
XH#YCM9!@+R&%=87X

1 2 3 4 5 6 7 8 9 10 11 5 12 13 14 15 16
17 18 2 7 19 20 21 22 23 24 25 26 27 28 29 30 31
5 24 14 32 33 34 35 36 14 37 38 22 13 21 39 40 41
6 42 43 3 26 44 5 45 46 7 24 18 26 10 47 1 17
11 48 7 49 14 15 34 39 29 49 27 44 43 45 12 27 5
20 28 50 9 51 20 16 52 48 6 21 5 30 46 53 51 31
36 33 49 24 17 54 26 47 34 13 39 14 10 38 34 55 18
43 30 20 30 31 39 5 49 56 24 9 40 45 42 7 26 48
33 3 5 45 52 42 36 19 11 21 8 57 12 35 29 27 19
10 12 38 26 5 46 58 59 22 8 24 7 20 32 30 11 56
26 15 28 31 5 24 47 36 51 43 14 21 55 42 33 45 56
52 19 26 4 8 9 48 5 35 20 11 55 6 8 56 15 13
26 5 44 39 49 3 7 10 2 14 40 24 47 21 38 26 34
60 42 39 11 52 1 6 53 55 12 46 27 58 30 61 18 46
4 29 62 19 32 51 38 33 42 35 12 29 44 5 3 20 24
26 56 5 45 5 63 11 16 42 17 34 36 49 2 38 13 4
10 39 30 19 48 8 36 29 55 12 19 46 24 6 42 38 26
11 12 47 5 15 9 40 24 19 43 18 26 28 31 1 46 5
21 17 25 10 27 11 13 43 48 19 24 26 35 31 30 9 55
5 14 24 48 33 3 10 8 26 36 9 7 42 11 29 39 5

Re: Evidence of transposition misalignment in the 340

PostPosted: Tue Dec 13, 2016 7:39 pm
by smokie treats
Jarlve, could you give me a hint? There is no distinctive spike, except maybe at period 6 reading right left top bottom. Could you tell me the period and transcription direction, and if there are nulls at the end of the message? Thanks.

Re: Evidence of transposition misalignment in the 340

PostPosted: Wed Dec 14, 2016 5:31 am
by Jarlve
Period 1, left-to-right, top-to-bottom. No hint toward the chunk size.

Here's something else. I was looking at bigram repeats map. Images that show where the periodical repeats are and visually found a large repeating structure in the 408 at period 10. Here is a compiled image with in the middle the actual bigram repeat map at period 10. On the left and the right side the structure is offset on eachother to show the amount of repeats involved. If not a coincidence then how does this work?

Image

Re: Evidence of transposition misalignment in the 340

PostPosted: Wed Dec 14, 2016 5:46 am
by smokie treats
Period 1?

Re: Evidence of transposition misalignment in the 340

PostPosted: Wed Dec 14, 2016 7:05 am
by Jarlve
You were describing a way to detect Largo's columnar rearrangement scheme. I thought it would be interesting if we could come up with some statistic to do just that so I made such a cipher to see if we could. No extra periodical transposition or direction was applied. The message existed from left-to-right, top-to-bottom and then I chose a chunk size and randomized the order. Can someone figure out the chunk size? If so, then it would be interesting to see what it would yield on the 340.

Largo wrote:
Code: Select all
Plaintext:
AVADAKEDAVRA

Chunk size 4, rearrangement order "3421"

Plaintext:
1234 1234 1234
AVAD AKED AVRA

Ciphertext:
4312 4312 4312
DAAV DEAK ARAV


Re: Evidence of transposition misalignment in the 340

PostPosted: Wed Dec 14, 2016 8:06 am
by Largo
I have tried to solve your cipher but had no success. I had tried chunk sizes 2, 3, 4, 5, 6, 7 and 8 (a total amount of 46232 permutations). Then I read the last posts again and recognized that you are talking about columnar transpositions. Just to clarify I want to make sure that we are talking about the same thing and so I have created a sheet which describes what chunk based transposition means to me. May you have a look and tell me if we are talking about the same thing?

Leer.pdf