Re: AZdecrypt WinGUI
Posted: Sat Apr 06, 2019 12:36 amZodiackillersite.com
Discussion About the Zodiac killer
http://www.zodiackillersite.com/
AZdecrypt 1.19
Page 18 of 29
Posted: Sat Apr 06, 2019 12:36 amRe: AZdecrypt WinGUI
Posted: Thu Apr 18, 2019 3:20 amThere is a problem with the solvers "Substitution + polyphones [auto] and [user]" in AZdecrypt 1.14. In the meanwhile go back to 1.13 to get them to work properly.
Re: AZdecrypt WinGUI
Posted: Mon Apr 22, 2019 5:11 amWow, great update mr Jarvle!!!
Im honored y the quote!!
thanks !!!
Im honored y the quote!!
thanks !!!
Re: AZdecrypt WinGUI
Posted: Sat May 25, 2019 2:28 amAZdecrypt 1.15:
https://drive.google.com/open?id=1YOBOX ... z7FYJsVuVg
Again a huge update and I will illustrate some examples of the new functionality during the next days.
- beijinghouse made a special 8-gram system that is only 50% slower than 7-grams and uses only 8 GB of memory! It works very well and he can elaborate on how exactly he achieved this. Get his 8-grams here: https://drive.google.com/open?id=1oIHa6 ... ISz5nLU94c You will still need about 12 GB of ram to use these.
- beijinghouse also came up with a speed optimization that aborts the n-gram score calculation when it is known that the new_score cannot be higher than the old_score. This works very well with sparse n-grams such as 7-grams or higher. For now it is only used with beijinghouse's 8-grams.
- Added extensive cipher and plain text library, it can be found in the "Ciphers/" folder.
- Internally the score calculation is now using entropy instead of the index of coincidence. This may change the scores you are used to seeing a little bit. A small adjustment to the n-gram ini files has been made to reflect this. Old ini files still work via internal conversion.
- You may notice a slowdown of the solving speed (about 20%) due to a big improvement to most of the solvers such that for example 500,000 iterations are now worth about 1,000,000 iterations with a greater effect on ciphers that tend to get stuck.
- New function "Bias n-grams", bejinghouse asked for this, it can be found under "File, Load n-gram bias" and it tunes the n-grams in memory to some text of your choice. For example: to offer support for rare occurring n-grams such as names.
- "Batch ciphers (substitution)" now supports plain text accuracy testing, a example can be found in the "Ciphers/Batch/" folder.
- New solver "Substitution + units", it allows to solve a variety of cipher schemes such as masootz's cipher or operations on symbols, rows or columns such as remove, expand, separate into 2nd key, replace and reverse.
- New option "(Substitution + sequential homophones) Sequential weight: 5", setting it lower will decrease the weight of the sequential homophones and setting it higher will increase it.
- New function "Normalize n-gram factor", the button can be found under "Options, Solvers" and it will normalize the n-gram factor to some internal plain text. This ensures that AZdecrypt performs optimally with your n-grams.
- Fixed major 1.14 issues with the following solvers: columnar transpostion & rearrangement, nulls and skips, and poly [auto, user].
- Removed many code redundancies in the source (merged solvers etc) and it is about 33% smaller.
- New function "Batch settings" to optimize temperature etc, suggested by beijinghouse.
- Solvers are using much less memory outside of the regular n-gram memory use.
- Vigenère solver works much better now.
beijnghouse's new 7-grams: https://drive.google.com/open?id=1NFV-P ... 35wlL47CFC
https://drive.google.com/open?id=1YOBOX ... z7FYJsVuVg
Again a huge update and I will illustrate some examples of the new functionality during the next days.
- beijinghouse made a special 8-gram system that is only 50% slower than 7-grams and uses only 8 GB of memory! It works very well and he can elaborate on how exactly he achieved this. Get his 8-grams here: https://drive.google.com/open?id=1oIHa6 ... ISz5nLU94c You will still need about 12 GB of ram to use these.
- beijinghouse also came up with a speed optimization that aborts the n-gram score calculation when it is known that the new_score cannot be higher than the old_score. This works very well with sparse n-grams such as 7-grams or higher. For now it is only used with beijinghouse's 8-grams.
- Added extensive cipher and plain text library, it can be found in the "Ciphers/" folder.
- Internally the score calculation is now using entropy instead of the index of coincidence. This may change the scores you are used to seeing a little bit. A small adjustment to the n-gram ini files has been made to reflect this. Old ini files still work via internal conversion.
- You may notice a slowdown of the solving speed (about 20%) due to a big improvement to most of the solvers such that for example 500,000 iterations are now worth about 1,000,000 iterations with a greater effect on ciphers that tend to get stuck.
- New function "Bias n-grams", bejinghouse asked for this, it can be found under "File, Load n-gram bias" and it tunes the n-grams in memory to some text of your choice. For example: to offer support for rare occurring n-grams such as names.
- "Batch ciphers (substitution)" now supports plain text accuracy testing, a example can be found in the "Ciphers/Batch/" folder.
- New solver "Substitution + units", it allows to solve a variety of cipher schemes such as masootz's cipher or operations on symbols, rows or columns such as remove, expand, separate into 2nd key, replace and reverse.
- New option "(Substitution + sequential homophones) Sequential weight: 5", setting it lower will decrease the weight of the sequential homophones and setting it higher will increase it.
- New function "Normalize n-gram factor", the button can be found under "Options, Solvers" and it will normalize the n-gram factor to some internal plain text. This ensures that AZdecrypt performs optimally with your n-grams.
- Fixed major 1.14 issues with the following solvers: columnar transpostion & rearrangement, nulls and skips, and poly [auto, user].
- Removed many code redundancies in the source (merged solvers etc) and it is about 33% smaller.
- New function "Batch settings" to optimize temperature etc, suggested by beijinghouse.
- Solvers are using much less memory outside of the regular n-gram memory use.
- Vigenère solver works much better now.
beijnghouse's new 7-grams: https://drive.google.com/open?id=1NFV-P ... 35wlL47CFC
Re: AZdecrypt WinGUI
Posted: Sat May 25, 2019 11:03 amI'm astounded that you put so much effort into this and then make it available for free to anybody who wants it. Your generosity absolutely blows my mind. Thank you for this -- I can't wait to start using it!
Re: AZdecrypt WinGUI
Posted: Sun May 26, 2019 12:36 amDMW wrote:I'm astounded that you put so much effort into this and then make it available for free to anybody who wants it. Your generosity absolutely blows my mind. Thank you for this -- I can't wait to start using it!
Thanks for your kind words DMW.
Re: AZdecrypt WinGUI
Posted: Sun May 26, 2019 12:46 amHere's an example of the new solver "Substitution + units".
If you have a cipher in the input window and click on Solve with the units solver selected you will get a new window with some stuff in it. If someone may remember smokie's cipher 5, it has 2 different homophonic keys by rows, odd even in its case. This is something the units solver can tackle.
I set Unit to "Row" and Mode to "Separate into 2nd key". And Key length start and stop at 10 so that any distribution will be covered (in this case the key length are the amount of rows). I also put Multiplicity weight at 1 to punish to solver for hill climbing the multiplicity:
And then clicked Start solver with 6-grams loaded for good measure:
If you have a cipher in the input window and click on Solve with the units solver selected you will get a new window with some stuff in it. If someone may remember smokie's cipher 5, it has 2 different homophonic keys by rows, odd even in its case. This is something the units solver can tackle.
I set Unit to "Row" and Mode to "Separate into 2nd key". And Key length start and stop at 10 so that any distribution will be covered (in this case the key length are the amount of rows). I also put Multiplicity weight at 1 to punish to solver for hill climbing the multiplicity:
And then clicked Start solver with 6-grams loaded for good measure:
- Code: Select all
Restart: 1 Hill climber: 1635/5000 @ 500000
Score: 19531.07 IOC: 0.06304 Multiplicity: 0.30588
Repeats: 1262 PC-cycles: 272
Unit: row(17,20)
Mode: separate
Key(1,3,5,7,9,11,13,15,17,19)
DLIKEKILLINGPEOPL
EBECAUSEITISSOMUC
HFUNITISMOREFUNTH
ANKILLINGWILDGAME
INTHEFORRESTBECAU
SEMANISTHEMOSTDAN
GEROUSANIMALOFALL
TOKILLSOMETHINGGI
VESMETHEMOSTTHRIL
LINGEXPERENCEITIS
EVENBETTERTHANGET
TINGYOURSOCKSOUTW
ITHAGIRLTHEBESTPA
RTOFITISTHATWHENI
FIEDWILLBEREBORNI
NPARADICEANDALLTH
EDHAVEKILLEDWILLB
ECOMEMYSLAVESIWIL
LNOTGIVEYOUMYNAME
BECAUSEYOUWILLTRJ
Re: AZdecrypt WinGUI
Posted: Sun May 26, 2019 5:35 pmWhile it's certainly not necessary to understand how the new 8-gram solver works to use it, he's a few details on how I designed it. Hopefully this partial documentation helps someone who wants to make their own 8-gram files or modify my solver routines or just understand how it's so space-efficient while also being rather fast.
* First off, the 8-grams are encoded in a way that allows them to load into two 4GB scoring tables rather than one massive 208GB sparse array. Each 8-gram is addressed by two 4-letter sub-grams for each half of the 8-gram. The two tables have separate encodings but there is a primary table that has roughly 85% of all the data and then a fail-over table that has the other 15%. If you glance at the code, you can see how I address the tables and use the failover call if there's no value in the first table when looking up values in the main solver routines.
* For the file format, rather than distribute and load a 900+ million line raw text file, I made a custom file encoding method. Some of how it is encoded is just to have a concise bit-level encoding. But the trickier quirks are there so that the resulting bit-fields compress better than average with gzip. First, the file has four table, each with 65565 entries in 4 plain-text letter groupings. These are the encodings for all the 1) 1st 4-letter subgram aka letters 1-4 of the main table, 2) 2nd 4-letter subgram of the main table aka letters 5-8, 3) 1st 4-subgram of the backup table, and 4) 2nd 4-subgrams of the backup table. These encodings are only used internally during the n-gram loading routine. Someone making their own 8-grams would need to also decide which 8-grams they want to be able to represent, and lay them out the same way.
* The next 4 x 2 x 255 bytes in the file are raw binary indicating how many 8-grams are coming next with each score from 255 --> 1. This covers the number of entries present in both the main table and the backup table. This is important because the next part is the actual n-gram scoring section.
* The ngrams themselves are stored in what I'm referring to as a "spread format". Each sub-ngram in the first 4 tables implicitly corresponds to values 1 --> 65565 in this table. Because these values can be stored in 2^16 bits, I merely allocate 4 bytes for each value. But rather than put them sequentially, I order them with all the top bytes for all values of each score first, then all second bytes, then all third bytes, then all low order bytes. The reason for this interleaved encoding is only to assist gzip. These files compress 60% more because of the massive redundancy that this encoding format surfaces.
* These encoding methods for the 8-grams and their internal table representation are a bit extreme. But they work really well. Download time, loading time, and system memory requirements are all cut by factors of 20. And the iteration time is not much worse than 7-grams despite being run over a much larger state space.
Happy to answer other questions about the 8-gram solver if others are curious.
* First off, the 8-grams are encoded in a way that allows them to load into two 4GB scoring tables rather than one massive 208GB sparse array. Each 8-gram is addressed by two 4-letter sub-grams for each half of the 8-gram. The two tables have separate encodings but there is a primary table that has roughly 85% of all the data and then a fail-over table that has the other 15%. If you glance at the code, you can see how I address the tables and use the failover call if there's no value in the first table when looking up values in the main solver routines.
* For the file format, rather than distribute and load a 900+ million line raw text file, I made a custom file encoding method. Some of how it is encoded is just to have a concise bit-level encoding. But the trickier quirks are there so that the resulting bit-fields compress better than average with gzip. First, the file has four table, each with 65565 entries in 4 plain-text letter groupings. These are the encodings for all the 1) 1st 4-letter subgram aka letters 1-4 of the main table, 2) 2nd 4-letter subgram of the main table aka letters 5-8, 3) 1st 4-subgram of the backup table, and 4) 2nd 4-subgrams of the backup table. These encodings are only used internally during the n-gram loading routine. Someone making their own 8-grams would need to also decide which 8-grams they want to be able to represent, and lay them out the same way.
* The next 4 x 2 x 255 bytes in the file are raw binary indicating how many 8-grams are coming next with each score from 255 --> 1. This covers the number of entries present in both the main table and the backup table. This is important because the next part is the actual n-gram scoring section.
* The ngrams themselves are stored in what I'm referring to as a "spread format". Each sub-ngram in the first 4 tables implicitly corresponds to values 1 --> 65565 in this table. Because these values can be stored in 2^16 bits, I merely allocate 4 bytes for each value. But rather than put them sequentially, I order them with all the top bytes for all values of each score first, then all second bytes, then all third bytes, then all low order bytes. The reason for this interleaved encoding is only to assist gzip. These files compress 60% more because of the massive redundancy that this encoding format surfaces.
* These encoding methods for the 8-grams and their internal table representation are a bit extreme. But they work really well. Download time, loading time, and system memory requirements are all cut by factors of 20. And the iteration time is not much worse than 7-grams despite being run over a much larger state space.
Happy to answer other questions about the 8-gram solver if others are curious.
Re: AZdecrypt WinGUI
Posted: Tue May 28, 2019 1:15 amIt is an amazing piece of work beijinghouse and I am very happy to have your 8-gram system in AZdecrypt! It certainly pushes what we can do on the 340 in terms of hypothesis testing.
If you look at the ini file of the beijinghouse's 8-gram file you see:
The "bh8" at N-gram size tells AZdecrypt to branch off to beijinghouse's n-gram system.
If you look at the ini file of the beijinghouse's 8-gram file you see:
- Code: Select all
N-gram size=bh8
N-gram factor=36.039
Entropy weight=1
Alphabet=ABCDEFGHIJKLMNOPQRSTUVWXYZ
Temperature=600
The "bh8" at N-gram size tells AZdecrypt to branch off to beijinghouse's n-gram system.
Re: AZdecrypt WinGUI
Posted: Tue May 28, 2019 1:28 amThere's also a 2-gram solver in 1.15 which can do a 71.32% accurate decryption of the 408:
This can be further improved by using the "Substitution + sequential homophones" solver and tweaking the "Sequential weight" setting in the "Options, Solvers" menu. In the case of 2-grams I reduced the sequential weight all the way down to 0.2 (from 5) and got a 91.42% accurate solve on the 408:
- Code: Select all
Score: 23419.11 IOC: 0.06264 Multiplicity: 0.13235
Repeats: 646 PC-cycles: 6841
ISIKEKADSANGPEOPD
EBECOURAITISSOLUC
HFUNATAULOREFUNTH
ANKINDINGWASSGILE
ANTHEFORMERTBECOU
SELONISTHALOOTMAN
GERTUEINOLODOFAND
TOKINDSOLETHANGGA
VESLETHELOOTTHRIN
DINGEXPAMENCEATAR
EVENBETTERTHINGAT
TINGYOURMOCKSOFFW
ITHOGARSTHEBESTPO
RTOFATIOTHIEWHENI
MAEAWINDBEMEBORNI
NPARISACAUNMOSSTH
EAHAVEKISSESWINDB
ECOLELYRNOVESIWID
SNOTGAVEYOULYNALE
BECOUSEYOUWISNTRY
TORDOIMOWNOMUTOPL
YCONNACTINGOFRSOV
ESFORLYOFTERDIFEA
BEORIETALETHHPATA
This can be further improved by using the "Substitution + sequential homophones" solver and tweaking the "Sequential weight" setting in the "Options, Solvers" menu. In the case of 2-grams I reduced the sequential weight all the way down to 0.2 (from 5) and got a 91.42% accurate solve on the 408:
- Code: Select all
Score: 16967.41 IOC: 0.06256 Multiplicity: 0.13235
Repeats: 862 PC-cycles: 12958
IDIKEKILDINGPEOPL
EBECAUSEITISSOMUC
HFUNITISMORAFUNTH
ANKILLINGWIDSGAME
INTHEFORRESTBECOU
SEMANISTHEMOOTSAN
GARTUEANOMALOFALL
TOKILLSOMETHINGGI
VESMETHEMOOTTHRIL
LINGEXPERANCEITIS
EVENBETTERTHANGET
TINGYOURROCKSOFFW
ITHOGIRDTHABESTPA
RTOFITIOTHAAWHENI
SIEIWILLBEREBORNI
NPARASICESNSADDTH
AIHAVEKIDDESWILLB
ECOMEMYSLOVESIWIL
DNOTGIVEYOUMYNAME
BECAUSEYOUWIDLTRY
TOSLOISOWNORSTOPM
YCOLLECTINGOFSDAV
ESFORMYOFTERLIFEE
BEORIETEMETHHPITI