This is currently WIP, but please post feedback if you have any!
Best General-Purpose Compression
Abstract
This paragraph is meant to summarize, the binding rules are below.
Your task is to implement the best general purpose compression algorithm in the context of this challenge. You will be given a file corpus which contains a mix of different file types. The goal is to beat an existing answer in terms of overall size. The size is determined by the size of the compiled compression program (and decompression program if they are two different programs) plus the compressed size of the corpus. Your algorithm must beat standard 7z with default settings by at least 10 bytes, or if there is an existing answer, this answer by 10 bytes.
Corpus
This paragraph lists all files that are part of the input to your program, their size and the baseline 7z score in 7z 9.38 beta
. Licenses are linked where needed. Disclaimer: There is no political or humorous motivation behind the choices. All files were chosen because of their representation of certain formats or unique compression behavior.
Natural Language
#1 - ASCII - Bibliography of Computer Security Articles 1983-88: DownloadSave-As
Uncompressed: 146 119
7z 9.38 beta: 17 142
File name: articles.law
#2 - ASCII - Pre-Flight Launch Preparation for the Space Shuttle: DownloadSave-As
Uncompressed: 44 515
7z 9.38 beta: 15 304
File name: 6_2_4_4_2.TXT
#3 - ASCII - 1984 News report archive about the Bundespost Hack: DownloadSave-As (transcribed by Chaos Computer Club, cites full-text articles from Die ZEIT, taz et al.)
Uncompressed: 102 242
7z 9.38 beta: 32 382
File name: boh-20f8.txt
#4 - ASCII - (Unofficial historical transcript of) His Last Bow by Arthur Conan Doyle: DownloadSave-As (lic. Public Domain)
Uncompressed: 330 042
7z 9.38 beta: 110 481
File name: doyle-his-382.txt
#5 - UTF8 - Transcript of the Rosetta Stone translated to English: DownloadRAW->Save-As (lic./© The Nile, Notes for Travelers in Egypt, by E. A. Wallis Budge, 9th Edition, London, Thos. Cook and Son, [1905], pp. 199-211)
Uncompressed: 18 270
7z 9.38 beta: 6 299
File name: Rosetta_EN_UTF8+BOM.txt
Source Code (+ Markdown)
#1 - C - CPU control code taken from the linux kernel: DownloadRAW->Save-As (lic. GPL, Rusty Russell)
Uncompressed: 18 866
7z 9.38 beta: 5 683
File name: cpu.c
#2 - QBASIC - COMBAT ARENA (Game): DownloadRAW->Save-As (© Jeff Copperthite)
Uncompressed: 59 792
7z 9.38 beta: 11 921
File name: COMBAT.BAS
#3 - XML - The 9/11 Wikimedia dump: Download (lic. GFDL, CC-BY-SA 3.0)
Uncompressed: 40 557 621
7z 9.38 beta: 578 993
File name: sep11wiki-20071116-pages-meta-history.xml
#4 - C# - Source of the .NET Framework System.Array (CLR): DownloadRAW->Save-As (lic. MIT)
Uncompressed: 133 646
7z 9.38 beta: 15 172
File name: Array.cs
#5 - JavaScript - jQuery 1.11.3: DownloadRAW->Save-As (lic. MIT)
Uncompressed: 284 394
7z 9.38 beta: 73 585
File name: jquery-1.11.3.js
Binary Media
#1 - JPG - "Flower in the Garden": Download (lic. CC-BY-NC-ND 2.0 & © wellenkern)
Uncompressed: 10 482 521
7z 9.38 beta: 10 434 631
File name: flower.jpg
#2 - MP3 - "Mariam's Cake" by Robin Grey: Download (lic. CC-BY-NC-SA 3.0)
Uncompressed: 8 468 990
7z 9.38 beta: 8 162 146
File name: cake.mp3
#3 - Bitmap24 - DP1M7218 16:9 COLOR: Download (lic. CC-BY-SA 3.0 & © iKobe!)
Uncompressed: 37 834 326
7z 9.38 beta: 14 652 746
File name: street.bmp
#4 - MP4 - "Home 5x5": Download (lic. CC-BY 3.0 & © Michael Jones)
Uncompressed: 6 173 898
7z 9.38 beta: 6 013 047
File name: home.mp4
#5 - binary - Block of cryptographically secure random data, a.k.a. "The Incompressible File": Download
Uncompressed: 100 000
7z 9.38 beta: 100 129
File name: rnd.bin
Rules
(De/)Compression Program and Algorithm
- The program used to compress and decompress a file is the compiled executable (or executable script).
- You may use two or more programs and ship additional files if needed, however only one program per step (a step is de- or compression) is executed.
- No program or part of a program is allowed to request resources that are not included in your distribution. Exceptions are standard system resources (clean install, i.e. a Java runtime would not be available when you use Windows).
- You may use linux-, windows- or architecture-specific code.
- You must not modify files or file names. The file names listed above are binding. Note: Some downloads may not carry the file name, you have to rename them (case-sensitive). The file names of compressed files do not matter. However, the original file names must be restored after decompression.
- You can use any existing or new algorithm. Content mixing is of course allowed, however, you may not call any compression routines available on your system (or language). You program must be a self-contained implementation.
- Your algorithm must not specifically target this corpus, it has to be general purpose. You are however allowed to target the used file types (e.g. by identifying the type via "Magic Bytes" and adjusting the algorithm accordingly).
- The (de)compression program only accepts one argument, a file name. Piping program output somewhere or providing anything else is forbidden. (I.e. you'll have to write the output file yourself).
Running your entry
Your compression program is called 15 times, once for every file in the corpus:
comp articles.law
comp ...
This process must produce 15 compressed files on disk in the same directory (original files can be overwritten). (comp is just an example name, you don't have to use this name)
Decompression is similar. Your decompression program (which may be the same executable) is called 15 times, once for every file in the corpus:
decomp ...
Where ...
is the file name of the compressed file. This will produce the original 15 files on disk (with their original file names) (compare the SHA256 when in doubt).
Neither the compression or decompression program may take any additional command line parameters.
Scoring
Your score is P+C
, where
P
is the size of all programs used in your distribution
C
is 15 files, the compressed corpus
in bytes.
Your answer must include the following:
- The source code for all parts of
P
. (Is not counted in the score).
- A link to a GitHub repository containing both
P
and C
. Means you actually have to run your solution and include all compressed files. If your repository features additional files that should not be included in your score (gitignore, README etc.), put the scored files in a separate folder.
To beat someones score, your score has to beat it by at least 10 bytes. If some answer has the score X
, then your answer has to have a score of N <= X-10
, where N
is your score. The starting score is set by 7z.
Invalid Answers
Please refrain from posting invalid answers, such as:
- Answers without source code for
P
,
- answers without compressed files, or where compressed files run against
P
produce anything but the original corpus,
- answers that contain only partial solutions.
For clarity, here's how an answer might look like:
I did this algorithm X. This code is used for compression:
<comp.c>
and this for decompression:
<decomp.c>
I used windows, my score therefore is 9001:
comp.exe: 234 bytes
decomp.exe: 124 bytes
Compressed files: 8643 bytes
You can download the programs and compressed files from my
repository here: github.com/foobar42.
Leaderbaord and History
Current score: 423,424 (P) + 40,229,661 (C) = 40,662,085
Set by: 7z 9.38 beta (Win32) on 2015-12-18
code-challenge compression