Optimal infinite monkeys: output Shakespeare with the highest probability
code-challenge optimization compression
This challenge is something of a sequel to Write Moby Dick, approximately, but the mechanics and scoring system are quite different, and I expect it to lead to quite a different challenge.
We all know that a monkey hammering out random bytes will output the works of Shakespeare with some probability. This probability is extremely low.
But what if the monkey would type out a computer program instead, and then we run the computer program and see if that outputs the works of Shakespeare? The program might itself behave randomly, but in such a way that the probability of outputting Shakespeare is much higher overall.
The essence of this challenge is to maximise the following probability:
p(the monkey outputs your program) * p(your program outputs the works of Shakespeare)
To make that work as a practical challenge we will have to fake the randomness. Your program will not really behave randomly but will output probabilities instead, so that we can calculate the probability that your program outputs the works of Shakespeare, even if that probability is astronomically low.
more details
The following file contains the complete works of Shakespeare in ASCII format. [to do: create the file and upload it somewhere]
Your program is meant to represent a random process that might output the works of Shakespeare with some probability. But depending on how your program works that probability might be so low that it would never practically happen. So instead of using a random number generator, every time your program would output a letter it will output a probability distribution instead. Another program will keep track of the total probability.
It works like this: your program (or function etc.) will be called multiple times (about 3,500,000 times). On each invocation it will be given the first n characters of shakespeare.txt
. Its output will be a probability distribution over ASCII characters, which is its probability of guessing a given next character. This output can be in any reasonable format - for example, it could be a Python array of 128 floats. But it must be a probability distribution, i.e. in this example the floats must sum to 1.
The following pseudocode shows how your score is calculated:
log_p_monkey_outputs_program = -(size of your submission in bytes)*8
log_p_program_outputs_shakespeare = 0
target_text = contents of shakespeare.txt
for n = 1 to length(target_text)-1
probabilities = your_program(first n characters of target_text)
correct_prob = probabilities[(n+1)th character of target_text]
log_p_program_outputs_shakespeare += log2(correct_prob)
score = log_p_monkey_outputs_program + log_p_program_outputs_shakespeare
The score that this program calculates is the logarithm of the probability that the monkey outputs your program and your program then outputs Shakespeare, assuming that we always feed the program's output back in as input. We calculate the logarithm to avoid floating point errors, as the final probability will be extremely small.
Note that the logarithm is to base 2. If your language doesn't provide the log2
function you should use log(correct_prob)/log(2)
.
If the scoring program is implemented correctly, the score will always be negative. A higher score (closer to 0) is better.
Note that your program outputs a probability distribution but it should not itself behave randomly. Your program may not use a random number generator - it must always return the same probability distribution for a given input.
If you want to store state in between invocations this is allowed. You can do this by writing to an external file, by using static
or global variables, by submitting a class rather than a function, using a state monad, or whatever else works for your language.
Submission format
Your submission should include the following, which do count towards the size of your submission. If they are excessively large you can link to github etc.
- your program
- any data it needs in order to run
Your submission should also include the following, which don't count towards its size:
- the code used to calculate its score, implementing the pseudocode above (this doesn't need to be in the same language as your submission). Please don't golf this.
- any code that was used to generate your submission (e.g. to create any data files that you included)
- an explanation of how your submission works.
Rules
As mentioned, your program must run deterministically, so that it always outputs the same probability distribution given the same input (and hence always gets the same score).
If at any time the value of correct_prob
in the scoring pseudocode is 0, then your score is -∞, which is the worst possible score.
You may not use any libraries or functions that your language might have that include data or statistics about natural language. This includes pre-trained neural networks, word lists, etc. It also includes any built-in function that outputs any of Shakespeare's works. It's fine to use neural networks and word lists etc., but the data or weights must be included in your submission and count towards its byte count.
You may not use any libraries or functions designed for compressing or decompressing data. It's fine to use algorithms like bzip etc., but you have to implement them yourself (and hence include the implementation in your byte count).
If you want to store state between invocations you can do this however you like, as long as your program never has access to 'future' bytes from the shakespeare.txt
file. (So, for example, you can't just pass it a string containing all of the input and get back a big list of probability distributions as output.)
You must actually run your test program and calculate/verify your score before submitting your entry. If your submission runs too slowly for you to verify its score then it is not qualified to compete, even if you know what its score would be in principle.
You may import existing libraries other than the exceptions above, but you may not load any other external files unless they're included in your byte count. Your code may not access the shakespeare.txt
file in any way other than described above.
A note about the scoring system
In the sandbox, several people had an intuition that the optimal answers to this challenge will take the form of compressing the shakespeare.txt
file and outputting it deterministically. However, the scoring system has been carefully chosen so that this strategy will not be optimal. If you have a stochastic solution then in theory you can turn it into a perfect deterministic solution, at the cost of adding -log2(p(program outputs shakespeare))
bits of code into your program. This extra code decreases p(monkey outputs program)
by exactly the same factor that it increases p(program outputs shakespeare)
, resulting in zero change to the overall score.
But this is the absolute best you can do. Any real encoding will be less than perfect, so in reality the net result of such a strategy will be to slightly decrease the score. Because of this, it's likely that the most competitive answers will be stochastic. However, deterministic, compression-based answers are permitted and are welcome to complete, as long as they meet the spec.
sandbox notes:
On the off chance that I'm wrong about the properties of the scoring system, I intend to battle-test this challenge before I post it. To do this I will implement (i) the obvious illegal solution using bzip2 to compress the whole text, and (ii) the not-quite-so-obvious illegal solution using bzip2 to stochastically predict the next character from the ones already received, without including a compressed version of the text in the code. I expect solution (ii) to beat solution (i), for the reasons described in the last section of the question. If it doesn't I will either give up on the challenge or add a fudge factor into the scoring system to give slightly more reward to stochastic solutions.
I'll also post a sample answer (legal but not very competitive), in order to address the point about the spec being a bit daunting to understand.
I'm unsure about the rule banning built-in compression algorithms. It seems more elegant to leave it out, and it would still be a meaningful challenge without that rule. But in Paint Starry Night, objectively, in 1kB of code off-the-shelf compression spoiled the fun a bit, and I'm worried that with this scoring system the same could happen here. I'm happy to hear any thoughts about that.
I'd also really like feedback on the score calculation pseudocode - is it sufficiently clear how the score is calculated, and can I make it clearer?
Finally, a very specific query: the ban on word tables seems like it would rule out some golfing languages. I don't want to rule out any languages but don't want to open the floodgates either, so if someone can think of a good middle ground it would be helpful.