Character encoding-based rules with language without encoding

Question

Oftentimes people will ask challenges that have rules hinging on the characters encoded by the source file (see restricted-source). For example, "Your program may not contain the letter a". But some languages are sequences of bytes that are not associated with any character encoding (e.g. machine code or a deflate stream). How should such languages be treated when interpreting this kind of restriction?

Answer 1

Rather than propose a default encoding, I'm going to suggest an alternative route: close the challenge as unclear.

If it's unclear whether or not 65 66 67 contains A, that's the fault of the challenge. Get clarification from the challenge author on what is and isn't allowed.

Answer 2

I propose that the Windows-1252 character encoding be used as the default character set for languages that don't specify otherwise.

Why Windows-1252?

• By far the most character encodings are ASCII-based (extensions of ASCII-7), so if we were to pick a character encoding, it should be one of those. The sooner we get EBCDIC out of the picture, the better.
• Unicode is the go-to character set in these days, but which encoding? Some byte-sequences are invalid UTF-8, ditto for UTF-16 and UTF-32, so neither of these will fly.
• You could pick the first 256 characters in Unicode and be done with it. This character set already exists, and it's called Latin 1, AKA ISO-8859-1. The downside is, this character set includes a whole bunch of non-printable characters that nobody uses (other than those coming from ASCII).
• So, let's look for a superset of (ISO-8859-1 minus C1 control codes). Wikipedia says that the superset of this encoding is Windows-1252. It's also called "ANSI", and while the reason behind it isn't great, the naming certainly did benefit Windows-1252. It still leaves some bytes without an associated character, but there doesn't seem to be a standardized extension that fills them in.

As to why it's named ANSI, i'll let @ais523 explain:

Microsoft's calling of the encoding "ANSI" is a misunderstanding, rather than anything official. They basically have two sets of APIs, one for Unicode, and one for 8-bit character sets. When originally creating the terminology, they assumed that the 8-bit character sets would be ANSI-standardised, and thus used the name ANSI for them collectively, but it turned out that one of Microsoft's own became much more popular.

• The comparison of hex editors on Wikipedia also suggests Windows-1252, besides CP437, as a good choice. The hex editor I use personally uses Windows-1252 as well.
• HTML5 standard specifies that Windows-1252 be used whenever a web page says it's in ISO-8859-1. ISO-8859-1 / Windows-1252 is the most common encoding on the Internet after UTF-8, by a wide margin.¹

Why any encoding at all?

• It makes displaying the code much easier. Also, it makes character-based source restrictions relevant for the language. Which superset of ASCII is used mostly doesn't matter for the purpose of source restrictions.
• As a case study, the question mentions a deflate stream, but a deflate stream isn't a programming language, so I'll assume you meant Bubblegum, which is a proper superset thereof. But Bubblegum does use a charset - it's the same one as is used by the Python 3 interpreter and it becomes relevant when the program's SHA-256 hash is 5e247c455fde7711206ebaa3ad0793114b77a6d16ed0497eff8e3bf98c6dba23.
• The other language mentioned in the question is machine code (assumed x86). Sure, the instruction parser in a CPU couldn't care less how a byte within an instruction looks when you display it on the screen or which character it represents in ASCII, but there is a thing called printable machine code - that is, the subset of machine code restricted to printable ASCII. It becomes relevant when you try to pull off arbitrary code execution on a remote web server.