Why languages encode their SBCS into UTF-8 and vice versa?

Question

I saw that in many golfing programming languages with a SBCS other than UTF-8 there is a converter (encoding and decoding) that converts the code/file from the SBCS defined characters to UTF-8 and vice versa.

What is the point of having the UTF-8 representation of the code bytes? My only guess is to be able to save the code in 1-byte characters with known charset for golfing score evaluating purposes. However, if it for saving the program in UTF-8 encoding, isn't control characters like EOF can impact on reading back the file?

Plus, golfing-wise, does all challenges consider SBCS as valid byte-based scoring? or there are challenges that consider only UTF-8 as a 1 byte?

Thanks.

Answer 1

The UTF-8 representation allows the code to be human-readable (to the extent that this is possible for a dedicated golfing language) while being able to be scored by "number of characters". Since our default is to count bytes, golfing languages have an incentive to use all 256 bytes. Having a printable representation for each of them is just convenient. Note that the interpreter still has to correctly run the corresponding byte-encoded program, and that program is what we're actually scoring.

More technically, here's how a typical golfing language uses its SBCS, and how TIO and this site accommodate it:

• The interpreter has one internal representation for each of the 256 bytes.
• The interpreter has (at least) two modes: it can take the program as a bytestring and parse each byte as its internal representation, or it can parse a UTF-8 string of characters on its SBCS and parse each character like the internal representation of the corresponding byte. Apart from this part of the parsing process, the modes should be functionally identical.
• TIO displays and (as far as I know) runs the program as a UTF-8 string in UTF-8 mode, but displays its score as if it used the equivalent bytestring program in bytestring mode. We do the same. It's generally understood that the UTF-8 program represents the bytestring program. This is a compromise between correctness (the UTF-8 string itself is not what's scored) and convenience (no-one wants to read already cryptic programs as hexdumps).