23
\$\begingroup\$

In code-golf questions, we typically score submissions by their byte count. However, whenever a submission uses a character outside of the ASCII range, the question of which encoding the code uses comes up.

One language for which this is quite relevant is APL, due to its use of many characters outside of ASCII and which have multiple-byte UTF-8 encodings. It is also relevant in many other languages.

As far as I can tell, we don't have a policy on how to handle this currently.

Here are some possible ways to handle it:

  • Everything is UTF-8.

  • Languages may have specific encodings. If there is no standard encoding for a given language, use UTF-8.

  • Submissions may have specific encodings. If none is given, a language-wide default applies. If the language has no default, UTF-8.

I'm sure there are other possibilities, and I would be interested in hearing proposals. I do believe this is an issue we need a firm policy on, because this question seems to come up every single time someone writes an answer in APL, and reasonably often on other answers as well.

\$\endgroup\$
1
  • 4
    \$\begingroup\$ "As far as I can tell, we don't have a policy on how to handle this currently." That's not entirely correct. The code golf tag wiki says "Unless the question is specified to be scored by characters, it is scored by bytes. If it doesn't specify a character encoding to use for scoring, answers which use Unicode code points outside 0 to 255 should state the encoding used." \$\endgroup\$ Sep 3, 2015 at 11:57

3 Answers 3

27
\$\begingroup\$

Answer, else language, else UTF-8

I prefer the system where an answer may specify its encoding, or default to its langauge's standard encoding, and a language's standard encoding is assumed to be UTF-8 unless specified elsewhere.

This has a couple of nice properties:

  • An answerer can use whatever encoding they want for a particular answer.

  • Newer users can do the 'obvious thing', such as count characters in APL, and still be correct almost all of the time.

  • Every answer has a well-defined byte count.

The one major drawback is the need to maintain a list of language-by-language defaults, but there are only a small number of commonly used languages that don't use UTF-8 (APL uses the APL code page, Pyth uses ISO-8859-1, etc.), so it shouldn't be that hard.

EDIT: To clarify something with regards to @Ypnypn's comment: Answer-defined encodings would be subject to the same standard loopholes as programming languages, namely that the encoding must have been specified before the question was asked, and specifying an enormous list of encodings, in the style of MetaGolfscript, is not allowed either.

\$\endgroup\$
7
  • 2
    \$\begingroup\$ Personally I think giving individual answers the opportunity to use any encoding opens up a can of worms. One example is Antiferromagnetic Ordering, where many folks used unicode up- and down-arrows, which are 3 bytes each. These are also part of the APL code page, and are thus 1 byte each for APL. Per this meta answer, answers to that challenge may try to claim the EBCDIC-based APL encoding when in fact it's unnatural for their language to do so. \$\endgroup\$
    – Alex A.
    Sep 3, 2015 at 16:08
  • 12
    \$\begingroup\$ @AlexA. Is that a problem? \$\endgroup\$ Sep 4, 2015 at 14:15
  • 3
    \$\begingroup\$ So I can make a custom encoding for each answer, which "happens" to use 0.5 bytes for each character in the solution? -1. \$\endgroup\$
    – Ypnypn
    Sep 6, 2015 at 19:19
  • 4
    \$\begingroup\$ @Ypnypn no, you can use any existing encoding, (the same for languages: you can use any existing language) \$\endgroup\$
    – edc65
    Sep 6, 2015 at 19:33
  • 2
    \$\begingroup\$ I like the idea of being able to specify the preferred encoding. I'd especially like to use UTF-21 to encode text that primarily consist of non-BMP code points (which would otherwise be 4 bytes per character under both UTF-8 and UTF-16). ;-) \$\endgroup\$ Sep 7, 2015 at 3:50
  • \$\begingroup\$ This could be used for cheating, if the scored is measured in bytes. \$\endgroup\$
    – univalence
    Jul 17, 2016 at 11:29
  • 3
    \$\begingroup\$ You might want to mention explicitly that there must actually exist an interpreter/compiler which understands the source file in the chosen encoding (so you can't score your answer as ISO 8859-1 if the interpreter chokes on anything that isn't valid UTF-8). \$\endgroup\$ Sep 28, 2016 at 20:35
4
\$\begingroup\$

Standard language encoding, else UTF-8

Answers should use the most appropriate encoding for the language, not whatever is convenient for the answer. If no standard encoding for the language exists, the default should be UTF-8.

We could compile a list of languages that require a specific encoding and have that list on meta for reference. If the language is not on the list, assume UTF-8.

\$\endgroup\$
3
  • \$\begingroup\$ I'd be OK with this. \$\endgroup\$
    – isaacg
    Sep 3, 2015 at 16:31
  • 3
    \$\begingroup\$ It's fairly common practice (especially in kolmogorov complexity, or other challenges that require compression) to use extended ASCII (primarily ISO 8859-1), which means that you get the full 256 possible values for each byte. This answer would disallow that for most languages, which would have to use multi-byte characters for every value outside (7-bit) ASCII. \$\endgroup\$ Sep 4, 2015 at 14:15
  • 5
    \$\begingroup\$ I dislike this answer, because there are many languages that can use multiple character encodings (Python is one), and it doesn't clearly specify whether or not that would be valid. \$\endgroup\$
    – user45941
    Jun 24, 2016 at 4:54
-3
\$\begingroup\$

Burden of demonstration

[edited]

Allowing charsets that are actually accepted in practice seems entirely reasonable, with some variant of Unicode as a fallback.

However I'm concerned that someone unfamiliar with the language in question may find it difficult to confirming the existence of a custom code page, or more importantly, to prove the non-existence of one.

If a language has a more efficient encoding than UTF-8 or UCS-2, it would be help to at least mention this in an answer, rather than assume that everyone else knows which encodings are accepted by some compiler or interpreter. (A link to a code page would be helpful but need not be a formal requirement.)

This should be only a trivial impost on the claimant, compared with the effort by a reader to prove or disprove a tacit assumption of a codepage.

Why am I raising this?

I've recently noticed a few answers that consist almost entirely of non-ASCII characters. For example in #266049 How many umbrellas to cover the beach? there are answers in Charcoal and 05AB1E claiming 26 and 13 bytes respectively.

When I queried the byte count for the 05AB1E entry, there was a response that cited a custom code page, so that would seem to be valid, but it doesn't seem reasonable that a casual reader should be expected to know this.

As far as I can see Charcoal is built atop Python 3, which means it implicitly assumes UTF-8, which means the claim of 22 bytes is unsupportable by any of the metrics in this page. (OK, it turns out Charcoal does have a code page, despite me looking and not finding one.)

Other thoughts

Ideally it would be obvious where to find the site FAQ, but when viewing a challenge and its submissions, I don't see anything that even suggests "site FAQ" or "site policy" documents, nor do I see anything referring to this "meta" site. (Maybe "meta" is some kind of standard thing on StackExchange, but if so, that in turn needs to be made clearer, by StackExchange itself.)

Is there even a FAQ document, or are these meta questions the only approximation of such a resource?

\$\endgroup\$
4
  • 2
    \$\begingroup\$ Charcoal also has a code page. Regardless, it's usually fairly easy to find the code page, and even if one can't find it a comment of "in what encoding is this?" suffices, so I don't think we need to require that all SBCS answers have a link to their code page. \$\endgroup\$ Oct 20, 2023 at 4:41
  • \$\begingroup\$ Sorry, I hit "save" before I finished. If the code page exists then it's usually not too hard to find. But if no code page exists, verifying that is quite a large job. At least if a code page is named then we know what to search for. Even an informal description such as "the usual APL encoding" would be enough of a hint that it exists that I know it would be worth looking for. \$\endgroup\$ Oct 20, 2023 at 6:43
  • \$\begingroup\$ Point taken about simply asking in a comment; I'll do that in future. \$\endgroup\$ Oct 20, 2023 at 6:47
  • \$\begingroup\$ The main page has a link to Welcome to Code Golf and Coding Challenges Stack Exchange!, which has a link to the FAQ, which has a link to How to count bytes FAQ, which has a link to this question. I agree that it should probably be more visible. \$\endgroup\$ Oct 22, 2023 at 2:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .