4
\$\begingroup\$

Is there an online tool that will tell me the number of bytes in a UTF-8 character? Or, if not, how do I determine this?

I'm currently trying to determine the number of the bytes for the left and right floor characters:

⌊ (\[LeftFloor]) = U+230A https://unicode-table.com/en/230A/

⌋ (\[RightFloor]) = U+230B https://unicode-table.com/en/230B/

I looked through this FAQ, but was unable to find an answer.

Update

Thanks for the answers! I tried the various byte counters, and I ran into two issues:

(1) I found that one program, Mego's Byte Counter, gives a different UTF-8 value from the other three for the italicized pi (see first screenshot). Either (as seems more likely) Mego's is incorrect, or the other three all are. Whichever of these is incorrect, they may have other errors as well.

Here are my test glyphs:

glyph 1: 𝜋 (italicized pi)

glyph 2: π (plain-text pi)

For glyph 1 (𝜋), Mego's Byte Counter reports six bytes, while the others say four:

enter image description here

For glyph 2 (π), all report two bytes:

enter image description here

(2) With some glyphs, like those for pi, it's pretty easy to tell when you aren't using the lowest-byte-count version of the glyph (you want the plain-text rather than the italicized version). But what if you're dealing with ones that aren't so obvious, like arrows and other more unusual symbols? Are there cases where you might, say, use an arrow and get x bytes, not realizing there's another version of an arrow that is x-1 bytes? If so, is there a good online listing of minimal-UTF-8-byte-count versions of each type of glyph?

Now you might ask: Why not just copy the glyph from the front end of the program you're using (in my case, Mathematica), since that way you'll be able to determine the byte count of the actual glyph the program uses. The problem is that, when I do that, I don't get the glyph:

enter image description here

\$\endgroup\$
8
  • 1
    \$\begingroup\$ Try It Online can do this. Doesn't matter what language, just paste it into the box and it'll show you the byte count. \$\endgroup\$
    – Rydwolf Programs
    Jun 16, 2021 at 2:50
  • \$\begingroup\$ @RedwolfPrograms Well, that was easy enough! Perhaps that should be in the Intro section for new members. \$\endgroup\$
    – theorist
    Jun 16, 2021 at 2:55
  • \$\begingroup\$ Note that, as implied by the IBM article you linked, Unicode characters are not inherently tied to a specific number of bytes. Each character can be encoded in various ways, taking various amounts of bits. E.g. ⌊ can be encoded as 0000101000100011 in UTF-16 and as 111000101000110010001010 in UTF-8. \$\endgroup\$
    – Adám
    Jun 16, 2021 at 5:00
  • \$\begingroup\$ @Adám Yes, of course, since it says just that in the article I linked. But I assumed it was understood I was referring to UTF-8, since I thought CodeGolf's byte count was based on UTF-8. Is that not the case? It sounds like I need to be more explicit and specify that in my post. \$\endgroup\$
    – theorist
    Jun 16, 2021 at 5:14
  • \$\begingroup\$ @theorist No, that isn't the case. Answerers can choose any encoding that their interpreter/transpiler/compiler can handle. Only in the absence of a stated encoding do we assume UTF-8. This is per meta consensus. \$\endgroup\$
    – Adám
    Jun 16, 2021 at 5:16
  • \$\begingroup\$ @Adám Are you saying that in some cases an interpreter/transpiler/compiler might accept Unicode characters, but only with UTF-16 or UTF-32 encoding, and thus the answerer would be requried to use a UTF-16 or UTF-32 byte count in summing the length of their program? Or if not, what would be a case in which someone would choose to count bytes for UTF characters using a basis other than UTF-8? I.e., my understanding was that, while the byte count may not always be UTF-8, the UTF byte count is always UTF-8. Is that not correct? \$\endgroup\$
    – theorist
    Jun 16, 2021 at 5:26
  • 1
    \$\begingroup\$ Yes, that is exactly what I'm saying. E.g. as stated here, the NARS2000 interpreter (for APL) only handles UCS-2, and thus even ASCII characters are 2 bytes each. If an option, one might choose to use UTF-16 over UTF-8 if e.g. ones code consists mainly of characters that take 2 bytes each in UTF-16, but 3 bytes each in UTF-8. \$\endgroup\$
    – Adám
    Jun 16, 2021 at 5:29
  • \$\begingroup\$ @Adám Got it, thanks for the explanation. I edited my question to change Unicode to UTF-8. \$\endgroup\$
    – theorist
    Jun 16, 2021 at 5:31

3 Answers 3

Reset to default
3
\$\begingroup\$

TIO includes a byte counter. For languages with an SBCS, this just counts the number of characters, but for most languages, you'll notice they use UTF-8. For example, if you go to the Python 3 TIO, in the top-right corner, it says "UTF-8".

You can simply paste your code into the code box for any language that uses UTF-8 encoding and it'll give you the character and byte count in that encoding.

Just be careful not to use a language with an SBCS - that'll just give the character count as the byte count.

\$\endgroup\$
3
\$\begingroup\$

I've always used this byte counter which in fact gives you both character and byte counts.

\$\endgroup\$
3
\$\begingroup\$

Here a couple of counters made by CGCC community members:

  • Byte Counter by Mego (account deleted)

    Outputs all (supported) code pages where the inputted text is a valid encoding. Outputs the length and the list of bytes in hex/ The current list of supported encodings are:

    supported_pages = [
     'ascii', 'cp037', 'cp1006', 'cp1026', 'cp1140', 'cp1250',
     'cp1251', 'cp1252', 'cp1253', 'cp1254', 'cp1255', 'cp1256', 'cp1257',
     'cp1258', 'cp424', 'cp437', 'cp500', 'cp720', 'cp737', 'cp775', 'cp850',
     'cp852', 'cp855', 'cp856', 'cp857', 'cp858', 'cp860', 'cp861', 'cp862',
     'cp863', 'cp864', 'cp865', 'cp866', 'cp869', 'cp874', 'cp875', 'iso8859_1',
     'iso8859_10', 'iso8859_13', 'iso8859_14', 'iso8859_15', 'iso8859_16',
     'iso8859_2', 'iso8859_3', 'iso8859_4', 'iso8859_5', 'iso8859_6', 'iso8859_7',
     'iso8859_8', 'iso8859_9', 'jelly', 'koi8_r', 'koi8_u', 'mac_cyrillic',
     'mac_greek', 'mac_iceland', 'mac_latin2', 'mac_roman', 'mac_turkish',
     'ptcp154'
    ]

    Source

  • bytes by ETHproductions

    Supports 20 different encodings, able to be chosen from a dropdown menu. Has permalinks and a bookmarklet available. Currently supports UTF-8, UTF-16, ISO-8859-1, CP437, ASCII and more

Source

\$\endgroup\$
1
  • \$\begingroup\$ Mego's Byte Counter may have an error for the italicized version of pi (𝜋). It says it's six UTF-8 bytes, while TIO, ETHproductions' bytes, and the mothereff.in byte counter all say four (see screenshots added to my OP). \$\endgroup\$
    – theorist
    Jun 21, 2021 at 3:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .