As Martin already explained, Jelly uses a custom code page that is incompatible with ASCII and UTF-8. This isn't a made-up way of counting bytes, but the Jelly interpreter can take the source code (and input/output, if desired) using this code page. This is actually the default for the official interpreter, which requires the u
flag to process UTF-8 encoded source code.
For the answer in questionthe answer in question, the byte count with Jelly's code page is in fact 28 bytes, and the 28 byte file is functionally equivalent to the UTF-8 encoded 44 byte file.
$ cat nest-utf8.jelly; echo
ḃ2-*µSN;+\>-Ạ
1Ç#Ṫḃ2ṭ2;1ị⁾][
$ wc -c nest-utf8.jelly
44 nest-utf8.jelly
$ xxd -g 1 nest-cpj.jelly
0000000: e7 32 2d 2a 09 53 4e 3b 2b 5c 3e 2d ab 7f 31 0e .2-*.SN;+\>-..1.
0000010: 23 ce e7 32 e0 32 3b 31 d8 8e 5d 5b #..2.2;1..][
$ wc -c nest-cpj.jelly
28 nest-cpj.jelly
$ jelly fun nest-utf8.jelly 8
[[[[]]]]
$ jelly fn nest-cpj.jelly 8
[[[[]]]]
If Jelly was only able to process UTF-8 encoded files, I would have to score (and would have) the answer as 44 bytes. However, since there is a 28 byte file that solves the task at hand, I am able to claim a score of 28 bytes.