Count big-ints/symbols/variables as N tokens in atomic-code-golf

Question

The goal of atomic-code-golf was to "level the playing field". However, languages with built-in bigints, symbols/atoms, regex literals etc can easily encode arbitrary programs into a few tokens.

For example, in Python 3 any program can be encoded in 17 tokens using big-int + exec

exec(int.to_bytes(641958435660384548604510984268902312586768556585,20,'big'))
#1  23  45       67                                               89 01234567

In Javascript any string can be extracted from a regexp in 3 tokens:

/this is some huge string/.source
#1                        23

Many languages allow converting an identifier to a string through name lookup or built-in stringification method:

# Python, 20 tokens/string.
try:
    This_is_a_huge_string
except NameError as n:
    print(n.args[0][6:-16])

# Ruby, using symbols, 4 tokens/string.
p :This_is_a_huge_string.to_s

# Ruby, using variable names, 7 tokens/string.
p This_is_a_huge_string rescue $!.name.to_s

// C++, using macros, 7 tokens + 4 tokens/string
#define STR(N) #N
int main() {
    puts(STR(This_is_a_huge_string));
}

And these are actively used in recent atomic-code-golf questions:

• "Atomic" code golf - determine the number of neutrons in an isotope (using big-int and undefined symbols to encode strings)
• Shortest expression for {0, 3, 2, 5, 0, 3, 5, 1, 4, 6, 2, 4} (using big-int to encode a lookup table)

I suggest we close the loophole in the future by modifying the rules:

User-defined Variables (1 point each):

Example: %helloworld%, func_a

Strings (1 point per char)

Example: Hello World is 11 points

to

Non-built-in tokens (1 point per char)

integers, strings, user-defined identifiers, etc.

Example:

• %helloworld% (12 points)
• hello_world (11 points)
• "Hello world" (13 points)
• "\r\n\U00012345" (5 points, each escape count as 1 char)
• /[a-zA-Z]/g (11 points)
• 1234567890 (10 points)
• &He110 (6 points)
• 12.3e-123 (9 points)
• :symbol (7 points)
• %w<Hello world> (15 points)

Variables that are introduced in prelude or external libraries are not counted towards user-defined variables, e.g.

  float radius = Math.sqrt(area) / Math.PI;
//                   11   111111 1 1   22 2
//1     234567 8 9   01   234567 8 9   01 2

Answer 1

Even though originally, atomic-code-golf was created to have a justified counting system across golfing and non-golfing based languages, it has expanded its scope a lot. For example, this xnor's question is about calculating the operations you do in your algorithm. No code at all. Same goes for this question and this and many others.

Moreover, this tag was never about counting by the right version of token in the code, but about counting atomic operations in the code. These atomic operations are defined by the question, thus do not carry a general meaning.

For instance, if the original question which led to this meta post had a scoring criteria of overall operations in the code (splitting, base conversion, indexOf etc), then even though the CJam answer is 6 tokens long, the Python answer is 17 tokens long or the Mathematica answer is 9 tokens long, the actual score would have been almost equivalent and would have depended on the algorithm chosen.

Thus, summarizing things, I leave it up to the question asker to define the scoring terms. If the question says its the number of tokens, let it be and lets count the code by total token count. These tokens are language tokens so things like ., [, ;, ( etc also count.