To compare solutions on performance, we should form a kind of template, what to specify or how to specify the question.
I realized, when trying to solve fast line drawing algorithm, that it is not clearly specified:
How are the results compared? What is it, what shall be compared? How fast a line is drawn, but we know, that because of caches and cache-size and cache-misses, a solution which is fast with 10 000 elements might be slow with ten millions.
The size of source (code-golf) can be compared more easily², but if I look at a bunch of c/python/ruby-code I can't estimate how many lines it might draw per second or minute. Not roughly, and not even for languages which I use on a daily basis.
Possible approaches:
A) Let chaos rule. Every question shall choose its own way.
B) Let every answer specify machine characteristics and use its own, private timing policy.
- User A: Pentium III 800 Mhz, 1000 runs / 0.4 s
- User B: 3 dual-cores @ 3Ghz, 1s: 23 786 000 runs
C) Let the quizz master fix the settings, for a minimal comparision, and the users have to specify their machine:
- How many solutions in 30s
- How many time for 1 Mio. solutions
D) Let the quizz master fix the settings, for a minimal comparision, and the users have to provide ready-to-run code/applications, not just functions/snippets.
- Full source code A.cc, B.Java, C.scala
- How to run it:
g++ A.cc -lm -o A && time A
javac B.java; java B
# internal timing
scalac C.scala && time scala C 1000000
# specify problem sizeE) Let the users use IDEONE or something similar, to have it executed on the same machine. Specify number of solutions or time to run. Drawback: Not every language is available.
I'm not sure how to handle the problem best, and I know, that most users go online, without discussing their riddle in chat - maybe in a unjustified fear, that somebody else will copy it? However; if we have something in the faq, or it is easy to find on meta, we can often provide links 'how to ask performance tasks' so some people will use it from the beginning.
If you worked on your solution and meanwhile the asker decides, how to compare the resulst, you might have to do doubled work.
I think we should look for a ruleset which will fit 90% of the performance questions, and be fine for as much languages as possible. It should be short and easy to explain.
Pleas argue for one of the ideas A-E, argue for a modification like B' or C'', or introduce your solution F-Z.
²) Don't discuss it here: Can C/C++-Users and the like substract their main-boilerplate or not - this might be argued about, but we can at least estimate, what the substraction or addition of a main method would mean in terms of character costs. Or include-files - we know, we know.