13
\$\begingroup\$

I like to ask questions. However, I don't want to run everyone's code on my PC, partly because I only have access to a slow Windows laptop at the moment and I can't imagine I could even run most code in most languages.

Is it acceptable to ask people to include a TIO link in their answers and to use TIO's timer facility?

If so, does anyone have recommendations for how best to do this to make the challenges fair?

Another option is just to ask people to report the largest input size they can solve on their own computers. However some people dislike this as it advantages people with faster computers.

\$\endgroup\$

4 Answers 4

Reset to default
22
\$\begingroup\$

Unfortunately, I don't think TIO is a very reliable way of timing code, even if counting CPU time rather than wall time.

First of all, TIO uses multiple arena servers, where all user-supplied code is run. At point, there are only two arenas and both have identical specs, but even now, one of them is consistently and considerably slower than the other. I have no idea why this happens, but it does.

Also – and this applies to all method of timing code – two processes running at the same time can affect each other in subtle ways that will impact CPU time. E.g., speed of memory access may be affected, especially if the data should already be in the CPU's cache, but was pushed out by another process. Computers also tend to be measurably faster after a restart.

Of course, running code on TIO is still better than having everyone time their code on their own machines. I wouldn't even consider the latter an objective winning criterion for a fastest-code challenge.

\$\endgroup\$
15
  • 3
    \$\begingroup\$ Sorry, I meant having people time their code on their own machines. That's a fastest-computer challenge, not a fastest-code challenge. There's also no way of verifying the scores. \$\endgroup\$
    – Dennis
    Jun 3, 2017 at 15:13
  • \$\begingroup\$ Do you have any other suggestions for how we can run fastest-code challenges? We do have fastest-algorithm but that tends to get mathematical and hence less popular. \$\endgroup\$
    – user9206
    Jun 3, 2017 at 15:13
  • 13
    \$\begingroup\$ Aside from running code on somebody else's computer if you cannot use your own, not really. I'd like to see more fastest-code challenges, but I suspect the problems that come with timing the submissions will put most people off. Maybe it's time for TimeItOnline... \$\endgroup\$
    – Dennis
    Jun 3, 2017 at 15:28
  • 20
    \$\begingroup\$ +1 for TimeItOnline !! \$\endgroup\$
    – user9206
    Jun 3, 2017 at 15:28
  • 4
    \$\begingroup\$ @Dennis And the fix is... fastest-algorithm! \$\endgroup\$
    – Erik the Outgolfer
    Jun 4, 2017 at 20:20
  • 3
    \$\begingroup\$ @EriktheOutgolfer That's hardly a fix. Asymptotic complexity can produce ties between a clearly superior and a clearly inferior algorithm. \$\endgroup\$
    – Dennis
    Jun 4, 2017 at 20:42
  • \$\begingroup\$ @Dennis How about fastest algorithm, with ties broken by speed? \$\endgroup\$
    – Esolanging Fruit
    Jun 5, 2017 at 0:44
  • 3
    \$\begingroup\$ @Challenger5 Then we have to time code and are back at square one. \$\endgroup\$
    – Dennis
    Jun 5, 2017 at 1:05
  • 3
    \$\begingroup\$ Even if TIO were reliable and consistent, doesn't it have a fairly low cap on the execution time? To properly judge fastest-code you need the option to time an execution which runs for between 1 and 10 minutes. \$\endgroup\$
    – Peter Taylor
    Jun 5, 2017 at 10:15
  • 1
    \$\begingroup\$ @PeterTaylor Yes, the time limit is 60 seconds, which is a further limitation. \$\endgroup\$
    – Dennis
    Jun 5, 2017 at 13:31
  • \$\begingroup\$ I'm guessing that not every language can do this, but in an ideal world would counting instructions or execution cycles not be the most accurate way of timing? \$\endgroup\$
    – Pandacoder
    Jun 5, 2017 at 16:16
  • 2
    \$\begingroup\$ @Pandacoder execution cycles aren't everything; managing memory (caches) & concurrency better is often a much bigger win than reducing CPU cycles, once you're past the "big-O" optimisation. \$\endgroup\$
    – Dave
    Jun 5, 2017 at 17:14
  • 1
    \$\begingroup\$ @Pandacode Instruction count is a very unreliable way of approximating speed. On recent x86_64 CPUs, 64-bit divisonis hundreds of times slower than 64-bit addition. \$\endgroup\$
    – Dennis
    Jun 5, 2017 at 17:31
  • \$\begingroup\$ @Dave Ok, that makes sense. And Dennis, that's why I included cycles in my wording, but Dave's response takes into account things that I wasn't even thinking of that would also nix my idea. \$\endgroup\$
    – Pandacoder
    Jun 5, 2017 at 17:33
  • 1
    \$\begingroup\$ @Dennis I would be 100% happy with a TimeItOnline system that had a 60 seconds cap. That seems like a very reasonable solution. \$\endgroup\$
    – user9206
    Jun 6, 2017 at 9:24
12
\$\begingroup\$

Dennis already explained why TIO isn't a great choice. The solutions I've seen people use are:

Cloud providers (e.g. Amazon Web Services' EC2)

Tell people what specs you'll buy, wait a week or so for a batch of answers, then rent one for a few hours to keep the costs tiny (we're talking pence unless you decide your challenge needs immense power). When you're done, delete the instance and move on with your life.

If you've never used AWS before you can even rent one small EC2 instance for free during the first year (but keep an eye on the CPU credits for T2 instances or your challenge will be unfair). I imagine most cloud providers (Google & Azure spring to mind) have a similar promotion.

Cheap computers (e.g. Raspberry Pi)

Buy a common dirt-cheap computer and run submissions on it whenever you want — again, tell people in advance what the hardware specs are.

You don't even need to have a screen / keyboard / mouse: just connect it to a network and SSH in (but don't forget the power lead!)

Both of these options set a level playing field and (theoretically) allow people to run their submissions in the target environment in advance. Both come with some cost, but not much for occasional challenges. Another bonus is that these are real world environments, so through your challenges people may learn something that helps them in their day jobs / uni courses.

\$\endgroup\$
4
  • \$\begingroup\$ Upside to buying a raspberry pi: Now you own a raspberry pi and can make your house a smart house. <sub>Ocarina of time not included</sub> \$\endgroup\$
    – Draco18s no longer trusts SE
    Jun 5, 2017 at 18:13
  • 1
    \$\begingroup\$ @Draco18s Just be careful. IoT devices used in smart homes are especially vulnerable to hacking. \$\endgroup\$
    – mbomb007
    Jun 5, 2017 at 21:20
  • \$\begingroup\$ I don't know how easy it is to get the range of languages people like to use working under these two solution suggestions. \$\endgroup\$
    – user9206
    Jun 6, 2017 at 9:25
  • 4
    \$\begingroup\$ @Lembik most languages people use are very easy to install on Linux, which is the OS of choice for servers and cheap computers, so it should be pretty easy (though of course not as easy as simply running TIO). You may struggle with C# and other windows-based languages (though not so much with their modern versions) and an advantage to the Raspberry Pi is that it actually ships with a free version of Mathematica. Also if your environment is known, the answerers should be able to guide you pretty closely on how to install their language-of-choice (and if anything goes wrong just reformat!) \$\endgroup\$
    – Dave
    Jun 6, 2017 at 17:23
9
\$\begingroup\$

Here's another alternative:

Ask around to find someone willing to do timings.

I'm currently running a challenge where the canonical timings are the ones on my computer. I'd be happy to do the timings for your challenge as well, circumstances permitting. Obviously, I can't personally handle timings for every challenge ever, but I'd guess that there are other people willing to time things. There's no reason that the person who posts the challenge needs to be the one who does the timing.

\$\endgroup\$
5
  • \$\begingroup\$ Thanks very much isaacg! \$\endgroup\$
    – user9206
    Jun 6, 2017 at 7:45
  • \$\begingroup\$ I just tried this with my recent question and no one volunteered to help sadly. \$\endgroup\$
    – user9206
    Jun 15, 2017 at 9:34
  • 1
    \$\begingroup\$ @Lembik Sorry I wasn't available - I was having computer troubles. I'd be free to time things now. \$\endgroup\$
    – isaacg
    Jun 16, 2017 at 2:18
  • \$\begingroup\$ Could use your assistance at codegolf.stackexchange.com/q/175206/31257. See codegolf.stackexchange.com/questions/175206/…. \$\endgroup\$
    – guest271314
    Nov 4, 2018 at 16:58
  • \$\begingroup\$ @guest271314 Seeing as your question is currently on hold, I'll help out if it is off hold. That being said, I'm not too active on the site right now, so someone else might be a better choice. \$\endgroup\$
    – isaacg
    Nov 9, 2018 at 23:49
-1
\$\begingroup\$

I think your best option is to have them include some standard operation that is known to take some time. Then you can compare the time to run their algorithm against the time to run the known time.

They'd have to include code not for their algorithm but if you're golfing for time and not bytes that shouldn't be a problem. There might be an issue trying to find a suitable "clock" but something like "add up a billion randomly generated numbers" comes to mind. (Obviously just the adding, not the generation which varies drastically from language to language...)

\$\endgroup\$

You must log in to answer this question.