Should submissions be allowed to exit with an error?

Question

Suppose we have the following Python code:

f=lambda x:x*2
X

The first line defines a function which doubles a number. Then the second line is reached, is found to be invalid, and the program exits with an error.

Similarly, another case is:

print(1)
X

which prints the number 1, then exits with the same error.

One might argue that these sorts of submissions are valid because:

• In the first case the function f can still be used, in the sense that f(2) can be called to give 4 in the REPL that pops up (when using IDLE at least)
• The output to STDOUT is only 1 in the second case, as the error is written to STDERR

So I'd like to get the community's opinion on this: Are either of these types of submissions allowed? I'm asking because I've seen this used multiple times, most recently here.

Answer 1

There are a few different issues and cases to be considered here. I think most of what I wrote below is based on the unwritten rules we're currently applying (as far as I'm aware of them). But a few things are also my own opinion, so let the votes show what you think.

Ignoring STDERR

As far as I know, this is common practice. What goes to STDERR also depends on error/warning levels and such, so I guess it's easiest to just say that STDERR can be ignored completely, as long as STDOUT has the required output (or the required side effects happen or whatever the challenge asks for).

Throwing Errors and Exceptions

I think this really depends on whether the submissions is a full program, REPL snippet or a function, and in the latter case whether the error happens during function definition or function call.

Full Programs

I think terminating with an error or an uncaught exception is fine here, as long as it doesn't produce stray output to STDOUT. Under the premise that we're ignoring STDERR, a program terminating with an error still does the job.

REPL Snippets

As long as these don't break stuff so badly that you have to restart the REPL, I think they should be treated the same as full programs. If they produce the right output or side effects, barring STDERR, aborting with an error or exception is fine.

Errors during Function Definition

(By this I mean "at the time the function is defined". I.e. if the code snippet provided in the submission throws an error before ever calling the function.)

This is the example mentioned in the question. I would say this depends on whether REPL snippets are valid submissions to the challenge (which they are not by default).

• If REPL snippets are valid, I would treat the function definition just as a REPL snippet and the above rules apply.
• If REPL snippets are not valid (the default), I think this should not be allowed. If an error is thrown during function definition, this renders the function useless because it can never be called. If an exception is thrown that could be caught, the necessary try-catch block has to be included in the byte count.

Errors during Function Call

Again, if the challenge also allows REPL snippets, the REPL rules should apply.

Otherwise, I'm not sure. I would say it depends on whether the error would only abort the function call or the program. If it only aborts the function call that's fine. If it causes an error that aborts the entire program, or it throws an exception (even if it could be caught!), then I think this should not be allowed. If people answer with functions, they should take into account that the purpose of functions is to reuse them without restarting the program. If you really want to throw an error, answer with a full program.

---

In any case, answers should state explicitly if they are throwing errors. It's part of their interface, and we usually expect good answers to state I/O details as well.

And, as usual, any challenge author may override any of these defaults if they so wish.

Answer 2

Let me simplify this a bit.

They can exit with errors after the result is printed (or outputted in other ways)

Error messages in STDERR or those can be disabled in configurations can always be ignored. We are talking about the fatal errors which exits the program.

For function definitions, there must be a way to use the function, and retrieve the returned value (if specified in the question) without extra code to rescue from the error. However if the function does not need to return something specific, there is no reason to disallow errors in the function call as we already seemed to be allowing side effects.

For REPL it is similar. As long as the error does not stop the REPL from printing (or otherwise using) the result (if the question requires it to return something), it is fine.

If the function or REPL command printed something itself, it must really printed, not only into the internal buffer when it exited with an error.

Then there are run-time and syntax errors.

If we allow syntax errors, we'll see bash scripts exiting with ). But that isn't too useful anyway since it cannot appear in a if or while group command. And it's not easy to define syntax errors in every language. I think we should just allow them.

Most source-layout questions should explicitly state that the program should exit with no errors, which usually means having the exit code 0.

Related problem: a Windows Batch file can exit the current program (using goto :eof), or the whole cmd.exe (using exit). Both should be fine.

Answer 3

Error messages

Here I agree with Martin that programs which produce output on both standard output and standard error streams should be judged solely on the basis of standard output. However, if it is not possible to separate the streams via redirection, the program should be considered incorrect.

Types of errors

I define two types of errors:

1. Recoverable errors

These are errors which the program can recover from within the bounds of standard language semantics and continue execution from the point the function was called. The most common form is an exception which may be caught by a try/catch clause. Another is a warning which causes a message to be written to standard error, but never interrupts program flow.

2. Fatal errors

A fatal error cannot be stopped by standard exception handling mechanisms. Often they generate a signal. Catching an error with a signal handler would not an acceptable method of recovering an error. Other instances of errors may not cause a crash, but are still fatal if there is no way to catch them.

Valid Program Forms

1. Full Program

A full program should take its input from an external sources such as standard input, command line arguments, or files. It should write the results to standard output.

Programs should be allowed to make either kind of error. They should be judged only on the contents of standard output.

2. Functions

Functions take their arguments from within a program. Their output might be returned from the function call, written to a memory area provided by the caller, or printed to standard output.

Errors may result at two different times from functions: (I) at definition, or (II) at execution.

(I) Errors during definition

I divide function definitions into three further cases:

a. Code that defines a named function when compiled

This kind of function should not generate any Type I errors, i.e. it should compile successfully.

b. Code that defines a named function when executed

This type of code may generate recoverable errors.

c. An unnamed expression that evaluates to a function (compiled or executed)

The code (if interpreted) should evaluate successfully.

(II) Errors during execution

During executions, functions should be allowed to make recoverable errors, but not fatal errors. It should be possible for a caller to continue execution after calling the function without measures more extreme than a try/catch block. However, it is normally not possible for a function to exit with an error if it also needs to return a value.