Challenges about interpreting arbitrary invented languages

Question

This is a discussion about this challenge:

Make an interpreter for my new esolang InSpace

It creates a new language and asks for an interpreter for it. It was originally closed as unclear because OP did not specify many things well. OP has now clarified it so that it is clear what it is asking for.

In addition, for those who can see deleted posts (2k users until we get a design), there are two questions about interpreters that I've posted that were deleted, one of which was closed and the other which I deleted too quickly:

https://codegolf.stackexchange.com/questions/76995/write-an-interpreter-for-my-new-esoteric-programming-language-hexflow

https://codegolf.stackexchange.com/questions/76928/write-an-interpreter-for-my-new-esoteric-language-streamflow

(Note that these were bad challenges and I posted them when I was a new PPCG user... rip these cringy challenges)

What does the community think about challenges asking for interpreters for arbitrary languages that OP invents for the challenge?

Answer 1

Encourage high-quality interpreter challenges

I really enjoy interpreter challenges. Why?

• They are often non-trivial. (I'll address counterexamples below.) I like more-complex challenges in general, and it seems like challenges to interpret a toy language or a tarpit frequently hit the right balance between difficult and doable. Answers to them can showcase some really intensive golfing.
• They are approachable. Some of us aren't experienced with abstract math, or physics, or cryptography. But we all know programming languages, and enjoy playing with them. Learning how to write an interpreter isn't too much of a learning curve.
• They can be educational. The languages we're asked to interpret in these challenges draw from a wide variety of programming paradigms: stack-based, functional, 2-dimensional, etc., etc. Simply understanding how the language is supposed to work can be a mind-stretching experience in itself.

Now, one potential downfall of interpreter challenges is that they are hard to do well. Because they are more complicated, they need to be very carefully specified; they frequently have lots of caveats and corner cases that must be considered. An incomplete spec can easily result in the question getting closed--as it should! But DJMcMayhem is right:

That's a problem with each challenge, not the category of challenges.

So, how do we encourage high-quality interpreter challenges?

• Pick an interesting but manageable language to interpret. I know trivial "languages" get upvotes, but I'd like to see languages that actually do something--not necessarily Turing-complete, but at least with some control flow to handle (loops, gotos, recursion). On the other hand, an "Interpret Python" challenge is going to be way too involved for this site (probably). Turing tarpits are a good choice, since they have very few instructions that need to be implemented. Try to design the language to be as small as possible while still being interesting.
• Search existing challenges to see if something similar has been done before. A quality interpreter challenge should add something new to the genre; we don't need fifty "Interpret this variant of BF" challenges. There's enough variety and downright weirdness among programming languages to find something interesting to implement.
• Use the sandbox. This isn't going to solve all the problems, for two reasons: 1) some people don't use the sandbox, and 2) even after sandboxing, a complex challenge will inevitably have more corner cases that didn't get caught. However, posting in the sandbox and asking for feedback over the course of a couple weeks will greatly increase the quality of a challenge, especially a complicated one. It can also help weed out the boring ones before they hit the main site. Get as many eyes on your challenge as possible. If/when your sandbox post stops getting feedback, post in chat asking for more feedback.
• Implement it yourself. Writing your own reference implementation forces you to tackle the corner cases and work out the exact behavior you want, which you can then write into the spec. But more importantly, a reference implementation is very helpful to potential answerers who will inevitably look at your spec and go, "Huh?" I may not understand how the language works from reading about it, especially if it uses an unusual execution model. But if I can experiment on my own with an existing interpreter, I'll be able to figure it out.
• A corollary to the above point: Write extensive example programs in the language (i.e., test cases). Example programs need to use all the features of the language, particularly any corner cases or "gotchas." If I can run all the example programs successfully with my submission, I should be pretty confident that my submission works according to spec.

This list is nothing new: these are essentially the ways we encourage high-quality challenges of any type. But these considerations are more important for categories like interpreter where the task and the spec are complicated. Trivial challenges may be ready to go after a short time in the sandbox, but for complex ones a longer stay is almost mandatory.

When low-quality challenges pop up, address them in the same way as any bad challenge. Downvote if it's trivial/uninteresting. Vote to close as unclear if it's unclear. Comment on improvements that can be made.

Finally, let me point out that the interpreter tag actually has quite a lot of good challenges, and only a few low-quality ones. As Wheat Wizard points out, several of the low-quality challenges are recent, which may contribute to your negative impression of the tag. I suggest browsing through some of the older ones--there are quite a few with 10-25 upvotes that deserve more attention.

Answer 2

Users should not use PPCG to get interpreters for underspecced languages

If you want an interpreter created, by someone else, for your brainchild, you need to make sure that your spec is 110%. This may involve creating an interpreter for it yourself, first, and testing it, to see how your language will actually work.

Answer 3

What does the community think about challenges asking for interpreters for arbitrary languages that OP invents for the challenge?

I think there's a more fundamental question about challenges which ask us to implement a task that the OP invented for the challenge.

In my opinion, one of the elements of a good challenge is a clear motivation. I should be able to see why OP cares about the challenge and why I should care. That doesn't mean that it has to have a practical application: sometimes the motivation is just that when you think about the problem it turns out to be surprisingly tricky. But there should be a motivation.

With questions which are created just for the sake of posting a question, OP's motivation is posting something, anything, as long as it gets them rep; and who knows what the answerer's motivation is supposed to be. It's not a promising start.

With questions such as "Interpret my snazzy new language", OP's motivation is strong, but because it's their baby they can lose objectivity when it comes to asking why anyone else should care about it.

Chat and the sandbox can both be used to gauge whether a task is interesting enough to motivate other people. The gold standard evidence for an interpreter question would be that there are two or three people other than you using your language (and not just trying it out once). So in addition to the reasons given in other answers for writing a reference implementation and example programs, a good reason to write a reference implementation is to try to gather a small community.

Answer 4

I dislike challenges like this for a few reasons:

• You could create way too many challenges just by inventing arbitrary languages, and this seems like a way to repfarm. Sometimes it's interesting but it often becomes a challenge where IO is the main component of it (especially in the linked challenge; the command names will end up being a large portion of the challenge).
• They end up being quite boring often