2
\$\begingroup\$

Problem

With a recent influx of challenges, the poor old API has been taking a bit of a hammering and some of us are finding ourselves hitting the throttle limit and being blocked from accessing the API for extended periods of times.

The answers below propose possible solutions to counter this problem and, hopefully, from them we can come up with something viable.

| |
\$\endgroup\$
  • \$\begingroup\$ I don't think this is a good idea, mostly because the website would hit the API limit really quickly, even if it has the 10k for registered applications. What the website can do instead is take the DB dump at Data.SE and query that. \$\endgroup\$ – ASCII-only May 12 '17 at 12:15
  • \$\begingroup\$ @ASCII-only, I'm not proposing that the site query the API for every request made through it, I'm proposing that the site store local copies of the response to each URL submitted to it and serve up that local file instead so, for each different request URL it would only ever make one request to the API. \$\endgroup\$ – Shaggy May 12 '17 at 12:18
  • \$\begingroup\$ That isn't a good idea, since even though live data is not needed, the data needs to be at least a bit up to date, data that's a year old is not going to help. Furthermore, since almost every request is different, I'd expect the site would get well over 10000 requests per day. \$\endgroup\$ – ASCII-only May 12 '17 at 12:21
  • \$\begingroup\$ Actually, @ASCII-only, I like your suggestion that the site use Data.SE instead - could you expand on it and post it as an answer? I'll lift my proposal out of the question and post it as an answer to; hopefully we'll get a few different ideas and be able to come up with a viable solution. \$\endgroup\$ – Shaggy May 12 '17 at 12:30
  • \$\begingroup\$ A very late (and probably no longer as relevant) answer: ask people to stop posting [stack-exchange-api] questions for a while, spread them out more \$\endgroup\$ – caird coinheringaahing Oct 14 at 10:25
1
\$\begingroup\$

Proposal

To solve the problem and as (with very rare exception) live data is not required in order to complete API based challenges, I propose the creation of a website, built and maintained by the community, to implement the API as a search on SEDE's data dump, which is updated every week.


How it would work

  • A user posting a new challenge would enter the URL(s) of the API request required to complete the challenge, except under the domain of the website instead of that of Stack Exchange. e.g. instead of https://api.stackexchange.com/2.2/comments?order=desc&min=10&sort=votes&site=codegolf users would enter something likeĀ https://semirrorapi.com/2.2/comments?order=desc&min=10&sort=votes&site=codegolf
  • The site would then execute a database query to retrieve the response for that request.
  • All Stack Exchange API questions would be required to use this website instead of the pfficial Stack Exchange API.
| |
\$\endgroup\$
  • 1
    \$\begingroup\$ would enter the URL of the API request required to complete the challenge There's usually more than one. \$\endgroup\$ – Dennis May 12 '17 at 15:53
-1
\$\begingroup\$

Proposal

To solve the problem and as (with very rare exception) live data is not required in order to complete API based challenges, I propose the creation of a website, built & maintained by the community, to cache responses from the API that members can then use in their solutions.


How it Would Work

  • A user posting a new challenge would enter the URL of the API request required to complete the challenge.
  • The site would retrieve the response for that request and save it locally as a JSON file.
  • A local URL would be returned to the user for inclusion in their question.
    • If a user enters a URL for a request with a response already saved then the existing URL would be returned and, optionally, the local file would be updated with fresh data if it was older than a set period of time.
  • All users attempting to answer the challenge would use that all users answering it must use in their solution.

Questions

  • Isn't figuring out the API request usually part of the challenge?
    Yes, it can be seen to be, however, once one user figures out the optimal URL to use then all users attempting the challenge will use the same URL so it's not really an issue.
  • What about challenges that do require live data?
    Hopefully, they will remain so few and far between and only require one request to be made to the API that we could conceivably still use the API for those particular challenges.
  • What about questions that necessitate requesting multiple pages from the API?
    In most cases, removing the need to request multiple pages does not detract from the challenge. In fact, in my opinion, it improves it. Consider the "Whose rep is tied?" and "PPCG's Unanswered Questions" challenges; both require the requesting of multiple pages from the API but neither is improved by having that requirement nor would either be worse for the its removal.
| |
\$\endgroup\$
  • \$\begingroup\$ once one user figures out the optimal URL to use then all users attempting the challenge will use the same URL That's incorrect. With different filters, the output has to be parsed in different ways, and the optimal one might very well depend on the language of choice. \$\endgroup\$ – Dennis May 12 '17 at 15:54
  • \$\begingroup\$ A fair point, @Dennis. I have an idea on how to alter my proposal to accommodate that, I just need to work on wording it properly. But, first ... sleep! \$\endgroup\$ – Shaggy May 13 '17 at 2:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .