Make a search engine!
code-golf
Background
You are now the first employee of DuckDuckGoogle, a company founded by a Google employee and a DuckDuckGo employee! They have a new vision for search engine! But obviously, they need to make a search engine first! However, they are bad programmers (nobody knows how they were even hired at DuckDuckGo and Google!).
So you are assigned the task of making a search engine.
Search engines
There are three parts to a search engine, a web crawler, an indexer, and a searcher.
The web crawler takes a link, and saves the page and stores the page data (the HTML code). It then looks for links in the page and visits those webpages and does the same. It will continue to do this until there are no more links left.
The web indexer will take the text from each page, take each word, and add it to a dictionary of words and the pages it is associated with. If the word is already in the dictionary, it adds the page as a value of that word that is found in the page. For example, lets say www.example.com/index.html contains the text "hello. thanks! goodbye." and the ww.example.com/index.html is linked to www.example.com/hello.html with the. text "hello. goodbye." The crawler would extract the text from the index.html, and find hello.html, and extract the text from hello.html. The indexer would make a dictionary like this:
dictionary ={"hello.": ["www.example.com/index.html","ww.example.com/hello.html"], "goodbye.":["www.example.com/index.html","ww.example.com/hello.html"], "thanks!": ["ww.example.com/index.html"]}
The simplest part of the search engine is the searcher, or retriever. It simply retrieves the value for the search term (i.e. the list of pages the search term is found in) and prints it. That is the final result of the program. So if the search term is "hello.", the program will print ["www.example.com/index.html","ww.example.com/hello.html"]
Now, here is the difficult part. You must take into account punctuation. So "hello" returns the same as "hello." Also the search engine should be case-insensitive. So "Hello" is the same as "hello", which, as we said, is the same as "hello." which was ["www.example.com/index.html","ww.example.com/hello.html"]
Challenge
The challenge is to write a search engine in the least number of bytes as possible. Why? Because your bosses have decided to use 90% of the one hard drive they have (budget problems!) for the Holy Grail part of their project and you only get 10%.
Rules
- The input is the "seed", a url at which the crawler should start at, and also the search term.
- The output is a list of webpage urls where the search term is found.
- The program cannot use builtins for a search engine or even a crawler (or related, like a scraper or spider library).
- No need to worry about Javascript at all. Assume that the pages will not have Javascript.
- The webpages are assumed to have HTML 4.0.1 Strict.
Testing
I am pretty sure people know how search engines work. So I will only include two testcases. You can use the websites in the testcases for testing purposes as well.
> "https://www.udacity.com/cs101x/index.html" "crawl"
["https://www.udacity.com/cs101x/index.html","https://www.udacity.com/cs101x/crawling.html"]
> "https://www.udacity.com/cs101x/urank/index.html" "hummus"
["https://www.udacity.com/cs101x/urank/index.html","https://www.udacity.com/cs101x/urank/hummus.html","https://www.udacity.com/cs101x/urank/arsenic.html","https://www.udacity.com/cs101x/urank/kathleen.html","https://www.udacity.com/cs101x/urank/nickel.html","https://www.udacity.com/cs101x/urank/zinc.html"]