I am not sure this post should be here. Please advise.
I have been using Indeed.com, a job search engine. I am wondering how it achieves its technical functions, how it gets all job data from different companies, websites, not only the big company like dice.com, also some small companies. Has some web service connection ? or just using html parsing ?....
If it wants to stay in business it probably has back-end agreements with those sites, promising more hits for a share of the revenue.
If it is just scanning sites it will quickly run into problems with those sites' Terms of Service, which more than likely prohibit site scraping.