Robert Houston wrote:On my browser I can see //webcode.me, so it really exists.
Paul Clapham wrote:If the latter is the case then I've had success in the past by putting in a browser description in the User-Agent header.
Robert Houston wrote:Tim Moores: I'd like to try your suggestion. the line that says: "con.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:73.0) Gecko/20100101 Firefox/73.0");" I need to change for my own environment (Edge). Where did you get this info for your environment. Can I construct it for my own and how?
Paul Clapham wrote:
My code which uses this property actually scrapes a whole site, about 10000 pages. But it has a 5-to-10-second delay between each access so it doesn't burn up the server, so it takes a couple of days to run. And I only run it once a year. And it is for personal use only.