i am a paid subscriber of a website, and i want to write a program that automatically downloads the subscriber-only content and make sense of it automatically (the scale is too large to do it manually).
however, im stuck at the login. how do i make the java program log in to the site so that the program will be able to download the subscriber only content to make sense of it all automatically?
Joined: Jul 07, 2009
why do people reply without actually replying to the topic?
Do you think it's unimportant whether the site owner permits you to write a bot to download the contents of the site? We here at the Ranch don't think it's unimportant. In fact we don't support this kind of activity at all. And if you're okay with breaking contracts, then perhaps the "ego" problem is yours.
Now it might be that the owner of that site doesn't mind you downloading the site's contents. But it's your responsibility to find that out first.
One other benefit of reading the ToS might be you find an API is available for the purpose you wish to use it for. For example if the content is truly too large for manual processing, the site owner may understand this and provide a programmatic means for you to get at it - without the need to screen-scrape.
So do some research (i.e. contact the web site) and see if there is something available to make your job easier.