How do I copy and paste to a text file?

Every day I use I.E. and go to about 30 web pages and I click "Select All" then "Copy" and then I "Paste Unformatted Text" into my word processor and then I "save as text file" after which I can use C to parse through the text file to get the data I am after.

Needless to say that navigating to 30 pages per day is very tedious, so I would like to write a JAVA program to automate this process for me.

This does not sound like a difficult thing to do but since all I know about JAVA I learned from Starbucks, I will need a little help to know where to get the information I need to start this project.
Any help is appreciated as I have spent hours wandering the Internet without getting any smarter.

thanx

carl

Well JSP isn't a technology that's going to help you in this so I'm moving this to Java in General (intermediate) for further discussion.

There are classes in the java.net packages that will help you out in this, but first you'll need to get the basics of java programming uner your belt.

This Sun Tutorial will give you some idea what it takes to read a web page into a Java program.

That part leaves you with HTML in a giant string. Then you'll need to parse the HTML to find the bits of data you want. I like the Quiotix Parser for this task. This part is non-trivial, but if you've done sophisticated stuff in C maybe you'll pick it right up.

Oh, and Welcome to the Ranch! The best approach around here is to take a shot at the smallest part of the problem you can think of, and if you get stuck post some code for us all to look at. Why not try some of the exercises in the Sun network tutorial and see if you can retrieve one page into a String? Maybe write class Page make this work:

You could also use the Url and InputStream classes and read the stream. Then you can 'scrape' everything between the <body> tags. Just a thought. I'm still a sorta newbie myself so maybe someone else could post a sample. I've done it a few times but all of my code is at work. Hope that helps.

Here is a sample class I wrote that contacts two different URLs, read the HTML, and parses it. I hope this helps.

import java.io.*;
import java.awt.*;
import java.net.*;
import java.awt.event.*;
import javax.swing.*;
 
// This program illustrates how to have buttons control text on
// a GUI.
 
// Two seperate JPanels are placed on a JFrame. The first JPanel
// will read information from a URL based on a button clicked.
 
// The second URL will constantly read information from
// weatherunderground.com about Carlisle, UK.
 
public class ReadList extends JFrame implements ActionListener {
 private JButton[] lists;
 private String topic;
 private String date;
 private MyPanel displayPanel;
 private TimeTemperaturePanel timeTemperaturePanel;
 
 private JButton createJButton(String label,
 String actionCommand,
 int width,
 int height,
 int xlocation,
 int ylocation,
 ActionListener listener) {
 JButton button = new JButton(label);
 button.setSize(width,height);
 button.setLocation(xlocation,ylocation);
 
 button.setActionCommand(actionCommand);
 
 Insets insets = button.getInsets();
 
 insets.left = 0;
 insets.right = 0;
 
 button.setMargin(insets);
 
 button.addActionListener(listener);
 
 return(button);
 }
 
// In this method, we create a BufferedReader to read efficiently from a URL.
// We keep reading from the BufferedReader until there are no more characters to
// read. The characters are read into a buffer which is then added to an array.
// After all of the characters are in an array, we convert the characters to a String.
 
 private String readURL(String url) {
 try {
 BufferedReader input = new BufferedReader(new InputStreamReader(new URL(url).openStream()));
 
 char[] chars = new char[0];
 
 char[] buffer = new char[4000*1024];
 
 int length = 0;
 
 while ((length = input.read(buffer,0,buffer.length)) != -1) {
 char[] temp = new char[chars.length+length];
 
 System.arraycopy(chars,0,temp,0,chars.length);
 System.arraycopy(buffer,0,temp,chars.length,length);
 
 chars = temp;
 }
 
 input.close();
 
 return(String.valueOf(chars));
 } catch (IOException ie) {
 return("");
 }
 }
 
// This method parses the output from a URL and locates a top ten list.
 
 private String parseURL(String url) {
 String line = readURL(url);
 
 int index = line.indexOf("");
 
 line = line.substring(index);
 
 index = line.indexOf("<center><a href");
 
 line = line.substring(0,index);
 
 line = line.replaceAll("","");
 line = line.replaceAll("","");
 line = line.replaceAll("","");
 line = line.replaceAll(" ","");
 line = line.replaceAll("<[/]*b>","");
 line = line.replaceAll("",";");
 line = line.replaceAll("","");
 line = line.replaceAll("\\s+"," ");
 
 String[] list = line.split(";");
 
 topic = list[0];
 
 String output = "";
 
 for (int counter=0;counter<list.length;counter++)
 output += list[counter] + " ";
 
 return(output);
 }
 
// This method contacts weatherunderground.com and gets the local time for
// Carlisle, UK.
 
 private String localTime() {
 String line = readURL("http://www.weatherunderground.com/global/stations/03220.html");
 
 int index = line.indexOf("Local Time");
 
 line = line.substring(index);
 
 index = line.indexOf("");
 
 line = line.substring(index+3,line.length());
 
 index = line.indexOf("");
 
 line = line.substring(0,index);
 
 return(line);
 }
 
// This method contacts weatherunderground.com and gets the local temperature fro
// Carlisle, UK.
 
 private String temperature() {
 String line = readURL("http://www.weatherunderground.com/global/stations/03220.html");
 
 int index = line.indexOf(" °F");
 
 line = line.substring(0,index);
 
 index = line.lastIndexOf("");
 
 line = line.substring(index);
 
 line = line.replaceAll("<[/]*b>","");
 
 return(line);
 }
 
// This method causes the currently executing thread to pause for a specified number of seconds.
 
 private void pause(double seconds) {
 try {
 Thread.sleep((int)(seconds*1000));
 } catch (InterruptedException ie) {
 }
 }
 
/**********************************************************
 
Definition for inner class MyPanel
 
**********************************************************/
 
// This panel is used to display a scrolling top ten list.
 
 public class MyPanel extends JPanel implements Runnable {
 private String line;
 private int startx;
 private boolean keepGoing;
 
 public MyPanel(int width,
 int height,
 int xlocation,
 int ylocation) {
 super(true); // Create the JPanel so that it uses double buffering.
 
 setSize(width,height);
 setLocation(xlocation,ylocation);
 
 line = "";
 
 startx = getWidth();
 }
 
 public void setLine(String line) {
 startx = getWidth();
 
 this.line = line;
 }
 
 public void start() {
 keepGoing = true;
 
 new Thread(this).start();
 }
 
 public void paint(Graphics g) {
 g.clearRect(0,0,getWidth(),getHeight());
 
 super.paint(g);
 
 g.setFont(new Font("Arial",Font.BOLD + Font.ITALIC,20));
 
 FontMetrics metrics = g.getFontMetrics();
 
 int width = metrics.stringWidth(line);
 
// If the text has gone off the left side of the screen, put it back on the right side.
 
 if (startx + width < 0)
 startx = getWidth();
 else
 startx -= 10;
 
 int verticalBase = getHeight()/2 + metrics.getHeight()/4;
 
 g.drawString(line,startx,verticalBase);
 }
 
 public void stop() {
 keepGoing = false;
 }
 
 public void run() {
 while (keepGoing) {
 pause(0.1);
 
 Thread.yield();
 
 repaint();
 }
 }
 }
 
// End of definition for inner class MyPanel
 
/**********************************************************
 
Definition for inner class TimeTemperaturePanel
 
**********************************************************/
 
 public class TimeTemperaturePanel extends JPanel implements Runnable {
 private String line1;
 private String line2;
 private boolean keepGoing;
 
 public TimeTemperaturePanel(int width,
 int height,
 int xlocation,
 int ylocation) {
 super(true);
 
 setSize(width,height);
 setLocation(xlocation,ylocation);
 
 line1 = "";
 line2 = "";
 }
 
 public void start() {
 keepGoing = true;
 
 new Thread(this).start();
 }
 
 public void paint(Graphics g) {
 g.clearRect(0,0,getWidth(),getHeight());
 
 super.paint(g);
 
 FontMetrics metrics = g.getFontMetrics();
 
 int width = metrics.stringWidth(line1);
 int height = metrics.getHeight();
 
 g.drawString(line1,getWidth()/2-width/2,getHeight()/2-height);
 
 width = metrics.stringWidth(line2);
 
 g.drawString(line2,getWidth()/2-width/2,getHeight()/2+height);
 }
 
 public void run() {
 while (keepGoing) {
 line1 = "Local Time at Carlisle, UK " + localTime() + " ";
 line2 = "Temperature is " + temperature() + " degree Fahrenheit";
 
 pause(0.1);
 
 Thread.yield();
 
 repaint();
 }
 }
 }
 
// End of definition for inner class TimeTemperaturePanel
 
 public ReadList(String title) {
 super(title);
 
 Container container = getContentPane();
 
 container.setLayout(null);
 
 container.setBackground(Color.white);
 
 setSize(400,400);
 
 setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
 
 String[] labels = {"12/1/05","12/12/05","11/03/05","11/10/05","11/08/05"};
 
 String[] actionCommands = {"20051201","20051212","20051103","20051110","20051108"};
 
 lists = new JButton[labels.length];
 
 for (int counter=0;counter<lists.length;counter++)
 container.add(lists[counter] = createJButton(labels[counter],
 actionCommands[counter],
 60,20,
 20+counter*70,20,
 this));
 
 container.add(displayPanel = new MyPanel(400,100,0,150));
 container.add(timeTemperaturePanel = new TimeTemperaturePanel(400,100,0,250));
 
 topic = "";
 date = "12/1/05";
 
 displayPanel.setLine(parseURL("http://www.cbs.com/latenight/lateshow/top_ten/archive/ls_topten_archive2005/ls_topten_archive_20051201.shtml"));
 
 displayPanel.start();
 
 timeTemperaturePanel.start();
 
 setVisible(true);
 }
 
 public void paint(Graphics g) {
 super.paint(g);
 
 FontMetrics metrics = g.getFontMetrics();
 
 int width = metrics.stringWidth(date);
 
 g.drawString(date,getWidth()/2-width/2,100);
 
 width = metrics.stringWidth(topic);
 
 g.drawString(topic,getWidth()/2-width/2,120);
 }
 
 public void actionPerformed(ActionEvent e) {
 String url = "http://www.cbs.com/latenight/lateshow/top_ten/archive/ls_topten_archive2005/ls_topten_archive_" + e.getActionCommand() + ".shtml";
 
 date = ((JButton)e.getSource()).getText();
 
 displayPanel.setLine(parseURL(url));
 
 repaint();
 }
 
 public static void main(String[] args) {
 ReadList readList = new ReadList("Read List");
 }
 
}

Originally posted by carl sjostrom:
Every day I use I.E. and go to about 30 web pages and I click "Select All" then "Copy" and then I "Paste Unformatted Text" into my word processor and then I "save as text file" after which I can use C to parse through the text file to get the data I am after.

Needless to say that navigating to 30 pages per day is very tedious, so I would like to write a JAVA program to automate this process for me.

This does not sound like a difficult thing to do but since all I know about JAVA I learned from Starbucks, I will need a little help to know where to get the information I need to start this project.
Any help is appreciated as I have spent hours wandering the Internet without getting any smarter.

thanx

carl

If I may suggest, I think HtmlUnit might be just what you need. This lets you object-orientedly connect to any URL and pull out exactly the data you need. This might solve both your problems at one go, avoiding you to have to first copy-paste and then parse.

Let's us know how you did it finally.

So much help and so little time.
Thanks for all the information, it is more than I expected, it will keep me busy (happy) for a long time. As bad as I am at programming, I just love it more than anything.

thanx again

carl

Okay it works (sort of).
Java is easier than I thought it would be.
And here is my next problem.
My simple program will download the text of a web page and save it to a .txt file, as long as I do not need a password to access the page.
So how do I include my user id and password info to get access to the pages that are password protected?

For now I am trying to get into my Yahoo! mailbox.
Here is what my code looks like so far.

import java.io.*;
import java.awt.*;
import java.net.*;
import java.awt.event.*;
import javax.swing.*;
 
public class ReadList extends JFrame implements ActionListener 
{ 
 public ReadList(String title) 
 { 
 super(title);
 String line = readURL("http://www.yahoo.com/");
 saveStringToFile( "c:/yahooFile.txt", line );
// This works fine since no password is required
// I get to -> <title>Yahoo!</title> and the text
// is saved to "c:/yahooFile.txt"
 
// I copied the url of my mailbox to see if I could get there
// but it would not work
 String line2 = readURL("http://us.f523.mail.yahoo.com/ym/login?.rand=1ndrg7ajiie7g");
 saveStringToFile( "c:/yahooMailFile.txt", line2 );
// I want to get this page ->
// <title>Yahoo! Mail - olavisjo@yahoo.com</title> 
// But without a password all I get is this ->
// <title>Yahoo! Mail</title> 
 } 
 public static void saveStringToFile(String filepath,String text)
 {
 try
 {
 // check if file exists; if not create it
 File file=new File(filepath);
 if( ! file.exists()) file.createNewFile();
 // write file
 PrintWriter out=new PrintWriter(new BufferedWriter(new FileWriter(filepath)));
 out.write(text);
 out.close();
 
 } 
 catch(Exception e){ e.printStackTrace();}
 }
 public void actionPerformed(ActionEvent e) 
 { 
 //useless function but seems to be necessary to overide something.
 } 
 
 // In this method, we create a BufferedReader to
 // read efficiently from a URL.
 // We keep reading from the BufferedReader until
 // there are no more characters to
 // read. The characters are read into a buffer
 // which is then added to an array.
 // After all of the characters are in an array, we
 // convert the characters to a String. 
 private String readURL(String url) 
 { 
 try 
 { 
 BufferedReader input = new BufferedReader(new InputStreamReader(new URL(url).openStream()));
 char[] chars = new char[0];
 char[] buffer = new char[4000*1024];
 int length = 0;
 while ((length = input.read(buffer,0,buffer.length)) != -1) 
 { 
 char[] temp = new char[chars.length+length];
 System.arraycopy(chars,0,temp,0,chars.length);
 System.arraycopy(buffer,0,temp,chars.length,length);
 chars = temp;
 } 
 input.close();
 return(String.valueOf(chars));
 } 
 catch (IOException ie) 
 { 
 return(" readURL error ");
 } 
 }
 public static void main(String[] args) 
 { 
 ReadList readList = new ReadList("ReadList");
 }
}

You should note that if your code is deployed on an application server, it contains a potential resource leak. Specifically, if anything after your opening of the PrintWriter throws an exception, your PrintWriter will remain open indefinitely. There is one specific method call that explicitly declares that it may throw an exception, which makes it all the more plausible.

The correct solution to this common issue (at least in my experience) is to use an embedded try/finally. Avoid the common, but erroneous idiom of assigning the PrintWriter reference to null, hthen using 'nullness' to determine if the PrintWriter was opened.

Here is something related:
http://jqa.tmorris.net/GetQAndA.action?qids=38&showAnswers=true