regex confusion.

[ilteris kaplan]

Hello All,
Basically I am reading an url and ideally looking for "/comments-all" string. This unfortunately didn't work at all. I tried a very simple "comments" string but it didn't work either. Here's the code below, I think I am missing very simple thing here.

   
/* Daniel Shiffman               */
/* Programming from A to Z       */
/* Spring 2006                   */
/* <a href="http://www.shiffman.net" target="_blank" rel="nofollow">http://www.shiffman.net</a>       */
 

// Simple example of a web crawler
// URL queue: linked list
// Sites already visited: hash table
 
// Needs to be updated to comply with ROBOTS.TXT!
 
package a2z;
 
import java.util.*;
import java.util.regex.*;
 
public class Crawler {

private LinkedList urlsToVisit;// A queue of URLs to visit
private HashMap urlsVisited;// A table of already visited URLs
private Pattern href;// A Pattern to match an href tag
private String ignore;// To be used as a regex for ignoring media files (JPG,MOV, etc.)

public Crawler() {
urlsToVisit = new LinkedList();
urlsVisited = new HashMap();

// Match URLs
// Note using Pattern.COMMENTS flag which ignores white spaces and anything after '#' in a regex
//href = Pattern.compile("href# match href \n" + 
//"\\s*=\\s*\"# 0 or more spaces, =, 0 ore more spaces, quote \n" + 
//"(http[^\"\\s]*)# capture the URL itself, http followed by no spaces and no quotes \n" +
//"\"# ending with a quote \n",
//Pattern.CASE_INSENSITIVE | Pattern.COMMENTS);

// it's funny because above reads it correctly and give me back all the urls in my println.
 
String regex= "comments"; // this doesn't read and give no results back.
href = Pattern.compile(regex, Pattern.CASE_INSENSITIVE);

// We will ignore URLs ending with certain extensions
ignore = ".*(mov|jpg|gif|pdf)$";
}


public void addUrl(String urlpath) {
// Add it to both the LinkedList and the HashMap
urlsToVisit.add(urlpath);
urlsVisited.put(urlpath,urlpath);
 
}


// A method to determine if the queue is empty or not
public boolean queueEmpty() {
return urlsToVisit.isEmpty();
}

// A method to crawl one URL
public void crawl() {
 // Get a URL in the queue
String urlpath = (String) urlsToVisit.removeFirst();
// Read that URL
read(urlpath);
}

// A method to read a URL and look for other URLs
// (and possibly do some sort of analysis)
private void read(String urlpath) {

//System.out.println(urlsVisited.size() + " " + urlsToVisit.size() + " " + urlpath);
try {
// Grab the URL content
A2ZUrlReader urlr = new A2ZUrlReader(urlpath);
String stuff = urlr.getContent();
//System.out.println(stuff);
// This lame example just searches for web sites with the word "valentine" in them
// We could break up the documents and throw all the words into a concordance here
// Or implement some other type of storage / analysis algorithm
// (we also probably shouldn't compile a regular expresssion everytime here, but. . .)
Pattern valenp = Pattern.compile("comments", Pattern.CASE_INSENSITIVE);
int valentiney = 0;
Matcher valenm = valenp.matcher(stuff);
while (valenm.find()) {
valentiney++;
}

if (valentiney > 0) System.out.println(urlpath + ", would you be my valentine? My love for you is multipled by " + valentiney + "!");
/************************************/

// Match the URL pattern to the content
Matcher m = href.matcher(stuff);
// While there are URLs
while (m.find()) {
// Grab the captured part of the regex (the URLPath itself)
String newurl = m.group(1);
System.out.println("de"); 
// print out "de" when match it. never fires with my my regex
// If it hasn't already been visited (or if it matches the ignore pattern)
if (!newurl.matches(ignore) && !urlsVisited.containsKey(newurl)) {

addUrl(newurl);
}
}

} catch (Exception e) {
// System.out.println("Problem reading from " + urlpath + " " + e);
// e.printStackTrace();
}
}
 
}
 

 

[Stevi Deter]

ilteris,

What behavior are you seeing that makes you think this isn't working?

I ran your program (after finding the code for A2ZUrlReader.java, and found 6 matches when I passed in this page's url (http://www.coderanch.com/t/385301/java/java/regex-confusion) as the URL to be parsed. I changed the pattern in read(String) to "/comments-all" and had one hit. Both correspond to the content of this page before I added this comment.

Looks like the code works - it correctly retrieves a page and parses the page content looking for matches. What are you expecting it to do?

[ilteris kaplan]

Hey Stevi,

Thanks a lot for your time. I realized I am dumb once again. Never ends I guess

it's working fine here.

thanks again,
ilteris

[Stevi Deter]

Sometimes you just need that second pair of eyes!