coding simple bayes classifier in java . need help with the algo

[karthik raghunathan]

edit : deleted original post, removed links to github, copied code here instead. corrected spelling.

My naive bayes classifier started out as a spam filter and now has been recruited to classify whether a text is by Dickens or Twain.
First of all, would this be the right forum to ask this question ?

Second, it doesn't work very well. Can anyone help me correct the algo ? I sorta copied some of it from shiffman.net tutorial, which sorta uses the 'paulgraham : a plan for spam' approach.
ps : the code is not in OOP style, it's more or less procedural. Is this a problem ?

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

import org.jsoup.Jsoup;

public class Counter {

    static Map<String, Integer> good = new HashMap<String, Integer>();
	static Map<String, Integer> bad = new HashMap<String, Integer>();

	static String[] specialCharacters = { ",", "#", ";", "\"", "\'", };
	static String empty = "";

	static void print(String s) {
		System.out.println(s);
	}

	static void store(String fname, String data) {
		try {
			FileOutputStream fos = new FileOutputStream(new File(fname));
			fos.write(data.getBytes());
			fos.close();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	static String getFile(String path) {
		try {
			FileInputStream fis = new FileInputStream(new File(path));
			byte[] b = new byte[fis.available()];
			fis.read(b);
			fis.close();
			fis = null;
			return new String(b);
		} catch (Exception e) {
			e.printStackTrace();
			return null;
		}
	}

	static String getURL(String url) {
		try {
			URL u = new URL(url);
			URLConnection uc = u.openConnection();
			InputStream is = uc.getInputStream();
			byte[] b=  new byte[is.available()];
			is.read(b);
			is.close();
			return new String(b);
		} 
		catch (IOException e) {
			e.printStackTrace();
			return null;
		}
	}

	static String takeOutSpecialCharacters(String s) {
		for (String p : specialCharacters) {
			s = s.replaceAll(p, empty);
		}
		return s.toLowerCase();
	}

	static int count(Map<String, Integer> map) {
		int count = 0;
		for (Integer i : map.values()) {
			count += i;
		}
		// System.out.println(count);
		return count;
	}

	static void train(String text, boolean ham) {
		String[] tokens = text.split(" ");

		Map<String, Integer> map = ham ? good : bad;

		for (String token : tokens) {
			token = takeOutSpecialCharacters(token);
			if (map.containsKey(token)) {
				map.put(token, map.get(token) + 1);
			} else {
				map.put(token, 1);
			}
		}
	}

	static float calculateProbability(String s) {
		s = takeOutSpecialCharacters(s);

		float pSpam = 0f;

		float goodRatio = 0;
		if (good.containsKey(s)) {
			goodRatio = (float) (((float) good.get(s)) / (float) count(good));
		}
		float badRatio = 0;
		if (bad.containsKey(s)) {
			badRatio = (float) (((float) bad.get(s)) / (float) count(bad));
		}
		if (goodRatio + badRatio > 0) {
			pSpam = (badRatio / (goodRatio + badRatio));
		}

		if (pSpam > 0.99f)
			pSpam = 0.99f;
		if (pSpam < 0.1f)
			pSpam = 0.1f;

		return pSpam;

	}

	static float interesting(Float f) {
		return Math.abs(0.5f - f);
	}

	static float mul(Float[] f, boolean oneMinus) {
		float a = 1.0f;
		for (float f1 : f) {
			if (oneMinus) {
				a = a * (1 - f1);
			} else {
				a = a * f1;
			}
		}
		return a;
	}

	static float classify(String text) {
		String[] tokens = text.split(" ");

		Set<String> set = new HashSet<String>();

		for (String token : tokens) {
			set.add(takeOutSpecialCharacters(token));
		}

		List<Float> problist = new ArrayList<Float>();

		for (String token : set) {

			float f = calculateProbability(token);

			if (f > 0.5f) {
				// System.out.println( ""+ feature + ":"+ f);
				problist.add(f);
			}
		}

		Collections.sort(problist, new Comparator<Float>() {
			@Override
			public int compare(Float f1, Float f2) {
				return  (int)(100*interesting(f2) - 100*interesting(f1)); //descending order
			}
		});

		Float[] probabilities = problist.subList(0, 16).toArray(new Float[15]);

		float product = mul(probabilities, false);
		float oneMinusTerm = mul(probabilities, true);

		print("[]" + product + "," + oneMinusTerm);

		return (product / (product + oneMinusTerm));
	}
	
	static String[] pullNStore(String nub, String[] files){
		List<String> names = new ArrayList<String>();
		for(String file : files){
			String name = new File(file).getName();
			name = name.substring( 0, name.indexOf("."));
			names.add(nub+name);
			store( nub + name, Counter.getURL(file));
			print( "downloading :"+file +" as " + nub+name);
		}
		return names.toArray( new String[names.size()]);
	}

	public static void main(String[] args) {

		String[] dickens = {"http://www.gutenberg.org/ebooks/1400.txt.utf8",
				"http://www.gutenberg.org/ebooks/98.txt.utf8",
				"http://www.gutenberg.org/ebooks/730.txt.utf8",
				"http://www.gutenberg.org/ebooks/766.txt.utf8",
				"http://www.gutenberg.org/ebooks/580.txt.utf8",
				"http://www.gutenberg.org/ebooks/1023.txt.utf8",
				"http://www.gutenberg.org/ebooks/786.txt.utf8",
				"http://www.gutenberg.org/files/564/564-0.txt",
				"http://www.gutenberg.org/ebooks/967.txt.utf8",
				"http://www.gutenberg.org/ebooks/963.txt.utf8",
				"http://www.gutenberg.org/ebooks/883.txt.utf8",
				"http://www.gutenberg.org/ebooks/700.txt.utf8",
				"http://www.gutenberg.org/ebooks/821.txt.utf8"};
		
		String[] twain = {   "http://www.gutenberg.org/ebooks/76.txt.utf8",
				"http://www.gutenberg.org/ebooks/74.txt.utf8",
				"http://www.gutenberg.org/ebooks/10947.txt.utf8",
				"http://www.gutenberg.org/ebooks/19640.txt.utf8",
				"http://www.gutenberg.org/ebooks/30165.txt.utf8",
				"http://www.gutenberg.org/ebooks/86.txt.utf8",
				"http://www.gutenberg.org/ebooks/1837.txt.utf8",
				"http://www.gutenberg.org/ebooks/3176.txt.utf8",
				"http://www.gutenberg.org/ebooks/10135.txt.utf8",
				"http://www.gutenberg.org/ebooks/245.txt.utf8",
				"http://www.gutenberg.org/ebooks/119.txt.utf8",
				"http://www.gutenberg.org/ebooks/3177.txt.utf8",
				"http://www.gutenberg.org/ebooks/8525.txt.utf8",
				"http://www.gutenberg.org/ebooks/70.txt.utf8",
				"http://www.gutenberg.org/ebooks/3186.txt.utf8",
				"http://www.gutenberg.org/ebooks/2895.txt.utf8",
				"http://www.gutenberg.org/ebooks/142.txt.utf8",
				"http://www.gutenberg.org/ebooks/3200.txt.utf8",
				"http://www.gutenberg.org/ebooks/3178.txt.utf8",
				"http://www.gutenberg.org/ebooks/3250.txt.utf8",
				"http://www.gutenberg.org/ebooks/102.txt.utf8",
				"http://www.gutenberg.org/ebooks/3190.txt.utf8",
				"http://www.gutenberg.org/ebooks/26203.txt.utf8",
				"http://www.gutenberg.org/ebooks/18381.txt.utf8",
				"http://www.gutenberg.org/ebooks/3187.txt.utf8"};
		
		
		String[] storedDickens = pullNStore( "dickens", Arrays.copyOfRange(dickens,0,1));
		String[] storedTwain = pullNStore( "twain", Arrays.copyOfRange(twain,0,1));
		pullNStore( "twain", Arrays.copyOfRange(twain,6,7));
		
		for(String s : storedDickens) {
			train( getFile(s), true);
			System.out.println( "trained,dickens:"+ s);
		}
		
		for(String s : storedTwain) {
			train( getFile(s), false);
			System.out.println( "trained,twain:"+ s);
		}
		
		float f = classify( getFile("twain1837") );
		System.out.println( "is a twain book :" + f);
	}

}

[Dave Trower]

I read an article on how spam filters work but my experience is limited to that one article. You need to create a dictionary where for each word used in any book returns a probability of if the book is twain or Dickens. I think you do this by counting the frequency of each word.
Then when you are given a sample book, you look up the probabilities for each word from the dictionary and then apply the Bayesian algorithm.
Let me know if this helps.

[karthik raghunathan]

Looks like I read the same article too. Except I don't know what I am doing wrong in my code 'cos I don't know what output to expect

[Dave Trower]

I would suggest you google the words "bayesian for spam". I do not have the original article but this is a good one:
webpage

Here is a quote from the article:
This word probability is calculated as follows: If the word “mortgage” occurs in 400 of 3,000 spam mails and
in 5 out of 300 legitimate emails, for example, then its spam probability would be 0.8889 (that is, [400/3000]
divided by [5/300 + 400/3000]).

So now in the dictionary, the word mortgage probability is 0.8889.

So if the word mortgage is used in an e-mail, there 88.89% chance the e-mail is spam. However, the bayesian filter looks at all words in an e-mail. So the total probability of an e-mail being spam would change based on the other words.

In your case, you build a dictionary based on how often a word appears in which of the two works.
I think the output of your program should be something like:
There is a 99.3% chance the book I just looked at is Twain.

[karthik raghunathan]

Okay , that is pretty much what I am doing ,except for the final part, where I combine probabilities for each of the words.
I am making two dictionaries - one for spam and one for ham, then I calculate spam probability by doing rSpam/ (rSpam + rHam)
So I _am_ on the right track. Let me see if I am doing something wrong in the combining of probabilities .....
I also return a default of 0.1 if the probability is 0. That might be a downer..
I'll update in a day

[karthik raghunathan]

I'm going to try with different data. Maybe twain and dickens aren't too different ...... marking this resolved.
Thanks for all the help. really appreciate it.