Refactoring is a pretty general term and can mean a lot of things.
My coding style used to be called Stepwise Refinement and that has morphed into sort of a constant refactoring style. Write something that works and present it to the users with no personal investment. If they don't like it, throw it away, if they do like it bullet proof it.
So assuming you have a program that works the way you want it, the most common reason to refactor would be to make it more maintainable.
My recommendation would be to start by making sure the Javadoc is complete and accurate. This should give you a good idea where the problems are.