This week's book giveaway is in the OO, Patterns, UML and Refactoring forum. We're giving away four copies of Refactoring for Software Design Smells: Managing Technical Debt and have Girish Suryanarayana, Ganesh Samarthyam & Tushar Sharma on-line! See this thread for details.
The edges are smooth and anti-aliased (at least on my machine), could be that your eyes are more sensitive to the anti-aliasing algorithm or you are running off different hardware or system software which causes the jaggies for you.
changing from StageStyle.UNDECORATED to StageStyle.TRANSPARENT appeared to work - although I cant figure out why I'm not getting the standard title bar icons now.
If you want decorations like a title bar or icons, you will need to create new nodes in the dialog which render these things and write further code to setup appropriate event handlers for them (see the Ensemble application for an example of how to do this: http://www.oracle.com/technetwork/java/javafx/samples/index.html).