This week's book giveaway is in the OCAJP 8 forum. We're giving away four copies of OCA Java SE 8 Programmer I Study Guide and have Edward Finegan & Robert Liguori on-line! See this thread for details.
The edges are smooth and anti-aliased (at least on my machine), could be that your eyes are more sensitive to the anti-aliasing algorithm or you are running off different hardware or system software which causes the jaggies for you.
changing from StageStyle.UNDECORATED to StageStyle.TRANSPARENT appeared to work - although I cant figure out why I'm not getting the standard title bar icons now.
If you want decorations like a title bar or icons, you will need to create new nodes in the dialog which render these things and write further code to setup appropriate event handlers for them (see the Ensemble application for an example of how to do this: http://www.oracle.com/technetwork/java/javafx/samples/index.html).