Anti-aliasing of 3D scenes has not been implemented in JavaFX 2.0's Prism rendering engine (as of the 2.0.2 version).
The feature is scheduled for a future update release.
See http://javafx-jira.kenai.com/browse/RT-5534 for the related feature request - login required but anybody can signup to view the request.
Anti-aliasing did work for the previous JavaFX releases (1.3.x) because they were not, be default, using JavaFX's new Prism rendering engine.
Anti-aliasing may work with the 2.0.2 release if you switch it to use a software or java2d graphics pipeline - I think there is a -D java command line option which can be used to set a property for that, but I don't know what the command line option is, nor how to determine it.