I don't know of any project that has been started specifically to build a JavaFX-specific game engine, but there certainly has been a lot of interest in using JavaFX for games. It seems like a natural fit for JavaFX's strengths in graphics and animation.
There have certainly been a lot of JavaFX game demos from pac man clones to tank battles (a google search will turn up plenty).
There is also Michael Heinrichs inverse kinematics library for JavaFX:
http://fxutil.netopyr.com/page.php?3
And several blog entries on using a Java-based physics engine in JavaFX games:
http://piliq.com/javafx/?page_id=729