It looks like you're familiar with code complex GUIs. Full disclosure: I am not. But I do know a little about JavaFX.
It looks to me like the left-hand menu could become an accordion control
in JavaFX. The MDI panes could become a tabbed control. Of course, you lose the detachable windows.
Another idea is to use multiple scenes. (In JavaFX, you add controls to a root container, add this container to a Scene, then add the Scene to a Stage.) By changing scenes you can bring up a complete scene graph (controls in a container) one by one. The problem is that each Scene uses the same Stage (window) so they are effectively modal.
You could also use multiple stages for your multi-windows effect. I have only used at most two stages, the second one for a modal display, so I don't know about controlling several stages at once. I can imagine that you could launch several stages with an initial stage as the owner.
I have a simple app written in JavaFX 8 that uses a second Stage to display results, if you want to look at it: