- OSGi itself runs fine on top of the Java 9 classpath
- The Java module system has no inherent compatibility story with OSGi
- The OSGi alliance has been exploring interop between OSGi and Java modules, but there's no official story on that yet
In short, both worlds are currently quite separated. As to when you'd use OSGi vs. the Java module system, that really depends on your requirements. OSGi has many features support dynamic systems (e.g. modules/services can come and go at run-time). If that's what you need, stick with OSGi. If you need a more down-to-earth module system (with proper toolsupport, since its officially part of the platform and language), then checking out the Java module system is a good idea.