I think it's because you're setting the SecureRMISocketFactory as the RMISocketFactory - it's probably assuming that *all* socket communication is going to be using SSL. If you start the registry outside the server, it's not going to be using this SocketFactory, so when your server tries to register with the registry, it uses an SSL socket, but gets a "regular" socket response from the registry.
It's possible to use the "regular" registry, but still use a custom socket factory between your client and server - the
Using a Custom Socket Factory guide on Sun's site shows how.