The problem is that in your while loop, you check for three placed ships. i only gets increased if rowSpace >= 3, which will never happen if two ships are in the same row, because col gets incremented to 3, then once more to 4, and after that (5+1-col) will never be >= 3. So you're stuck in an infinite loop.
An important note about your code. NEVER use assignment expressions within another expression, like you do with rowSpace. Assign the value to rowSpace in the line before it, and then use rowSpace in the expression afterwards. When I saw the code I immediately thought your bug was there, because you had intended it to be an == instead of an =. So don't do it.
The mind is a strange and wonderful thing. I'm not sure that it will ever be able to figure itself out, everything else, maybe. From the atom to the universe, everything, except itself.
I’ve looked at a lot of different solutions, and in my humble opinion Aspose is the way to go. Here’s the link: http://aspose.com