I haven't seen a book dedicated to this topic, although many of the patterns books talk about it. One thing I've found is that if I at least design, and (even better) implement a command line interface for my GUI programs, the separation stays much cleaner. Besides, then the Unix folks love you
I realize that this won't work in every situation, but is more applicable than it might appear at first blush. For example, you might think this suggestion utterly inapplicable for an image editing program, but think about many of the functions an image editing program must perform. Convert among image formats? You can write a CLI for that. Apply some sort of effect to an image? You can write a CLI for that.
The trick, or course, is to make sure you're not writing two separate programs, and cutting and pasting code between them. Make sure you're using the same engine in both, and the interfaces (GUI and CLI) are just asking for and receiving input.
HTH.