That's not a DOCTYPE, that's the prolog. If you don't want it, there's probably a parameter in the serializer you used which allows you to exclude it. And you should realize that if you remove the root element, there won't be anything left in your document.
The point being, you're confusing an XML document -- which in JDOM is a tree structure consisting of elements and text and other things -- with its text representation when you serialize it. When you're working in JDOM you should think of it as a tree structure. In particular the tree structure doesn't contain start tags or end tags, it just contains elements.
If you really wanted to remove just the dark red lines and leave the black lines, then the result wouldn't be a well-formed XML document because it wouldn't have a root element. So you can't use any XML serializer to output such a thing and you can't even represent it in JDOM.
However if you really want such a document, and I suspect you might not really want it, you can produce it with an XSLT transformation.