The "root set" is the set of all... roots. Imagine memory as a collection of "trees" -- objects refer to other objects, which refer to other objects. If you have a list of the roots of those trees, then you have a way of getting at every object that's "live" -- i.e., every object that should not be garbage collected. If an object isn't in a tree, then you can safely discard it.
There's a fairly simple recipe for coming up with the root set; I don't have time to look it up right now but it's in the Hotspot documentation somewhere, if not the JVM specification. Basically the root set includes all the variables in all the frames on all the stacks of all the threads, plus all the static variables of all loaded classes, plus the
string pool contents. There may be other things I've forgotten, but you get the idea. It's not a difficult thing.