It probably is based on some algorithm, but the specification doesn't mandate any particular order for a Hash based collection, so it is up to the implementation what order items are returned in. If you run your program on a different version of the JRE or a JRE from a different manufacturer you may (or may not) get a different order and it may (or may not) be consistent in the ordering.
HashMap, Hashtable and HashSet are not ordered collections.
In other words: they don't contain items in any particular order, like a List would. You can see for example a HashSet as a bag that contains items. The items are just jumbled up inside the bag, not in any particular order - if you put your hand in and take an item from the bag, you don't know which one you're going to take.
Don't write a program that relies on the items being picked out of the Set in any particular order, because the order is undefined. If the order is important, use another collection class (an ArrayList or LinkedList, for example, or a LinkedHashSet).