The constructor you used for TreeSet is new TreeSet<String>(comp), in Java API it says:
public TreeSet(Comparator<? super E> comparator)
Constructs a new, empty tree set, sorted according to the specified comparator.
And for the Comparator:
The ordering imposed by a comparator c on a set of elements S is said to be consistent with equals if and only if c.compare(e1, e2)==0 has the same boolean value as e1.equals(e2) for every e1 and e2 in S.
Since TreeSet implements SortSet, and in API SortSet notes:
Note that the ordering maintained by a sorted set (whether or not an explicit comparator is provided) must be consistent with equals if the sorted set is to correctly implement the Set interface. (See the Comparable interface or Comparator interface for a precise definition of consistent with equals.) This is so because the Set interface is defined in terms of the equals operation, but a sorted set performs all element comparisons using its compareTo (or compare) method, so two elements that are deemed equal by this method are, from the standpoint of the sorted set, equal. The behavior of a sorted set is well-defined even if its ordering is inconsistent with equals; it just fails to obey the general contract of the Set interface.
When trying to add "Third" to the TreeSet, it compares with existing element "First" and returns 0 and won't be able to sort the set, so it failed.
I have a question though, the TreeSet's add method says:
public boolean add(E e)
Adds the specified element to this set if it is not already present. More formally, adds the specified element e to this set if the set contains no element e2 such that (e==null ? e2==null : e.equals(e2)). If this set already contains the element, the call leaves the set unchanged and returns false.
It does not mention sort or compare stuff, so it seems as long as the new element fulfil the condition (e==null ? e2==null : e.equals(e2)), it should be able to be added. It seems contradictory on this example?