The instanceof operator tests whether its left-hand operand can be cast to the class named on its right and returns a boolean. If the left operand is an object of a subclass of the named class, it still returns true, and if you write null instanceof Foo, you always get false.
I have never seen getclass, but the Object#getClass() method returns the Class<?> object which that object was created from. If you call it on any non-null reference type you get a Class<?> object, but if you call it on a null reference you suffer a NullPointerException.
If you refer to the use of both in equals implementations, the difference is allowing sub types like Campbell said.
Let's take java.util.Date as an example. If the equals method is written like this then for any instance of, let's say, java.sql.Timestamp with the same return for getTime(), equals will return true.
If equals is written like this Now an instance of java.sql.Timestamp will never be equal to an instance of java.util.Date because their classes are different.
Now you may want to know why the second mechanism exists. The reason is simple: to keep to the contract of Object.equals. It says that if an object X equals another object Y, then Y must also equal X. Another example when this can be broken using instanceof:
Assume you have an instance of Point called p and an instance of ColorPoint called cp. Both have values for x and y of 0. Now p.equals(cp) because cp is also a Point with the same x and y values. However, cp.equals(p) is false since p is not a ColorPoint. If you would have used getClass() then p.equals(cp) would also return false since they have different classes.
In short, only use instanceof in the equals method if you cannot override it. Either make the method or the class final. Otherwise getClass() is the only way to go to guarantee that the contract for Object.equals is fulfilled.