Integer is a sub class of Number. Why we can't put a Integer where we can put a Number?
The compiler is merely doing what you asked for. You stated that the set type is for a wildcard with a lower bound, and then you assigned it a set that don't conform to the bound.
Keep in mind, we are not talking about the elements in the set. We are talking about the type that the set can handle. So, yes, you can add in Integer into a set designed for Numbers, but that doesn't make an integer set and a number set interchangeable.