Any time anything runs (the GC, Diablo III, your browser running Facebook apps), they take time away from the CPU. When the GC runs, it can slow/pause things in your application, but depending on what your application does, you may never notice.
As I understand it, the GC is pretty well tuned. It doesn't run for no reason. If the JVM is running it a lot, you have to ask yourself if you are creating/releasing a bunch of stuff unnecessarily.
There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors
GCs can cause the JVM to pause for a while. When the heap is large enough (several GBs) this effect is quite predominant and can even be observed over a few seconds. Yes it is bad for performance. Unless your code churns out many objects for the GC to collect, the GC should not run all that frequently. There are GCs that do not stop the world all at once, but even they have several phases (like compaction) that can take a long time.