The usual answer is that by using buffered streams you avoid extra calls to the operating system disk file methods. Operating system calls are more expensive time-wise than within JVM calls.
A disadvantage of a buffered output stream could occur if you are logging important data. If your program crashes before the buffer is actually committed to the file, you lose that data.
Note that operating systems also do their own buffering, pre-fetching, etc etc and disk drives have their own hardware buffers so experimentation with a particular system is required to determine exactly how performance is affected.
Whenever you make an OS function call (sometimes known as a supervisor interrupt or supervisor call or kernel call), you're doing a major environment switch. Internal hardware states of the CPU are swapped in and out, the actual virtual memory management tables may be altered, various bits of system overhead will be attended to, and control will pass over to the I/O subsystem. Physical I/O is MUCH slower than logical I/O - we're talking thousands, maybe 10s of thousands of times slower. In modern-day OS's using a fixed-block disk architecture, there will usually still be some buffering inside the OS itself, and typically the request will be queued so that the actual I/O can be done using external I/O processors instead of loading down the CPU, but it's still a long, slow process, so doing it it efficiently is important.
Then, on top of that, once the supervisor request is done, the OS kernel will examine the system task queue and determine which task it thinks needs attention. Which probably won't be yours, so there will be further delays before your app comes to the front of the queue again.
So buffering is very important. Just remember to flush the buffers when writing. Because sooner or later, you will need to get the data written to disk or it will all get lost!
I haven't done any measurements on recent-vintage OS's, but on the old IBM mainframes, fully half the time spent executing in some of our apps was spent in the OS itself, not the apps themselves. And that was after we did the stock optimizations.
An IDE is no substitute for an Intelligent Developer.