This week's book giveaway is in the OO, Patterns, UML and Refactoring forum. We're giving away four copies of Refactoring for Software Design Smells: Managing Technical Debt and have Girish Suryanarayana, Ganesh Samarthyam & Tushar Sharma on-line! See this thread for details.
Since it does not have any child components, it's preferred size is 0x0. BorderLayout does maintain the preferred height for the NORTH and SOUTH components, and the preferred width for the WEST and EAST components. The CENTER component gets what's left. So your frame has a panel the size of the button at the top, a panel with height 0 at the bottom, and nothing in the center.
Either give the panel a preferred size, or add it using BorderLayout.CENTER.