Since it does not have any child components, it's preferred size is 0x0. BorderLayout does maintain the preferred height for the NORTH and SOUTH components, and the preferred width for the WEST and EAST components. The CENTER component gets what's left. So your frame has a panel the size of the button at the top, a panel with height 0 at the bottom, and nothing in the center.
Either give the panel a preferred size, or add it using BorderLayout.CENTER.