Is it normal to use LayoutManagers and Containers? Yes, of course. The only alternative is to use a "null" LayoutManager and place components at absolute coordinates. That's a terrible idea because just resizing a window breaks it, not to mention how it will break as the GUI is run on different platforms.
How to divide a window into 4 panels with their own LayoutManager? Easy. Set the LayoutManager of the JFrame's content pane to a GridLayout or GridbagLayout, and add four JPanels to that grid. GridLayout is simpler, but GridBagLayout would let the four panel's size vary.
Then you can independently set the LayoutManager of each of the JPanels; they can all be different.
Why do your components "take up the whole panel?" Well, each LayoutManager has its own strategy for laying out components; some of these strategies do, indeed, involve expanding components to fill available space. You need to read the Javadoc for each LayoutManager you use, carefully. JFrames have a BorderLayout by default; JPanels have a FlowLayout by default.
Have a look at the often-neglected Box class and the BoxLayout, which is a sort of improved FlowLayout.