One of the limits of GridBagLayout is that it lays out its components in columns. This makes it difficult to layout varying numbers of components in different rows. One technique is to add components to a JPanel for some/all rows. In this case you could use a JPanel for each of the three rows and add the JButtons to the panels with FlowLayout, GridBagLayout or Box/BoxLayout. Then add the three panels to the parent container with GridBagLayout. In the code below I used a separate JPanel for the second and third rows.
Using a single instance of GridBagLayout for all three containers is okay as long as you don't need to go back and query/change any of the layouts during runtime. Using a single instance of GridBagConstraints requires care to adjust the constraints properly for each component that uses it. Some people seem more comfortable with using a new instance for each component. This is demonstrated in the 27 July 2004 tech tip article. Also, and especially in the beginning, it may seem easier to use the 'gridx' and 'gridy' constraints. I started this way and later learned to use the 'gridwidth' and 'gridheight' constraints from the api and find they are easier to work with, especially in complex layouts such as a calculator gui. GridBag seems like the mystery: we each have to find our own way.