Yes, you can remove a component from the containment hierarchy. Yes, you can add a different button to the container as well. Whether or not that new button is in the same place as the old button is going to depend entirely on your layout and whether or not any state or constraints that are used by the layout are the same.
The two buttons happen to have a very similar size because the text is roughly the same size. If the buttons had vastly different sizes then obviously that would cause a problem since BorderLayout honors them.