My problem is that when I click on a button at the bottom, the key listener stops working.
KeyEvents are handled by the component that has focus. When you click on a button the button now has focus, not your panel. First of all you must use panel.setFocusable(true) to allow the panel to receive focus. Now in the ActionListener of your button you can add panel.requestFocusInWindow().
However, a better solution is to use
Key Bindings. This is how Swing components handle KeyEvents.