Also, is there a way to close the JPanel upon clicking the "Add" button? This is going into a game and it's pretty clunky for the user to have to click the X every time.
A JPanel can't be closed. However your mention of "click the X" suggests that you actually want to close the JFrame. So, going back a step, I'm going to guess that you want to use this code to prompt the user for something before you carry on. If that's the case then don't use a JFrame. Use a JDialog instead. JDialog is specifically designed to be a "dialog box" which goes away once the user clicks "OK" or whatever.