This week's book giveaway is in the OCAJP 8 forum. We're giving away four copies of OCA Java SE 8 Programmer I Study Guide and have Edward Finegan & Robert Liguori on-line! See this thread for details.
I have no idea what that code is supposed to be doing, but it doesn't appear to deal with whether cells can be edited or not. I would do this: override the isCellEditable method of the TableModel. Have it return false if it's referring to an unchecked checkbox and three other checkboxes are checked, and true otherwise.