Not going to be easy. For starters, the GMail check box you're talking about is a tristate component whereas JCheckBox has only two selection states, so you'd have to fist develop tristate functionality
If I were to target doing this, I would start with a JToggleButton or maybe a JButton, add tristate functionality (probably by extending DefaultButtonModel) and apply a custom icon wrapped around UIManager.getIcon(CheckBox.icon). The paintIcon implementation would query the button state and paint accordingly. A popup menu would house the selection conditions, and ActionListeners on the menu items would take care of the filtering to apply the selection status of the items in the list.