This week's book giveaway is in the OCAJP 8 forum. We're giving away four copies of OCA Java SE 8 Programmer I Study Guide and have Edward Finegan & Robert Liguori on-line! See this thread for details.
Not going to be easy. For starters, the GMail check box you're talking about is a tristate component whereas JCheckBox has only two selection states, so you'd have to fist develop tristate functionality
If I were to target doing this, I would start with a JToggleButton or maybe a JButton, add tristate functionality (probably by extending DefaultButtonModel) and apply a custom icon wrapped around UIManager.getIcon(CheckBox.icon). The paintIcon implementation would query the button state and paint accordingly. A popup menu would house the selection conditions, and ActionListeners on the menu items would take care of the filtering to apply the selection status of the items in the list.