Yes, well I figured out the actual problem, the UIManager can't apply the selectionBackground color on a static component. I made a simple window with just a menu and a checkbox inside it and what do you know it worked perfectly. So the problem now, is how to make the UIManager work on a static component.
Inventing stuff rarely works. I can't see what led you to believe that "CheckBoxMenuItem[Selected].checkIcon" would be recognized by JCheckBoxMenuItem as a relevant UIDefaults key.
You could read the source of the private static class javax.swing.plaf.metal.MetalIconFactory.CheckBoxMenuItemIcon to see how the Swing designers approached the issue.
As I see it, you have two primary options, with several variations possible:
1. Set the selected icon for each item in code, or create and use a custom subclass that does that in its constructor. You may need to experiment to determine which Icon of selected/pressed/rollover is used for the state in which you are interested.
2. Write an Icon implementation that wraps the two icons and decides which to paint based on the state of the component, queried after appropriate casting.