Take a look at my proposals of the solution - I hope I understand your intention properly.
The first example is somewhat "brutal", simply invoking scheduler.shutdown() after cancelling tasks.
It seems to work well but I'm not sure whether it will indeed work correctly - this is a sort of "self-closing".
I have to consider this way in details yet because I'm afraid some unexpected after-effects. I strive to be as watchful as a hawk during concurrence programming.
But the latter is based on creation of my own cleaning service. It is the simplest but a bit more verbose example.