ie if we do set.remove(k), we don't decrement k -- we decrement
whatever was in the set, which can be different.