We use two flags, one to indicate an invalidation is needed after
creating a new entry and one to indicate an invalidation is needed
after removing an entry. However we were testing the wrong flag
in the remove case.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>