As part of ndo_vlan_rx_kill_vid() implementation,
qede is requesting firmware to remove the vlan filter.
This currently happens even if the vlan wasn't previously
added [In case device ran out of vlan credits].

For PFs this doesn't cause any issues as the firmware
would simply ignore the removal request. But for VFs their
parent PF is holding an accounting of the configured vlans,
and such a request would cause the PF to fail the VF's
removal request.

Simply fix this for both PFs & VFs and don't remove filters
that were not previously added.

Fixes: 7c1bfcad ("qede: Add vlan filtering offload support")
Signed-off-by: Yuval Mintz <Yuval.Mintz@qlogic.com>
Signed-off-by: David S. Miller <davem@davemloft.net>