These devices are likely going to be reprogrammed when disconnected so
this avoid a whole bunch of commands attempt to remove and the add back
to the list.
Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
Tested-by: Zhengping Jiang <jiangzp@google.com>