Using perm_addr is always wrong, it may be reassigned by
anyone using standard netdev APIs. Remove that from the
match function and also use the match function where only
the perm_addr was used now.
Signed-off-by: Johannes Berg <johannes.berg@intel.com>