Use the fine-grained mutex for the assigned info object, instead.
Signed-off-by: Takashi Iwai <tiwai@suse.de>