Let's wait the inflight permits before free it.

Link: https://lore.kernel.org/r/20201217141915.56989-10-jinpu.wang@cloud.ionos.comSigned-off-by: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Reviewed-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>