Commit 531724ab authored by Christoph Hellwig's avatar Christoph Hellwig Committed by Jens Axboe

block: avoid extra bio reference for async O_DIRECT

The bio referencing has a trick that doesn't do any actual atomic
inc/dec on the reference count until we have to elevator to > 1. For the
async IO O_DIRECT case, we can't use the simple DIO variants, so we use
__blkdev_direct_IO(). It always grabs an extra reference to the bio
after allocation, which means we then enter the slower path of actually
having to do atomic_inc/dec on the count.

We don't need to do that for the async case, unless we end up going
multi-bio, in which case we're already doing huge amounts of IO. For the
smaller IO case (< BIO_MAX_PAGES), we can do without the extra ref.

Based on an earlier patch (and commit log) from Jens Axboe.
Signed-off-by: default avatarChristoph Hellwig <hch@lst.de>
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
parent 27fae429
...@@ -302,6 +302,7 @@ static void blkdev_bio_end_io(struct bio *bio) ...@@ -302,6 +302,7 @@ static void blkdev_bio_end_io(struct bio *bio)
} }
dio->iocb->ki_complete(iocb, ret, 0); dio->iocb->ki_complete(iocb, ret, 0);
if (dio->multi_bio)
bio_put(&dio->bio); bio_put(&dio->bio);
} else { } else {
struct task_struct *waiter = dio->waiter; struct task_struct *waiter = dio->waiter;
...@@ -343,14 +344,15 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) ...@@ -343,14 +344,15 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages)
return -EINVAL; return -EINVAL;
bio = bio_alloc_bioset(GFP_KERNEL, nr_pages, &blkdev_dio_pool); bio = bio_alloc_bioset(GFP_KERNEL, nr_pages, &blkdev_dio_pool);
bio_get(bio); /* extra ref for the completion handler */
dio = container_of(bio, struct blkdev_dio, bio); dio = container_of(bio, struct blkdev_dio, bio);
dio->is_sync = is_sync = is_sync_kiocb(iocb); dio->is_sync = is_sync = is_sync_kiocb(iocb);
if (dio->is_sync) if (dio->is_sync) {
dio->waiter = current; dio->waiter = current;
else bio_get(bio);
} else {
dio->iocb = iocb; dio->iocb = iocb;
}
dio->size = 0; dio->size = 0;
dio->multi_bio = false; dio->multi_bio = false;
...@@ -400,6 +402,13 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) ...@@ -400,6 +402,13 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages)
} }
if (!dio->multi_bio) { if (!dio->multi_bio) {
/*
* AIO needs an extra reference to ensure the dio
* structure which is embedded into the first bio
* stays around.
*/
if (!is_sync)
bio_get(bio);
dio->multi_bio = true; dio->multi_bio = true;
atomic_set(&dio->ref, 2); atomic_set(&dio->ref, 2);
} else { } else {
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment