• Qu Wenruo's avatar
    btrfs: change timing for qgroup reserved space for ordered extents to fix reserved space leak · 7dbeaad0
    Qu Wenruo authored
    [BUG]
    The following simple workload from fsstress can lead to qgroup reserved
    data space leak:
      0/0: creat f0 x:0 0 0
      0/0: creat add id=0,parent=-1
      0/1: write f0[259 1 0 0 0 0] [600030,27288] 0
      0/4: dwrite - xfsctl(XFS_IOC_DIOINFO) f0[259 1 0 0 64 627318] return 25, fallback to stat()
      0/4: dwrite f0[259 1 0 0 64 627318] [610304,106496] 0
    
    This would cause btrfs qgroup to leak 20480 bytes for data reserved
    space.  If btrfs qgroup limit is enabled, such leak can lead to
    unexpected early EDQUOT and unusable space.
    
    [CAUSE]
    When doing direct IO, kernel will try to writeback existing buffered
    page cache, then invalidate them:
      generic_file_direct_write()
      |- filemap_write_and_wait_range();
      |- invalidate_inode_pages2_range();
    
    However for btrfs, the bi_end_io hook doesn't finish all its heavy work
    right after bio ends.  In fact, it delays its work further:
    
      submit_extent_page(end_io_func=end_bio_extent_writepage);
      end_bio_extent_writepage()
      |- btrfs_writepage_endio_finish_ordered()
         |- btrfs_init_work(finish_ordered_fn);
    
      <<< Work queue execution >>>
      finish_ordered_fn()
      |- btrfs_finish_ordered_io();
         |- Clear qgroup bits
    
    This means, when filemap_write_and_wait_range() returns,
    btrfs_finish_ordered_io() is not guaranteed to be executed, thus the
    qgroup bits for related range are not cleared.
    
    Now into how the leak happens, this will only focus on the overlapping
    part of buffered and direct IO part.
    
    1. After buffered write
       The inode had the following range with QGROUP_RESERVED bit:
       	596		616K
    	|///////////////|
       Qgroup reserved data space: 20K
    
    2. Writeback part for range [596K, 616K)
       Write back finished, but btrfs_finish_ordered_io() not get called
       yet.
       So we still have:
       	596K		616K
    	|///////////////|
       Qgroup reserved data space: 20K
    
    3. Pages for range [596K, 616K) get released
       This will clear all qgroup bits, but don't update the reserved data
       space.
       So we have:
       	596K		616K
    	|		|
       Qgroup reserved data space: 20K
       That number doesn't match the qgroup bit range anymore.
    
    4. Dio prepare space for range [596K, 700K)
       Qgroup reserved data space for that range, we got:
       	596K		616K			700K
    	|///////////////|///////////////////////|
       Qgroup reserved data space: 20K + 104K = 124K
    
    5. btrfs_finish_ordered_range() gets executed for range [596K, 616K)
       Qgroup free reserved space for that range, we got:
       	596K		616K			700K
    	|		|///////////////////////|
       We need to free that range of reserved space.
       Qgroup reserved data space: 124K - 20K = 104K
    
    6. btrfs_finish_ordered_range() gets executed for range [596K, 700K)
       However qgroup bit for range [596K, 616K) is already cleared in
       previous step, so we only free 84K for qgroup reserved space.
       	596K		616K			700K
    	|		|			|
       We need to free that range of reserved space.
       Qgroup reserved data space: 104K - 84K = 20K
    
       Now there is no way to release that 20K unless disabling qgroup or
       unmounting the fs.
    
    [FIX]
    This patch will change the timing of btrfs_qgroup_release/free_data()
    call.  Here it uses buffered COW write as an example.
    
    	The new timing			|	The old timing
    ----------------------------------------+---------------------------------------
     btrfs_buffered_write()			| btrfs_buffered_write()
     |- btrfs_qgroup_reserve_data() 	| |- btrfs_qgroup_reserve_data()
    					|
     btrfs_run_delalloc_range()		| btrfs_run_delalloc_range()
     |- btrfs_add_ordered_extent()  	|
        |- btrfs_qgroup_release_data()	|
           The reserved is passed into	|
           btrfs_ordered_extent structure	|
    					|
     btrfs_finish_ordered_io()		| btrfs_finish_ordered_io()
     |- The reserved space is passed to 	| |- btrfs_qgroup_release_data()
        btrfs_qgroup_record			|    The resereved space is passed
    					|    to btrfs_qgroup_recrod
    					|
     btrfs_qgroup_account_extents()		| btrfs_qgroup_account_extents()
     |- btrfs_qgroup_free_refroot()		| |- btrfs_qgroup_free_refroot()
    
    The point of such change is to ensure, when ordered extents are
    submitted, the qgroup reserved space is already released, to keep the
    timing aligned with file_write_and_wait_range().
    
    So that qgroup data reserved space is all bound to btrfs_ordered_extent
    and solve the timing mismatch.
    
    Fixes: f695fdce ("btrfs: qgroup: Introduce functions to release/free qgroup reserve data space")
    Suggested-by: default avatarJosef Bacik <josef@toxicpanda.com>
    Reviewed-by: default avatarJosef Bacik <josef@toxicpanda.com>
    Signed-off-by: default avatarQu Wenruo <wqu@suse.com>
    Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
    7dbeaad0
ordered-data.h 5.62 KB