• Dennis Zhou's avatar
    btrfs: add zstd compression level support · 3f93aef5
    Dennis Zhou authored
    Zstd compression requires different amounts of memory for each level of
    compression. The prior patches implemented indirection to allow for each
    compression type to manage their workspaces independently. This patch
    uses this indirection to implement compression level support for zstd.
    
    To manage the additional memory require, each compression level has its
    own queue of workspaces. A global LRU is used to help with reclaim.
    Reclaim is done via a timer which provides a mechanism to decrease
    memory utilization by keeping only workspaces around that are sized
    appropriately. Forward progress is guaranteed by a preallocated max
    workspace hidden from the LRU.
    
    When getting a workspace, it uses a bitmap to identify the levels that
    are populated and scans up. If it finds a workspace that is greater than
    it, it uses it, but does not update the last_used time and the
    corresponding place in the LRU. If we hit memory pressure, we sleep on
    the max level workspace. We continue to rescan in case we can use a
    smaller workspace, but eventually should be able to obtain the max level
    workspace or allocate one again should memory pressure subside.
    
    The memory requirement for decompression is the same as level 1, and
    therefore can use any of available workspace.
    
    The number of workspaces is bound by an upper limit of the workqueue's
    limit which currently is 2 (percpu limit). The reclaim timer is used to
    free inactive/improperly sized workspaces and is set to 307s to avoid
    colliding with transaction commit (every 30s).
    
    Repeating the experiment from v2 [1], the Silesia corpus was copied to a
    btrfs filesystem 10 times and then read back after dropping the caches.
    The btrfs filesystem was on an SSD.
    
    Level   Ratio   Compression (MB/s)  Decompression (MB/s)  Memory (KB)
    1       2.658        438.47                910.51            780
    2       2.744        364.86                886.55           1004
    3       2.801        336.33                828.41           1260
    4       2.858        286.71                886.55           1260
    5       2.916        212.77                556.84           1388
    6       2.363        119.82                990.85           1516
    7       3.000        154.06                849.30           1516
    8       3.011        159.54                875.03           1772
    9       3.025        100.51                940.15           1772
    10      3.033        118.97                616.26           1772
    11      3.036         94.19                802.11           1772
    12      3.037         73.45                931.49           1772
    13      3.041         55.17                835.26           2284
    14      3.087         44.70                716.78           2547
    15      3.126         37.30                878.84           2547
    
    [1] https://lore.kernel.org/linux-btrfs/20181031181108.289340-1-terrelln@fb.com/
    
    Cc: Nick Terrell <terrelln@fb.com>
    Cc: Omar Sandoval <osandov@osandov.com>
    Signed-off-by: default avatarDennis Zhou <dennis@kernel.org>
    Reviewed-by: default avatarDavid Sterba <dsterba@suse.com>
    Signed-off-by: default avatarDavid Sterba <dsterba@suse.com>
    3f93aef5
zstd.c 18.8 KB