• Chuck Lever's avatar
    NFSD: Optimize DRC bucket pruning · 8847ecc9
    Chuck Lever authored
    DRC bucket pruning is done by nfsd_cache_lookup(), which is part of
    every NFSv2 and NFSv3 dispatch (ie, it's done while the client is
    waiting).
    
    I added a trace_printk() in prune_bucket() to see just how long
    it takes to prune. Here are two ends of the spectrum:
    
     prune_bucket: Scanned 1 and freed 0 in 90 ns, 62 entries remaining
     prune_bucket: Scanned 2 and freed 1 in 716 ns, 63 entries remaining
    ...
     prune_bucket: Scanned 75 and freed 74 in 34149 ns, 1 entries remaining
    
    Pruning latency is noticeable on fast transports with fast storage.
    By noticeable, I mean that the latency measured here in the worst
    case is the same order of magnitude as the round trip time for
    cached server operations.
    
    We could do something like moving expired entries to an expired list
    and then free them later instead of freeing them right in
    prune_bucket(). But simply limiting the number of entries that can
    be pruned by a lookup is simple and retains more entries in the
    cache, making the DRC somewhat more effective.
    
    Comparison with a 70/30 fio 8KB 12 thread direct I/O test:
    
    Before:
    
      write: IOPS=61.6k, BW=481MiB/s (505MB/s)(14.1GiB/30001msec); 0 zone resets
    
    WRITE:
    	1848726 ops (30%)
    	avg bytes sent per op: 8340 avg bytes received per op: 136
    	backlog wait: 0.635158 	RTT: 0.128525 	total execute time: 0.827242 (milliseconds)
    
    After:
    
      write: IOPS=63.0k, BW=492MiB/s (516MB/s)(14.4GiB/30001msec); 0 zone resets
    
    WRITE:
    	1891144 ops (30%)
    	avg bytes sent per op: 8340 avg bytes received per op: 136
    	backlog wait: 0.616114 	RTT: 0.126842 	total execute time: 0.805348 (milliseconds)
    Signed-off-by: default avatarChuck Lever <chuck.lever@oracle.com>
    Signed-off-by: default avatarJ. Bruce Fields <bfields@redhat.com>
    8847ecc9
nfscache.c 16.9 KB