Commit 18a5daf0 authored by Mateusz Guzik's avatar Mateusz Guzik Committed by Christian Brauner

vfs: move d_lockref out of the area used by RCU lookup

Stock kernel scales worse than FreeBSD when doing a 20-way stat(2) on
the same tmpfs-backed file.

According to perf top:
  38.09%  lockref_put_return
  26.08%  lockref_get_not_dead
  25.60%  __d_lookup_rcu
   0.89%  clear_bhb_loop

__d_lookup_rcu is participating in cacheline ping pong due to the
embedded name sharing a cacheline with lockref.

Moving it out resolves the problem:
  41.50%  lockref_put_return
  41.03%  lockref_get_not_dead
   1.54%  clear_bhb_loop

benchmark (will-it-scale, Sapphire Rapids, tmpfs, ops/s):
FreeBSD:7219334
before:	5038006
after:	7842883 (+55%)

One minor remark: the 'after' result is unstable, fluctuating in the
range ~7.8 mln to ~9 mln during different runs.
Signed-off-by: default avatarMateusz Guzik <mjguzik@gmail.com>
Link: https://lore.kernel.org/r/20240613001215.648829-3-mjguzik@gmail.comSigned-off-by: default avatarChristian Brauner <brauner@kernel.org>
parent 267574de
...@@ -89,13 +89,18 @@ struct dentry { ...@@ -89,13 +89,18 @@ struct dentry {
struct inode *d_inode; /* Where the name belongs to - NULL is struct inode *d_inode; /* Where the name belongs to - NULL is
* negative */ * negative */
unsigned char d_iname[DNAME_INLINE_LEN]; /* small names */ unsigned char d_iname[DNAME_INLINE_LEN]; /* small names */
/* --- cacheline 1 boundary (64 bytes) was 32 bytes ago --- */
/* Ref lookup also touches following */ /* Ref lookup also touches following */
struct lockref d_lockref; /* per-dentry lock and refcount */
const struct dentry_operations *d_op; const struct dentry_operations *d_op;
struct super_block *d_sb; /* The root of the dentry tree */ struct super_block *d_sb; /* The root of the dentry tree */
unsigned long d_time; /* used by d_revalidate */ unsigned long d_time; /* used by d_revalidate */
void *d_fsdata; /* fs-specific data */ void *d_fsdata; /* fs-specific data */
/* --- cacheline 2 boundary (128 bytes) --- */
struct lockref d_lockref; /* per-dentry lock and refcount
* keep separate from RCU lookup area if
* possible!
*/
union { union {
struct list_head d_lru; /* LRU list */ struct list_head d_lru; /* LRU list */
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment