1. 10 Aug, 2023 13 commits
    • Darrick J. Wong's avatar
      xfs: create a big array data structure · 3934e8eb
      Darrick J. Wong authored
      Create a simple 'big array' data structure for storage of fixed-size
      metadata records that will be used to reconstruct a btree index.  For
      repair operations, the most important operations are append, iterate,
      and sort.
      
      Earlier implementations of the big array used linked lists and suffered
      from severe problems -- pinning all records in kernel memory was not a
      good idea and frequently lead to OOM situations; random access was very
      inefficient; and record overhead for the lists was unacceptably high at
      40-60%.
      
      Therefore, the big memory array relies on the 'xfile' abstraction, which
      creates a memfd file and stores the records in page cache pages.  Since
      the memfd is created in tmpfs, the memory pages can be pushed out to
      disk if necessary and we have a built-in usage limit of 50% of physical
      memory.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarKent Overstreet <kent.overstreet@linux.dev>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      3934e8eb
    • Darrick J. Wong's avatar
      xfs: use per-AG bitmaps to reap unused AG metadata blocks during repair · 014ad537
      Darrick J. Wong authored
      The AGFL repair code uses a series of bitmaps to figure out where there
      are OWN_AG blocks that are not claimed by the free space and rmap
      btrees.  These blocks become the new AGFL, and any overflow is reaped.
      The bitmaps current track xfs_fsblock_t even though we already know the
      AG number.
      
      In the last patch, we introduced a new bitmap "type" for tracking
      xfs_agblock_t extents.  Port the reaping code and the AGFL repair to use
      this new type, which makes it very obvious what we're tracking.  This
      also eliminates a bunch of unnecessary agblock <-> fsblock conversions.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      014ad537
    • Darrick J. Wong's avatar
      xfs: reap large AG metadata extents when possible · 1c7ce115
      Darrick J. Wong authored
      When we're freeing extents that have been set in a bitmap, break the
      bitmap extent into multiple sub-extents organized by fate, and reap the
      extents.  This enables us to dispose of old resources more efficiently
      than doing them block by block.
      
      While we're at it, rename the reaping functions to make it clear that
      they're reaping per-AG extents.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      1c7ce115
    • Darrick J. Wong's avatar
      xfs: allow scanning ranges of the buffer cache for live buffers · 9ed851f6
      Darrick J. Wong authored
      After an online repair, we need to invalidate buffers representing the
      blocks from the old metadata that we're replacing.  It's possible that
      parts of a tree that were previously cached in memory are no longer
      accessible due to media failure or other corruption on interior nodes,
      so repair figures out the old blocks from the reverse mapping data and
      scans the buffer cache directly.
      
      In other words, online fsck needs to find all the live (i.e. non-stale)
      buffers for a range of fsblocks so that it can invalidate them.
      
      Unfortunately, the current buffer cache code triggers asserts if the
      rhashtable lookup finds a non-stale buffer of a different length than
      the key we searched for.  For regular operation this is desirable, but
      for this repair procedure, we don't care since we're going to forcibly
      stale the buffer anyway.  Add an internal lookup flag to avoid the
      assert.  Skip buffers that are already XBF_STALE.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      9ed851f6
    • Darrick J. Wong's avatar
      xfs: rearrange xrep_reap_block to make future code flow easier · 77a1396f
      Darrick J. Wong authored
      Rearrange the logic inside xrep_reap_block to make it more obvious that
      crosslinked metadata blocks are handled differently.  Add a couple of
      tracepoints so that we can tell what's going on at the end of a btree
      rebuild operation.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      77a1396f
    • Darrick J. Wong's avatar
      xfs: use deferred frees to reap old btree blocks · 5fee784e
      Darrick J. Wong authored
      Use deferred frees (EFIs) to reap the blocks of a btree that we just
      replaced.  This helps us to shrink the window in which those old blocks
      could be lost due to a system crash, though we try to flush the EFIs
      every few hundred blocks so that we don't also overflow the transaction
      reservations during and after we commit the new btree.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      5fee784e
    • Darrick J. Wong's avatar
      xfs: only allow reaping of per-AG blocks in xrep_reap_extents · a55e0730
      Darrick J. Wong authored
      Now that we've refactored btree cursors to require the caller to pass in
      a perag structure, there are numerous problems in xrep_reap_extents if
      it's being called to reap extents for an inode metadata repair.  We
      don't have any repair functions that can do that, so drop the support
      for now.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      a55e0730
    • Darrick J. Wong's avatar
      xfs: only invalidate blocks if we're going to free them · 8e54e06b
      Darrick J. Wong authored
      When we're discarding old btree blocks after a repair, only invalidate
      the buffers for the ones that we're freeing -- if the metadata was
      crosslinked with another data structure, we don't want to touch it.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      8e54e06b
    • Darrick J. Wong's avatar
      xfs: move the post-repair block reaping code to a separate file · e06ef14b
      Darrick J. Wong authored
      Reaping blocks after a repair is a complicated affair involving a lot of
      rmap btree lookups and figuring out if we're going to unmap or free old
      metadata blocks that might be crosslinked.  Eventually, we will need to
      be able to reap per-AG metadata blocks, bmbt blocks from inode forks,
      garbage CoW staging extents, and (even later) blocks from btrees rooted
      in inodes.  This results in a lot of reaping code, so we might as well
      split that off while it's easy.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      e06ef14b
    • Darrick J. Wong's avatar
      xfs: cull repair code that will never get used · 86a46417
      Darrick J. Wong authored
      These two functions date from the era when I thought that we could
      rebuild btrees by creating an alternate root and adding records one by
      one.  In other words, they predate the btree bulk loader.  They're not
      necessary now, so remove them.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarDave Chinner <dchinner@redhat.com>
      86a46417
    • Darrick J. Wong's avatar
      MAINTAINERS: add Chandan Babu as XFS release manager · d6532904
      Darrick J. Wong authored
      I nominate Chandan Babu to take over release management for the upstream
      kernel's XFS code.  He has had sufficient experience merging backports
      to the 5.4 LTS tree, testing them, and sending them on to the LTS leads.
      
      NOTE: I am /not/ nominating Chandan to take on any of the other roles I
      have just dropped.  Bug triager, testing lead, and community manager are
      open positions that need to be filled.  There's also maintainer for
      supported LTS releases (4.14, 4.19, 5.10...).
      
      Cc: Chandan Babu R <chandan.babu@oracle.com>
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Acked-by: default avatarChandan Babu R <chandan.babu@oracle.com>
      Reviewed-by: default avatarCarlos Maiolino <cem@kernel.org>
      d6532904
    • Darrick J. Wong's avatar
      MAINTAINERS: drop me as XFS maintainer · d554046e
      Darrick J. Wong authored
      I burned out years ago trying to juggle the roles senior developer,
      reviewer, tester, triager (crappily), release manager, and (at times)
      manager liaison.  There's enough work here in this one subsystem for a
      team of 20 FT, but instead we're squeezed to half that.  I thought if I
      could hold on just a bit longer I could help to maintain the focus on
      long term development to improve the experience for users.  I was wrong.
      
      Nowadays, people working on XFS seem to spend most of their time on
      distro kernel backports and dealing with AI-generated corner case bug
      reports that aren't user reports.  Reviewing has become a nightmare of
      sifting through under-documented kernel code trying to decide if this
      new feature won't break all the other features.  Getting reviews is an
      unpleasant process of negotiating with demands for further cleanups,
      trying to figure out if a review comment is based in experience or
      unfamiliarity, and wondering if the silence means anything.
      
      For now, I will continue to review patches and will try to get online
      fsck, parent pointers, and realtime volume modernisation merged.
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      d554046e
    • Darrick J. Wong's avatar
      docs: add maintainer entry profile for XFS · 19e13b0a
      Darrick J. Wong authored
      Create a new document to list what I think are (within the scope of XFS)
      our shared goals and community roles.  Since I will be stepping down
      shortly, I feel it's important to write down somewhere all the hats that
      I have been wearing for the past six years.
      
      Also, document important extra details about how to contribute to XFS.
      
      Cc: corbet@lwn.net
      Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
      Reviewed-by: default avatarChandan Babu R <chandan.babu@oracle.com>
      19e13b0a
  2. 06 Aug, 2023 8 commits
    • Linus Torvalds's avatar
      Linux 6.5-rc5 · 52a93d39
      Linus Torvalds authored
      52a93d39
    • Linus Torvalds's avatar
      Merge tag 'v6.5-rc5.vfs.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs · 0108963f
      Linus Torvalds authored
      Pull vfs fixes from Christian Brauner:
      
       - Fix a wrong check for O_TMPFILE during RESOLVE_CACHED lookup
      
       - Clean up directory iterators and clarify file_needs_f_pos_lock()
      
      * tag 'v6.5-rc5.vfs.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
        fs: rely on ->iterate_shared to determine f_pos locking
        vfs: get rid of old '->iterate' directory operation
        proc: fix missing conversion to 'iterate_shared'
        open: make RESOLVE_CACHED correctly test for O_TMPFILE
      0108963f
    • Christian Brauner's avatar
      fs: rely on ->iterate_shared to determine f_pos locking · 7d84d1b9
      Christian Brauner authored
      Now that we removed ->iterate we don't need to check for either
      ->iterate or ->iterate_shared in file_needs_f_pos_lock(). Simply check
      for ->iterate_shared instead. This will tell us whether we need to
      unconditionally take the lock. Not just does it allow us to avoid
      checking f_inode's mode it also actually clearly shows that we're
      locking because of readdir.
      Signed-off-by: default avatarChristian Brauner <brauner@kernel.org>
      7d84d1b9
    • Linus Torvalds's avatar
      vfs: get rid of old '->iterate' directory operation · 3e327154
      Linus Torvalds authored
      All users now just use '->iterate_shared()', which only takes the
      directory inode lock for reading.
      
      Filesystems that never got convered to shared mode now instead use a
      wrapper that drops the lock, re-takes it in write mode, calls the old
      function, and then downgrades the lock back to read mode.
      
      This way the VFS layer and other callers no longer need to care about
      filesystems that never got converted to the modern era.
      
      The filesystems that use the new wrapper are ceph, coda, exfat, jfs,
      ntfs, ocfs2, overlayfs, and vboxsf.
      
      Honestly, several of them look like they really could just iterate their
      directories in shared mode and skip the wrapper entirely, but the point
      of this change is to not change semantics or fix filesystems that
      haven't been fixed in the last 7+ years, but to finally get rid of the
      dual iterators.
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: default avatarChristian Brauner <brauner@kernel.org>
      3e327154
    • Linus Torvalds's avatar
      proc: fix missing conversion to 'iterate_shared' · 0a2c2baa
      Linus Torvalds authored
      I'm looking at the directory handling due to the discussion about f_pos
      locking (see commit 79796425: "file: reinstate f_pos locking
      optimization for regular files"), and wanting to clean that up.
      
      And one source of ugliness is how we were supposed to move filesystems
      over to the '->iterate_shared()' function that only takes the inode lock
      for reading many many years ago, but several filesystems still use the
      bad old '->iterate()' that takes the inode lock for exclusive access.
      
      See commit 61922694 ("introduce a parallel variant of ->iterate()")
      that also added some documentation stating
      
            Old method is only used if the new one is absent; eventually it will
            be removed.  Switch while you still can; the old one won't stay.
      
      and that was back in April 2016.  Here we are, many years later, and the
      old version is still clearly sadly alive and well.
      
      Now, some of those old style iterators are probably just because the
      filesystem may end up having per-inode mutable data that it uses for
      iterating a directory, but at least one case is just a mistake.
      
      Al switched over most filesystems to use '->iterate_shared()' back when
      it was introduced.  In particular, the /proc filesystem was converted as
      one of the first ones in commit f50752ea ("switch all procfs
      directories ->iterate_shared()").
      
      But then later one new user of '->iterate()' was then re-introduced by
      commit 6d9c939d ("procfs: add smack subdir to attrs").
      
      And that's clearly not what we wanted, since that new case just uses the
      same 'proc_pident_readdir()' and 'proc_pident_lookup()' helper functions
      that other /proc pident directories use, and they are most definitely
      safe to use with the inode lock held shared.
      
      So just fix it.
      
      This still leaves a fair number of oddball filesystems using the
      old-style directory iterator (ceph, coda, exfat, jfs, ntfs, ocfs2,
      overlayfs, and vboxsf), but at least we don't have any remaining in the
      core filesystems.
      
      I'm going to add a wrapper function that just drops the read-lock and
      takes it as a write lock, so that we can clean up the core vfs layer and
      make all the ugly 'this filesystem needs exclusive inode locking' be
      just filesystem-internal warts.
      
      I just didn't want to make that conversion when we still had a core user
      left.
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      Signed-off-by: default avatarChristian Brauner <brauner@kernel.org>
      0a2c2baa
    • Aleksa Sarai's avatar
      open: make RESOLVE_CACHED correctly test for O_TMPFILE · a0fc452a
      Aleksa Sarai authored
      O_TMPFILE is actually __O_TMPFILE|O_DIRECTORY. This means that the old
      fast-path check for RESOLVE_CACHED would reject all users passing
      O_DIRECTORY with -EAGAIN, when in fact the intended test was to check
      for __O_TMPFILE.
      
      Cc: stable@vger.kernel.org # v5.12+
      Fixes: 99668f61 ("fs: expose LOOKUP_CACHED through openat2() RESOLVE_CACHED")
      Signed-off-by: default avatarAleksa Sarai <cyphar@cyphar.com>
      Message-Id: <20230806-resolve_cached-o_tmpfile-v1-1-7ba16308465e@cyphar.com>
      Signed-off-by: default avatarChristian Brauner <brauner@kernel.org>
      a0fc452a
    • Linus Torvalds's avatar
      Merge tag 'rust-fixes-6.5-rc5' of https://github.com/Rust-for-Linux/linux · f0ab9f34
      Linus Torvalds authored
      Pull rust fixes from Miguel Ojeda:
      
       - Allocator: prevent mis-aligned allocation
      
       - Types: delete 'ForeignOwnable::borrow_mut'. A sound replacement is
         planned for the merge window
      
       - Build: fix bindgen error with UBSAN_BOUNDS_STRICT
      
      * tag 'rust-fixes-6.5-rc5' of https://github.com/Rust-for-Linux/linux:
        rust: fix bindgen build error with UBSAN_BOUNDS_STRICT
        rust: delete `ForeignOwnable::borrow_mut`
        rust: allocator: Prevent mis-aligned allocation
      f0ab9f34
    • Linus Torvalds's avatar
      Merge tag 'ata-6.5-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/dlemoal/libata · fb0d9199
      Linus Torvalds authored
      Pull ata fix from Damien Le Moal:
      
       - Prevent the scsi disk driver from issuing a START STOP UNIT command
         for ATA devices during system resume as this causes various issues
         reported by multiple users.
      
      * tag 'ata-6.5-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/dlemoal/libata:
        ata,scsi: do not issue START STOP UNIT on resume
      fb0d9199
  3. 05 Aug, 2023 5 commits
  4. 04 Aug, 2023 13 commits
  5. 03 Aug, 2023 1 commit