1. 04 Jul, 2024 40 commits
    • Jan Kara's avatar
      readahead: properly shorten readahead when falling back to do_page_cache_ra() · 7c877586
      Jan Kara authored
      When we succeed in creating some folios in page_cache_ra_order() but then
      need to fallback to single page folios, we don't shorten the amount to
      read passed to do_page_cache_ra() by the amount we've already read.  This
      then results in reading more and also in placing another readahead mark in
      the middle of the readahead window which confuses readahead code.  Fix the
      problem by properly reducing number of pages to read.
      
      Link: https://lkml.kernel.org/r/20240625101909.12234-3-jack@suse.czSigned-off-by: default avatarJan Kara <jack@suse.cz>
      Reviewed-by: default avatarJosef Bacik <josef@toxicpanda.com>
      Tested-by: default avatarZhang Peng <zhangpengpeng0808@gmail.com>
      Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      7c877586
    • Jan Kara's avatar
      filemap: fix page_cache_next_miss() when no hole found · 901a269f
      Jan Kara authored
      page_cache_next_miss() should return value outside of the specified range
      when no hole is found.  However currently it will return the last index
      *in* the specified range confusing ondemand_readahead() to think there's a
      hole in the searched range and upsetting readahead logic.
      
      Link: https://lkml.kernel.org/r/20240625101909.12234-2-jack@suse.czSigned-off-by: default avatarJan Kara <jack@suse.cz>
      Reviewed-by: default avatarJosef Bacik <josef@toxicpanda.com>
      Tested-by: default avatarZhang Peng <zhangpengpeng0808@gmail.com>
      Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      901a269f
    • Jan Kara's avatar
      readahead: make sure sync readahead reads needed page · 8051b82a
      Jan Kara authored
      Patch series "mm: Fix various readahead quirks".
      
      When we were internally testing performance of recent kernels, we have
      noticed quite variable performance of readahead arising from various
      quirks in readahead code.  So I went on a cleaning spree there.  This is a
      batch of patches resulting out of that.  A quick testing in my test VM
      with the following fio job file:
      
      [global]
      direct=0
      ioengine=sync
      invalidate=1
      blocksize=4k
      size=10g
      readwrite=read
      
      [reader]
      numjobs=1
      
      shows that this patch series improves the throughput from variable one in
      310-340 MB/s range to rather stable one at 350 MB/s.  As a side effect
      these cleanups also address the issue noticed by Bruz Zhang [1].
      
      [1] https://lore.kernel.org/all/20240618114941.5935-1-zhangpengpeng0808@gmail.com/
      
      Zhang Peng reported:
      
      : I test this batch of patch with fio, it indeed has a huge sppedup
      : in sequential read when block size is 4KiB. The result as follow,
      : for async read, iodepth is set to 128, and other settings
      : are self-evident.
      : 
      : casename                upstream   withFix speedup
      : ----------------        --------   -------- -------
      : randread-4k-sync        48991      47
      : seqread-4k-sync         1162758    14229
      : seqread-1024k-sync      1460208    1452522 
      : randread-4k-libaio      47467      4730
      : randread-4k-posixaio    49190      49512
      : seqread-4k-libaio       1085932    1234635
      : seqread-1024k-libaio    1423341    1402214 -1
      : seqread-4k-posixaio     1165084    1369613 1
      : seqread-1024k-posixaio  1435422    1408808 -1.8
      
      
      This patch (of 10):
      
      page_cache_sync_ra() is called when a folio we want to read is not in the
      page cache.  It is expected that it creates the folio (and perhaps the
      following folios as well) and submits reads for them unless some error
      happens.  However if index == ra->start + ra->size, ondemand_readahead()
      will treat the call as another async readahead hit.  Thus ra->start will
      be advanced and we create pages and queue reads from ra->start + ra->size
      further.  Consequentially the page at 'index' is not created and
      filemap_get_pages() has to always go through filemap_create_folio() path.
      
      This behavior has particularly unfortunate consequences when we have two
      IO threads sequentially reading from a shared file (as is the case when
      NFS serves sequential reads).  In that case what can happen is:
      
      suppose ra->size == ra->async_size == 128, ra->start = 512
      
      T1					T2
      reads 128 pages at index 512
        - hits async readahead mark
          filemap_readahead()
            ondemand_readahead()
              if (index == expected ...)
      	  ra->start = 512 + 128 = 640
                ra->size = 128
      	  ra->async_size = 128
      	page_cache_ra_order()
      	  blocks in ra_alloc_folio()
      					reads 128 pages at index 640
      					  - no page found
      					  page_cache_sync_readahead()
      					    ondemand_readahead()
      					      if (index == expected ...)
      						ra->start = 640 + 128 = 768
      						ra->size = 128
      						ra->async_size = 128
      					    page_cache_ra_order()
      					      submits reads from 768
      					  - still no page found at index 640
      					    filemap_create_folio()
      					  - goes on to index 641
      					  page_cache_sync_readahead()
      					    ondemand_readahead()
      					      - founds ra is confused,
      					        trims is to small size
        	  finds pages were already inserted
      
      And as a result read performance suffers.
      
      Fix the problem by triggering async readahead case in ondemand_readahead()
      only if we are calling the function because we hit the readahead marker. 
      In any other case we need to read the folio at 'index' and thus we cannot
      really use the current ra state.
      
      Note that the above situation could be viewed as a special case of
      file->f_ra state corruption.  In fact two thread reading using the shared
      file can also seemingly corrupt file->f_ra in interesting ways due to
      concurrent access.  I never saw that in practice and the fix is going to
      be much more complex so for now at least fix this practical problem while
      we ponder about the theoretically correct solution.
      
      Link: https://lkml.kernel.org/r/20240625100859.15507-1-jack@suse.cz
      Link: https://lkml.kernel.org/r/20240625101909.12234-1-jack@suse.czSigned-off-by: default avatarJan Kara <jack@suse.cz>
      Reviewed-by: default avatarJosef Bacik <josef@toxicpanda.com>
      Tested-by: default avatarZhang Peng <zhangpengpeng0808@gmail.com>
      Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      8051b82a
    • David Hildenbrand's avatar
      mm/migrate: move NUMA hinting fault folio isolation + checks under PTL · ee86814b
      David Hildenbrand authored
      Currently we always take a folio reference even if migration will not even
      be tried or isolation failed, requiring us to grab+drop an additional
      reference.
      
      Further, we end up calling folio_likely_mapped_shared() while the folio
      might have already been unmapped, because after we dropped the PTL, that
      can easily happen.  We want to stop touching mapcounts and friends from
      such context, and only call folio_likely_mapped_shared() while the folio
      is still mapped: mapcount information is pretty much stale and unreliable
      otherwise.
      
      So let's move checks into numamigrate_isolate_folio(), rename that
      function to migrate_misplaced_folio_prepare(), and call that function from
      callsites where we call migrate_misplaced_folio(), but still with the PTL
      held.
      
      We can now stop taking temporary folio references, and really only take a
      reference if folio isolation succeeded.  Doing the
      folio_likely_mapped_shared() + folio isolation under PT lock is now
      similar to how we handle MADV_PAGEOUT.
      
      While at it, combine the folio_is_file_lru() checks.
      
      [david@redhat.com: fix list_del() corruption]
        Link: https://lkml.kernel.org/r/8f85c31a-e603-4578-bf49-136dae0d4b69@redhat.com
        Link: https://lkml.kernel.org/r/20240626191129.658CFC32782@smtp.kernel.org
      Link: https://lkml.kernel.org/r/20240620212935.656243-3-david@redhat.comSigned-off-by: default avatarDavid Hildenbrand <david@redhat.com>
      Reviewed-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
      Reviewed-by: default avatarZi Yan <ziy@nvidia.com>
      Tested-by: default avatarDonet Tom <donettom@linux.ibm.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      ee86814b
    • David Hildenbrand's avatar
      mm/migrate: make migrate_misplaced_folio() return 0 on success · 4b88c23a
      David Hildenbrand authored
      Patch series "mm/migrate: move NUMA hinting fault folio isolation + checks
      under PTL".
      
      
      Let's just return 0 on success, which is less confusing.
      
      ...  especially because we got it wrong in the migrate.h stub where we
      have "return -EAGAIN; /* can't migrate now */" instead of "return 0;". 
      Likely this wrong return value doesn't currently matter, but it certainly
      adds confusion.
      
      We'll add migrate_misplaced_folio_prepare() next, where we want to use the
      same "return 0 on success" approach, so let's just clean this up.
      
      Link: https://lkml.kernel.org/r/20240620212935.656243-1-david@redhat.com
      Link: https://lkml.kernel.org/r/20240620212935.656243-2-david@redhat.comSigned-off-by: default avatarDavid Hildenbrand <david@redhat.com>
      Reviewed-by: default avatarZi Yan <ziy@nvidia.com>
      Reviewed-by: default avatarBaolin Wang <baolin.wang@linux.alibaba.com>
      Cc: Donet Tom <donettom@linux.ibm.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      4b88c23a
    • Tetsuo Handa's avatar
      mm: mmap_lock: replace get_memcg_path_buf() with on-stack buffer · 7d6be67c
      Tetsuo Handa authored
      Commit 2b5067a8 ("mm: mmap_lock: add tracepoints around lock
      acquisition") introduced TRACE_MMAP_LOCK_EVENT() macro using
      preempt_disable() in order to let get_mm_memcg_path() return a percpu
      buffer exclusively used by normal, softirq, irq and NMI contexts
      respectively.
      
      Commit 832b5072 ("mm: mmap_lock: use local locks instead of disabling
      preemption") replaced preempt_disable() with local_lock(&memcg_paths.lock)
      based on an argument that preempt_disable() has to be avoided because
      get_mm_memcg_path() might sleep if PREEMPT_RT=y.
      
      But syzbot started reporting
      
        inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
      
      and
      
        inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage.
      
      messages, for local_lock() does not disable IRQ.
      
      We could replace local_lock() with local_lock_irqsave() in order to
      suppress these messages.  But this patch instead replaces percpu buffers
      with on-stack buffer, for the size of each buffer returned by
      get_memcg_path_buf() is only 256 bytes which is tolerable for allocating
      from current thread's kernel stack memory.
      
      Link: https://lkml.kernel.org/r/ef22d289-eadb-4ed9-863b-fbc922b33d8d@I-love.SAKURA.ne.jpReported-by: default avatarsyzbot <syzbot+40905bca570ae6784745@syzkaller.appspotmail.com>
      Closes: https://syzkaller.appspot.com/bug?extid=40905bca570ae6784745
      Fixes: 832b5072 ("mm: mmap_lock: use local locks instead of disabling preemption")
      Signed-off-by: default avatarTetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
      Reviewed-by: default avatarAxel Rasmussen <axelrasmussen@google.com>
      Cc: Nicolas Saenz Julienne <nsaenzju@redhat.com>
      Cc: <stable@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      7d6be67c
    • Ilya Leoshkevich's avatar
      kmsan: do not pass NULL pointers as 0 · b1a80f4b
      Ilya Leoshkevich authored
      sparse complains about passing NULL pointers as 0.  Fix all instances.
      
      Link: https://lkml.kernel.org/r/20240627145754.27333-3-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reported-by: default avatarkernel test robot <lkp@intel.com>
      Closes: https://lore.kernel.org/oe-kbuild-all/202406272033.KejtfLkw-lkp@intel.com/Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Marco Elver <elver@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      b1a80f4b
    • Ilya Leoshkevich's avatar
      kmsan: add missing __user tags · b072880d
      Ilya Leoshkevich authored
      sparse complains that __user pointers are being passed to functions that
      expect non-__user ones.  In all cases, these functions are in fact working
      with user pointers, only the tag is missing.  Add it.
      
      Link: https://lkml.kernel.org/r/20240627145754.27333-2-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reported-by: default avatarkernel test robot <lkp@intel.com>
      Closes: https://lore.kernel.org/oe-kbuild-all/202406272033.KejtfLkw-lkp@intel.com/Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Marco Elver <elver@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      b072880d
    • Ilya Leoshkevich's avatar
      kmsan: enable on s390 · 3a8f6f3b
      Ilya Leoshkevich authored
      Now that everything else is in place, enable KMSAN in Kconfig.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-39-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      3a8f6f3b
    • Ilya Leoshkevich's avatar
      s390/kmsan: implement the architecture-specific functions · 2a48c8c9
      Ilya Leoshkevich authored
      arch_kmsan_get_meta_or_null() finds the lowcore shadow by querying the
      prefix and calling kmsan_get_metadata() again.
      
      kmsan_virt_addr_valid() delegates to virt_addr_valid().
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-38-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Acked-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      2a48c8c9
    • Ilya Leoshkevich's avatar
      s390/unwind: disable KMSAN checks · 7e17eac2
      Ilya Leoshkevich authored
      The unwind code can read uninitialized frames.  Furthermore, even in the
      good case, KMSAN does not emit shadow for backchains.  Therefore disable
      it for the unwinding functions.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-37-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      7e17eac2
    • Ilya Leoshkevich's avatar
      s390/uaccess: add the missing linux/instrumented.h #include · e0bebfd6
      Ilya Leoshkevich authored
      uaccess.h uses instrument_get_user() and instrument_put_user(), which are
      defined in linux/instrumented.h.  Currently we get this header from
      somewhere else by accident; prefer to be explicit about it and include it
      directly.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-36-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Suggested-by: default avatarAlexander Potapenko <glider@google.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      e0bebfd6
    • Ilya Leoshkevich's avatar
      s390/uaccess: add KMSAN support to put_user() and get_user() · eb6efdfe
      Ilya Leoshkevich authored
      put_user() uses inline assembly with precise constraints, so Clang is in
      principle capable of instrumenting it automatically.  Unfortunately, one
      of the constraints contains a dereferenced user pointer, and Clang does
      not currently distinguish user and kernel pointers.  Therefore KMSAN
      attempts to access shadow for user pointers, which is not a right thing to
      do.
      
      An obvious fix to add __no_sanitize_memory to __put_user_fn() does not
      work, since it's __always_inline.  And __always_inline cannot be removed
      due to the __put_user_bad() trick.
      
      A different obvious fix of using the "a" instead of the "+Q" constraint
      degrades the code quality, which is very important here, since it's a hot
      path.
      
      Instead, repurpose the __put_user_asm() macro to define
      __put_user_{char,short,int,long}_noinstr() functions and mark them with
      __no_sanitize_memory.  For the non-KMSAN builds make them __always_inline
      in order to keep the generated code quality.  Also define
      __put_user_{char,short,int,long}() functions, which call the
      aforementioned ones and which *are* instrumented, because they call KMSAN
      hooks, which may be implemented as macros.
      
      The same applies to get_user() as well.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-35-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      eb6efdfe
    • Ilya Leoshkevich's avatar
      s390/traps: unpoison the kernel_stack_overflow()'s pt_regs · c1057a70
      Ilya Leoshkevich authored
      This is normally done by the generic entry code, but the
      kernel_stack_overflow() flow bypasses it.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-34-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      c1057a70
    • Ilya Leoshkevich's avatar
      s390/string: add KMSAN support · 05a6dde6
      Ilya Leoshkevich authored
      Add KMSAN support for the s390 implementations of the string functions. 
      Do this similar to how it's already done for KASAN, except that the
      optimized memset{16,32,64}() functions need to be disabled: it's important
      for KMSAN to know that they initialized something.
      
      The way boot code is built with regard to string functions is problematic,
      since most files think it's configured with sanitizers, but boot/string.c
      doesn't.  This creates various problems with the memset64() definitions,
      depending on whether the code is built with sanitizers or fortify.  This
      should probably be streamlined, but in the meantime resolve the issues by
      introducing the IN_BOOT_STRING_C macro, similar to the existing
      IN_ARCH_STRING_C macro.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-33-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      05a6dde6
    • Ilya Leoshkevich's avatar
      s390/mm: define KMSAN metadata for vmalloc and modules · 65ca73f9
      Ilya Leoshkevich authored
      The pages for the KMSAN metadata associated with most kernel mappings are
      taken from memblock by the common code.  However, vmalloc and module
      metadata needs to be defined by the architectures.
      
      Be a little bit more careful than x86: allocate exactly MODULES_LEN for
      the module shadow and origins, and then take 2/3 of vmalloc for the
      vmalloc shadow and origins.  This ensures that users passing small
      vmalloc= values on the command line do not cause module metadata
      collisions.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-32-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Acked-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      65ca73f9
    • Ilya Leoshkevich's avatar
      s390/irqflags: do not instrument arch_local_irq_*() with KMSAN · 1b301f5f
      Ilya Leoshkevich authored
      Lockdep generates the following false positives with KMSAN on s390x:
      
      [    6.063666] DEBUG_LOCKS_WARN_ON(lockdep_hardirqs_enabled())
      [         ...]
      [    6.577050] Call Trace:
      [    6.619637]  [<000000000690d2de>] check_flags+0x1fe/0x210
      [    6.665411] ([<000000000690d2da>] check_flags+0x1fa/0x210)
      [    6.707478]  [<00000000006cec1a>] lock_acquire+0x2ca/0xce0
      [    6.749959]  [<00000000069820ea>] _raw_spin_lock_irqsave+0xea/0x190
      [    6.794912]  [<00000000041fc988>] __stack_depot_save+0x218/0x5b0
      [    6.838420]  [<000000000197affe>] __msan_poison_alloca+0xfe/0x1a0
      [    6.882985]  [<0000000007c5827c>] start_kernel+0x70c/0xd50
      [    6.927454]  [<0000000000100036>] startup_continue+0x36/0x40
      
      Between trace_hardirqs_on() and `stosm __mask, 3` lockdep thinks that
      interrupts are on, but on the CPU they are still off.  KMSAN
      instrumentation takes spinlocks, giving lockdep a chance to see and
      complain about this discrepancy.
      
      KMSAN instrumentation is inserted in order to poison the __mask variable. 
      Disable instrumentation in the respective functions.  They are very small
      and it's easy to see that no important metadata updates are lost because
      of this.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-31-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      1b301f5f
    • Ilya Leoshkevich's avatar
      s390/ftrace: unpoison ftrace_regs in kprobe_ftrace_handler() · 0cfd60a6
      Ilya Leoshkevich authored
      s390 uses assembly code to initialize ftrace_regs and call
      kprobe_ftrace_handler().  Therefore, from the KMSAN's point of view,
      ftrace_regs is poisoned on kprobe_ftrace_handler() entry.  This causes
      KMSAN warnings when running the ftrace testsuite.
      
      Fix by trusting the assembly code and always unpoisoning ftrace_regs in
      kprobe_ftrace_handler().
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-30-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      0cfd60a6
    • Ilya Leoshkevich's avatar
      s390/diag: unpoison diag224() output buffer · 1f4cf639
      Ilya Leoshkevich authored
      Diagnose 224 stores 4k bytes, which currently cannot be deduced from the
      inline assembly constraints.  This leads to KMSAN false positives.
      
      Fix the constraints by using a 4k-sized struct instead of a raw pointer. 
      While at it, prettify them too.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-29-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Suggested-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      1f4cf639
    • Ilya Leoshkevich's avatar
      s390/cpumf: unpoison STCCTM output buffer · 81b6bde8
      Ilya Leoshkevich authored
      stcctm() uses the "Q" constraint for dest, therefore KMSAN does not
      understand that it fills multiple doublewords pointed to by dest, not just
      one.  This results in false positives.
      
      Unpoison the whole dest manually with kmsan_unpoison_memory().
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-28-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reported-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      81b6bde8
    • Ilya Leoshkevich's avatar
      s390/cpacf: unpoison the results of cpacf_trng() · 8c208bc5
      Ilya Leoshkevich authored
      Prevent KMSAN from complaining about buffers filled by cpacf_trng() being
      uninitialized.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-27-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Tested-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Acked-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      8c208bc5
    • Ilya Leoshkevich's avatar
      s390/checksum: add a KMSAN check · e1b1c7f9
      Ilya Leoshkevich authored
      Add a KMSAN check to the CKSM inline assembly, similar to how it was done
      for ASAN in commit e42ac778 ("s390/checksum: always use cksm
      instruction").
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-26-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Acked-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      e1b1c7f9
    • Ilya Leoshkevich's avatar
      s390/boot: add the KMSAN runtime stub · 008dead4
      Ilya Leoshkevich authored
      It should be possible to have inline functions in the s390 header files,
      which call kmsan_unpoison_memory().  The problem is that these header
      files might be included by the decompressor, which does not contain KMSAN
      runtime, causing linker errors.
      
      Not compiling these calls if __SANITIZE_MEMORY__ is not defined - either
      by changing kmsan-checks.h or at the call sites - may cause unintended
      side effects, since calling these functions from an uninstrumented code
      that is linked into the kernel is valid use case.
      
      One might want to explicitly distinguish between the kernel and the
      decompressor.  Checking for a decompressor-specific #define is quite
      heavy-handed, and will have to be done at all call sites.
      
      A more generic approach is to provide a dummy kmsan_unpoison_memory()
      definition.  This produces some runtime overhead, but only when building
      with CONFIG_KMSAN.  The benefit is that it does not disturb the existing
      KMSAN build logic and call sites don't need to be changed.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-25-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      008dead4
    • Ilya Leoshkevich's avatar
      s390: use a larger stack for KMSAN · 435dc41e
      Ilya Leoshkevich authored
      Adjust the stack size for the KMSAN-enabled kernel like it was done for
      the KASAN-enabled one in commit 7fef92cc ("s390/kasan: double the
      stack size").  Both tools have similar requirements.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-24-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      435dc41e
    • Ilya Leoshkevich's avatar
      s390/boot: turn off KMSAN · c5944a7e
      Ilya Leoshkevich authored
      All other sanitizers are disabled for boot as well.  While at it, add a
      comment explaining why we need this.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-23-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      c5944a7e
    • Ilya Leoshkevich's avatar
      kmsan: accept ranges starting with 0 on s390 · cd613bd6
      Ilya Leoshkevich authored
      On s390 the virtual address 0 is valid (current CPU's lowcore is mapped
      there), therefore KMSAN should not complain about it.
      
      Disable the respective check on s390.  There doesn't seem to be a Kconfig
      option to describe this situation, so explicitly check for s390.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-22-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      cd613bd6
    • Ilya Leoshkevich's avatar
      lib/zlib: unpoison DFLTCC output buffers · 89f42df6
      Ilya Leoshkevich authored
      The constraints of the DFLTCC inline assembly are not precise: they do not
      communicate the size of the output buffers to the compiler, so it cannot
      automatically instrument it.
      
      Add the manual kmsan_unpoison_memory() calls for the output buffers.  The
      logic is the same as in [1].
      
      [1] https://github.com/zlib-ng/zlib-ng/commit/1f5ddcc009ac3511e99fc88736a9e1a6381168c5
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-21-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reported-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      89f42df6
    • Ilya Leoshkevich's avatar
      mm: kfence: disable KMSAN when checking the canary · 4d7b5a2c
      Ilya Leoshkevich authored
      KMSAN warns about check_canary() accessing the canary.
      
      The reason is that, even though set_canary() is properly instrumented and
      sets shadow, slub explicitly poisons the canary's address range
      afterwards.
      
      Unpoisoning the canary is not the right thing to do: only check_canary()
      is supposed to ever touch it.  Instead, disable KMSAN checks around canary
      read accesses.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-20-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Tested-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      4d7b5a2c
    • Ilya Leoshkevich's avatar
      mm: slub: disable KMSAN when checking the padding bytes · adea9876
      Ilya Leoshkevich authored
      Even though the KMSAN warnings generated by memchr_inv() are suppressed by
      metadata_access_enable(), its return value may still be poisoned.
      
      The reason is that the last iteration of memchr_inv() returns `*start !=
      value ?  start : NULL`, where *start is poisoned.  Because of this,
      somewhat counterintuitively, the shadow value computed by
      visitSelectInst() is equal to `(uintptr_t)start`.
      
      One possibility to fix this, since the intention behind guarding
      memchr_inv() behind metadata_access_enable() is to touch poisoned metadata
      without triggering KMSAN, is to unpoison its return value.  However, this
      approach is too fragile.  So simply disable the KMSAN checks in the
      respective functions.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-19-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      adea9876
    • Ilya Leoshkevich's avatar
      mm: slub: let KMSAN access metadata · 0e9a8550
      Ilya Leoshkevich authored
      Building the kernel with CONFIG_SLUB_DEBUG and CONFIG_KMSAN causes KMSAN
      to complain about touching redzones in kfree().
      
      Fix by extending the existing KASAN-related metadata_access_enable() and
      metadata_access_disable() functions to KMSAN.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-18-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      0e9a8550
    • Ilya Leoshkevich's avatar
      kmsan: expose KMSAN_WARN_ON() · e6553e2f
      Ilya Leoshkevich authored
      KMSAN_WARN_ON() is required for implementing s390-specific KMSAN
      functions, but right now it's available only to the KMSAN internal
      functions.  Expose it to subsystems through <linux/kmsan.h>.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-17-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      e6553e2f
    • Ilya Leoshkevich's avatar
      kmsan: do not round up pg_data_t size · d1dac751
      Ilya Leoshkevich authored
      x86's alloc_node_data() rounds up node data size to PAGE_SIZE.  It's not
      explained why it's needed, but it's most likely for performance reasons,
      since the padding bytes are not used anywhere.  Some other architectures
      do it as well, e.g., mips rounds it up to the cache line size.
      
      kmsan_init_shadow() initializes metadata for each node data and assumes
      the x86 rounding, which does not match other architectures.  This may
      cause the range end to overshoot the end of available memory, in turn
      causing virt_to_page_or_null() in kmsan_init_alloc_meta_for_range() to
      return NULL, which leads to kernel panic shortly after.
      
      Since the padding bytes are not used, drop the rounding.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-16-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      d1dac751
    • Ilya Leoshkevich's avatar
      kmsan: use ALIGN_DOWN() in kmsan_get_metadata() · f6a202f3
      Ilya Leoshkevich authored
      Improve the readability by replacing the custom aligning logic with
      ALIGN_DOWN().  Unlike other places where a similar sequence is used, there
      is no size parameter that needs to be adjusted, so the standard macro
      fits.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-15-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      f6a202f3
    • Ilya Leoshkevich's avatar
      kmsan: support SLAB_POISON · f4168171
      Ilya Leoshkevich authored
      Avoid false KMSAN negatives with SLUB_DEBUG by allowing kmsan_slab_free()
      to poison the freed memory, and by preventing init_object() from
      unpoisoning new allocations by using __memset().
      
      There are two alternatives to this approach.  First, init_object() can be
      marked with __no_sanitize_memory.  This annotation should be used with
      great care, because it drops all instrumentation from the function, and
      any shadow writes will be lost.  Even though this is not a concern with
      the current init_object() implementation, this may change in the future.
      
      Second, kmsan_poison_memory() calls may be added after memset() calls. 
      The downside is that init_object() is called from free_debug_processing(),
      in which case poisoning will erase the distinction between simply
      uninitialized memory and UAF.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-14-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      f4168171
    • Ilya Leoshkevich's avatar
      kmsan: introduce memset_no_sanitize_memory() · 1fdb3c70
      Ilya Leoshkevich authored
      Add a wrapper for memset() that prevents unpoisoning.  This is useful for
      filling memory allocator redzones.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-13-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      1fdb3c70
    • Ilya Leoshkevich's avatar
      kmsan: allow disabling KMSAN checks for the current task · ec3e837d
      Ilya Leoshkevich authored
      Like for KASAN, it's useful to temporarily disable KMSAN checks around,
      e.g., redzone accesses.  Introduce kmsan_disable_current() and
      kmsan_enable_current(), which are similar to their KASAN counterparts.
      
      Make them reentrant in order to handle memory allocations in interrupt
      context.  Repurpose the allow_reporting field for this.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-12-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      ec3e837d
    • Ilya Leoshkevich's avatar
      kmsan: export panic_on_kmsan · f2d62702
      Ilya Leoshkevich authored
      When building the kmsan test as a module, modpost fails with the following
      error message:
      
          ERROR: modpost: "panic_on_kmsan" [mm/kmsan/kmsan_test.ko] undefined!
      
      Export panic_on_kmsan in order to improve the KMSAN usability for
      modules.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-11-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      f2d62702
    • Ilya Leoshkevich's avatar
      kmsan: expose kmsan_get_metadata() · 6b1709d4
      Ilya Leoshkevich authored
      Each s390 CPU has lowcore pages associated with it.  Each CPU sees its own
      lowcore at virtual address 0 through a hardware mechanism called
      prefixing.  Additionally, all lowcores are mapped to non-0 virtual
      addresses stored in the lowcore_ptr[] array.
      
      When lowcore is accessed through virtual address 0, one needs to resolve
      metadata for lowcore_ptr[raw_smp_processor_id()].
      
      Expose kmsan_get_metadata() to make it possible to do this from the arch
      code.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-10-iii@linux.ibm.comSigned-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      6b1709d4
    • Ilya Leoshkevich's avatar
      kmsan: remove an x86-specific #include from kmsan.h · 61849c89
      Ilya Leoshkevich authored
      Replace the x86-specific asm/pgtable_64_types.h #include with the
      linux/pgtable.h one, which all architectures have.
      
      While at it, sort the headers alphabetically for the sake of consistency
      with other KMSAN code.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-9-iii@linux.ibm.com
      Fixes: f80be457 ("kmsan: add KMSAN runtime core")
      Signed-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Suggested-by: default avatarHeiko Carstens <hca@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Alexander Gordeev <agordeev@linux.ibm.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      61849c89
    • Ilya Leoshkevich's avatar
      kmsan: remove a useless assignment from kmsan_vmap_pages_range_noflush() · e54024f0
      Ilya Leoshkevich authored
      The value assigned to prot is immediately overwritten on the next line
      with PAGE_KERNEL.  The right hand side of the assignment has no
      side-effects.
      
      Link: https://lkml.kernel.org/r/20240621113706.315500-8-iii@linux.ibm.com
      Fixes: b073d7f8 ("mm: kmsan: maintain KMSAN metadata for page operations")
      Signed-off-by: default avatarIlya Leoshkevich <iii@linux.ibm.com>
      Suggested-by: default avatarAlexander Gordeev <agordeev@linux.ibm.com>
      Reviewed-by: default avatarAlexander Potapenko <glider@google.com>
      Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Heiko Carstens <hca@linux.ibm.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
      Cc: <kasan-dev@googlegroups.com>
      Cc: Marco Elver <elver@google.com>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
      Cc: Pekka Enberg <penberg@kernel.org>
      Cc: Roman Gushchin <roman.gushchin@linux.dev>
      Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
      Cc: Sven Schnelle <svens@linux.ibm.com>
      Cc: Vasily Gorbik <gor@linux.ibm.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      e54024f0