Commit 3e79f082 authored by Aneesh Kumar K.V's avatar Aneesh Kumar K.V Committed by Michael Ellerman

libnvdimm/nvdimm/flush: Allow architecture to override the flush barrier

Architectures like ppc64 provide persistent memory specific barriers
that will ensure that all stores for which the modifications are
written to persistent storage by preceding dcbfps and dcbstps
instructions have updated persistent storage before any data
access or data transfer caused by subsequent instructions is initiated.
This is in addition to the ordering done by wmb()

Update nvdimm core such that architecture can use barriers other than
wmb to ensure all previous writes are architecturally visible for
the platform buffer flush.
Signed-off-by: default avatarAneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Reviewed-by: default avatarDan Williams <dan.j.williams@intel.com>
Signed-off-by: default avatarMichael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/20200701072235.223558-5-aneesh.kumar@linux.ibm.com
parent d3580427
...@@ -1935,6 +1935,20 @@ There are some more advanced barrier functions: ...@@ -1935,6 +1935,20 @@ There are some more advanced barrier functions:
relaxed I/O accessors and the Documentation/DMA-API.txt file for more relaxed I/O accessors and the Documentation/DMA-API.txt file for more
information on consistent memory. information on consistent memory.
(*) pmem_wmb();
This is for use with persistent memory to ensure that stores for which
modifications are written to persistent storage reached a platform
durability domain.
For example, after a non-temporal write to pmem region, we use pmem_wmb()
to ensure that stores have reached a platform durability domain. This ensures
that stores have updated persistent storage before any data access or
data transfer caused by subsequent instructions is initiated. This is
in addition to the ordering done by wmb().
For load from persistent memory, existing read memory barriers are sufficient
to ensure read ordering.
=============================== ===============================
IMPLICIT KERNEL MEMORY BARRIERS IMPLICIT KERNEL MEMORY BARRIERS
......
...@@ -536,7 +536,7 @@ static void ssd_commit_superblock(struct dm_writecache *wc) ...@@ -536,7 +536,7 @@ static void ssd_commit_superblock(struct dm_writecache *wc)
static void writecache_commit_flushed(struct dm_writecache *wc, bool wait_for_ios) static void writecache_commit_flushed(struct dm_writecache *wc, bool wait_for_ios)
{ {
if (WC_MODE_PMEM(wc)) if (WC_MODE_PMEM(wc))
wmb(); pmem_wmb();
else else
ssd_commit_flushed(wc, wait_for_ios); ssd_commit_flushed(wc, wait_for_ios);
} }
......
...@@ -1206,13 +1206,13 @@ int generic_nvdimm_flush(struct nd_region *nd_region) ...@@ -1206,13 +1206,13 @@ int generic_nvdimm_flush(struct nd_region *nd_region)
idx = this_cpu_add_return(flush_idx, hash_32(current->pid + idx, 8)); idx = this_cpu_add_return(flush_idx, hash_32(current->pid + idx, 8));
/* /*
* The first wmb() is needed to 'sfence' all previous writes * The pmem_wmb() is needed to 'sfence' all
* such that they are architecturally visible for the platform * previous writes such that they are architecturally visible for
* buffer flush. Note that we've already arranged for pmem * the platform buffer flush. Note that we've already arranged for pmem
* writes to avoid the cache via memcpy_flushcache(). The final * writes to avoid the cache via memcpy_flushcache(). The final
* wmb() ensures ordering for the NVDIMM flush write. * wmb() ensures ordering for the NVDIMM flush write.
*/ */
wmb(); pmem_wmb();
for (i = 0; i < nd_region->ndr_mappings; i++) for (i = 0; i < nd_region->ndr_mappings; i++)
if (ndrd_get_flush_wpq(ndrd, i, 0)) if (ndrd_get_flush_wpq(ndrd, i, 0))
writeq(1, ndrd_get_flush_wpq(ndrd, i, idx)); writeq(1, ndrd_get_flush_wpq(ndrd, i, idx));
......
...@@ -257,5 +257,15 @@ do { \ ...@@ -257,5 +257,15 @@ do { \
}) })
#endif #endif
/*
* pmem_wmb() ensures that all stores for which the modification
* are written to persistent storage by preceding instructions have
* updated persistent storage before any data access or data transfer
* caused by subsequent instructions is initiated.
*/
#ifndef pmem_wmb
#define pmem_wmb() wmb()
#endif
#endif /* !__ASSEMBLY__ */ #endif /* !__ASSEMBLY__ */
#endif /* __ASM_GENERIC_BARRIER_H */ #endif /* __ASM_GENERIC_BARRIER_H */
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment