• Venkatesh Srinivas's avatar
    virtio_ring: shadow available ring flags & index · f277ec42
    Venkatesh Srinivas authored
    Improves cacheline transfer flow of available ring header.
    
    Virtqueues are implemented as a pair of rings, one producer->consumer
    avail ring and one consumer->producer used ring; preceding the
    avail ring in memory are two contiguous u16 fields -- avail->flags
    and avail->idx. A producer posts work by writing to avail->idx and
    a consumer reads avail->idx.
    
    The flags and idx fields only need to be written by a producer CPU
    and only read by a consumer CPU; when the producer and consumer are
    running on different CPUs and the virtio_ring code is structured to
    only have source writes/sink reads, we can continuously transfer the
    avail header cacheline between 'M' states between cores. This flow
    optimizes core -> core bandwidth on certain CPUs.
    
    (see: "Software Optimization Guide for AMD Family 15h Processors",
    Section 11.6; similar language appears in the 10h guide and should
    apply to CPUs w/ exclusive caches, using LLC as a transfer cache)
    
    Unfortunately the existing virtio_ring code issued reads to the
    avail->idx and read-modify-writes to avail->flags on the producer.
    
    This change shadows the flags and index fields in producer memory;
    the vring code now reads from the shadows and only ever writes to
    avail->flags and avail->idx, allowing the cacheline to transfer
    core -> core optimally.
    
    In a concurrent version of vring_bench, the time required for
    10,000,000 buffer checkout/returns was reduced by ~2% (average
    across many runs) on an AMD Piledriver (15h) CPU:
    
    (w/o shadowing):
     Performance counter stats for './vring_bench':
         5,451,082,016      L1-dcache-loads
         ...
           2.221477739 seconds time elapsed
    
    (w/ shadowing):
     Performance counter stats for './vring_bench':
         5,405,701,361      L1-dcache-loads
         ...
           2.168405376 seconds time elapsed
    
    The further away (in a NUMA sense) virtio producers and consumers are
    from each other, the more we expect to benefit. Physical implementations
    of virtio devices and implementations of virtio where the consumer polls
    vring avail indexes (vhost) should also benefit.
    Signed-off-by: default avatarVenkatesh Srinivas <venkateshs@google.com>
    Signed-off-by: default avatarMichael S. Tsirkin <mst@redhat.com>
    f277ec42
virtio_ring.c 23.9 KB