• Glauber Costa's avatar
    x86, paravirt: Add a global synchronization point for pvclock · 59aab522
    Glauber Costa authored
    In recent stress tests, it was found that pvclock-based systems
    could seriously warp in smp systems. Using ingo's time-warp-test.c,
    I could trigger a scenario as bad as 1.5mi warps a minute in some systems.
    (to be fair, it wasn't that bad in most of them). Investigating further, I
    found out that such warps were caused by the very offset-based calculation
    pvclock is based on.
    
    This happens even on some machines that report constant_tsc in its tsc flags,
    specially on multi-socket ones.
    
    Two reads of the same kernel timestamp at approx the same time, will likely
    have tsc timestamped in different occasions too. This means the delta we
    calculate is unpredictable at best, and can probably be smaller in a cpu
    that is legitimately reading clock in a forward ocasion.
    
    Some adjustments on the host could make this window less likely to happen,
    but still, it pretty much poses as an intrinsic problem of the mechanism.
    
    A while ago, I though about using a shared variable anyway, to hold clock
    last state, but gave up due to the high contention locking was likely
    to introduce, possibly rendering the thing useless on big machines. I argue,
    however, that locking is not necessary.
    
    We do a read-and-return sequence in pvclock, and between read and return,
    the global value can have changed. However, it can only have changed
    by means of an addition of a positive value. So if we detected that our
    clock timestamp is less than the current global, we know that we need to
    return a higher one, even though it is not exactly the one we compared to.
    
    OTOH, if we detect we're greater than the current time source, we atomically
    replace the value with our new readings. This do causes contention on big
    boxes (but big here means *BIG*), but it seems like a good trade off, since
    it provide us with a time source guaranteed to be stable wrt time warps.
    
    After this patch is applied, I don't see a single warp in time during 5 days
    of execution, in any of the machines I saw them before.
    Signed-off-by: default avatarGlauber Costa <glommer@redhat.com>
    Acked-by: default avatarZachary Amsden <zamsden@redhat.com>
    CC: Jeremy Fitzhardinge <jeremy@goop.org>
    CC: Avi Kivity <avi@redhat.com>
    CC: Marcelo Tosatti <mtosatti@redhat.com>
    CC: Zachary Amsden <zamsden@redhat.com>
    Signed-off-by: default avatarMarcelo Tosatti <mtosatti@redhat.com>
    Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@suse.de>
    
    (Cherry-picked from commit 489fb490)
    59aab522
pvclock.c 4.99 KB