Samples are documented to be available every 0.667us, so in theory
the 8 sample deep FIFO should take 5.336us to fill.  However, during
thorough testing, it became apparent that filling the FIFO actually
takes closer to 12us.

Also take into consideration that udelay() can behave oddly i.e. not
delay for as long as requested.

  Suggested-by: Russell King <rmk+kernel@arm.linux.org.uk>:

    "IIRC, Linus recommends a x2 factor on delays, especially
     timeouts generated by these functions.
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>