As of commit 9326e4f1 ("spi: Limit the spi device max speed to
controller's max speed"), the SPI device max speed is set to the
controller's max speed if it is larger. The Cadence QSPI controller does
not set the controller's max speed so it is left at its initial value of
0. This means the SPI device max speed is always set to 0.

The SPI device max speed is used to calculate the baud rate divider when
performing an operation. If this speed is 0, the default divider of 32
is used. No matter what speed is specified by the device tree property
'spi-max-frequency', the device will always operate at ref_clk / 32.

Fix this by setting master->max_speed_hz to the ref clock speed so the
correct divider can be calculated.
Signed-off-by: Pratyush Yadav <p.yadav@ti.com>
Link: https://lore.kernel.org/r/20201222184425.7028-2-p.yadav@ti.comSigned-off-by: Mark Brown <broonie@kernel.org>