net: dsa: mv88e6xxx: Don't modify RGMII delays when not RGMII mode

The RGMII modes delays can be set via strapping pings or EEPROM.
Don't change them unless explicitly asked to change them.  The recent
refactoring of setting the MAC configuration changed this behaviours,
in that CPU and DSA ports have any pre-configured RGMII delays
removed. This breaks the Armada 370RD board. Restore the previous
behaviour, in that RGMII delays are only applied/removed when
explicitly asked for via an phy-mode being PHY_INTERFACE_MODE_RGMII*

Fixes: 7340e5ecdbb1 ("net: dsa: mv88e6xxx: setup port's MAC")
Signed-off-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>

drivers/net/dsa/mv88e6xxx/port.c

@@ -63,9 +63,10 @@ static int mv88e6xxx_port_set_rgmii_delay(struct mv88e6xxx_chip *chip, int port,
 		reg |= PORT_PCS_CTRL_RGMII_DELAY_RXCLK |
 			PORT_PCS_CTRL_RGMII_DELAY_TXCLK;
 		break;
-	default:
-		/* no delay */
+	case PHY_INTERFACE_MODE_RGMII:
 		break;
+	default:
+		return 0;
 	}
 
 	err = mv88e6xxx_port_write(chip, port, PORT_PCS_CTRL, reg);