net: fec: Avoid MX28 bus sync issue

The MX28 multi-layer AHB bus can be too slow and trigger the
FEC DMA too early, before all the data hit the DRAM. This patch
ensures the data are written in the RAM before the DMA starts.
Please see the comment in the patch for full details.

This patch was produced with an amazing help from Albert Aribaud,
who pointed out it can possibly be such a bus synchronisation
issue.

Signed-off-by: Marek Vasut <marex@denx.de>
Cc: Albert ARIBAUD <albert.u.boot@aribaud.net>
Cc: Fabio Estevam <fabio.estevam@freescale.com>
Cc: Stefano Babic <sbabic@denx.de>
Tested-by: Fabio Estevam <fabio.estevam@freescale.com>
Tested-by: Alexandre Pereira da Silva <aletes.xgr@gmail.com>

drivers/net/fec_mxc.c

@@ -736,6 +736,28 @@ static int fec_send(struct eth_device *dev, void *packet, int length)
 	addr = (uint32_t)fec->tbd_base;
 	flush_dcache_range(addr, addr + size);
 
+	/*
+	 * Below we read the DMA descriptor's last four bytes back from the
+	 * DRAM. This is important in order to make sure that all WRITE
+	 * operations on the bus that were triggered by previous cache FLUSH
+	 * have completed.
+	 *
+	 * Otherwise, on MX28, it is possible to observe a corruption of the
+	 * DMA descriptors. Please refer to schematic "Figure 1-2" in MX28RM
+	 * for the bus structure of MX28. The scenario is as follows:
+	 *
+	 * 1) ARM core triggers a series of WRITEs on the AHB_ARB2 bus going
+	 *    to DRAM due to flush_dcache_range()
+	 * 2) ARM core writes the FEC registers via AHB_ARB2
+	 * 3) FEC DMA starts reading/writing from/to DRAM via AHB_ARB3
+	 *
+	 * Note that 2) does sometimes finish before 1) due to reordering of
+	 * WRITE accesses on the AHB bus, therefore triggering 3) before the
+	 * DMA descriptor is fully written into DRAM. This results in occasional
+	 * corruption of the DMA descriptor.
+	 */
+	readl(addr + size - 4);
+
 	/*
 	 * Enable SmartDMA transmit task
 	 */