Drop the unused endpoints.  They should only be used when there is
an actual remote-endpoint connected.
Signed-off-by: Kevin Hilman <khilman@baylibre.com>
Signed-off-by: Sekhar Nori <nsekhar@ti.com>