Drop the unused endpoints.  They should only be used when there is an
actual remote-endpoint connected.

Cc: Sekhar Nori <nsekhar@ti.com>
Signed-off-by: Kevin Hilman <khilman@baylibre.com>
Signed-off-by: Sekhar Nori <nsekhar@ti.com>