Try power cycling the modem when you connect directly, that's a required step. You need to isolate the issue to either the ISP/modem or the router.