Recent researches show that virus-to-cell infection and cell-to-cell transmission are two HIV infection modes. In this paper, we propose a delayed HIV infection model including both virus-to-cell infection and cell-to-cell transmission and CTL immune response. The time delay describes the phenomenon between viral entry and viral production. We show the nonnegativity and boundedness of solution, obtain the equilibrium points and prove local asymptotic stability of the equilibrium points. Then the optimal control problem with antiretroviral therapy and pharmacological delay is posed. We establish and analyze two types of objective functions, one is linear control and the other is quadratic control. Numerical simulations have been performed to verify the stability of equilibrium points and show the optimal control strategies and the effects of control on cells concentration by Matlab and Lingo.