Time delayed feedback control is well-known as a practical method for stabilizing unstable periodic orbits embedded in chaotic attractors. The systems under the control method are described by delay differential equations and then become infinite dimensional system whose phase space is function space. However, no intrinsic discussion has been obtained for the control performance associated with global dynamics in function space. In this paper, we numerically discuss the influence of a remaining chaotic invariant set on control of chaos in Two-well Duffing system. The discussion is based on the existence of the global unstable manifold of the directly unstable periodic orbit which can not be stabilized by the control method. We reveal that it causes highly complicated domain of attraction for target orbits and long chaotic transient before the convergence.