We study the competition between the instability of isolated vortices and the nearly geostrophic balance in a rotating fluid of varying depth. The laboratory experiments were performed in a water tank placed on top of a rotating table and the numerical simulations were done with a vortex-in-cell method. The intrinsic vortex instability produces horizontally elongated vortex structures while the nearly geostrophic balance forces the vortices to remain on top of the seamount where they are generated. It was found that the condition for the topographic confinement of the laboratory vortices (which have a Rossby number of order one) is that the semount height be at least one half of the total fluid depth. This condition was confirmed by the numerical simulations.