Controlling multiple multi-joint fish-like robots has long captivated the attention of engineers and biologists, for which a fundamental but challenging topic is to robustly track the postures of the individuals in real time. This requires detecting multiple robots, estimating multi-joint postures, and tracking identities, as well as processing fast in real time. To the best of our knowledge, this challenge has not been tackled in the previous studies. In this paper, to precisely track the planar postures of multiple swimming multi-joint fish-like robots in real time, we propose a novel deep neural network-based method, named TAB-IOL. Its TAB part fuses the top-down and bottom-up approaches for vision-based pose estimation, while the IOL part with long short-term memory considers the motion constraints among joints for precise pose tracking. The satisfying performance of our TAB-IOL is verified by testing on a group of freely swimming fish-like robots in various scenarios with strong disturbances and by a deed comparison of accuracy, speed, and robustness with most state-of-the-art algorithms. Further, based on the precise pose estimation and tracking realized by our TAB-IOL, several formation control experiments are conducted for the group of fish-like robots. The results clearly demonstrate that our TAB-IOL lays a solid foundation for the coordination control of multiple fish-like robots in a real working environment. We believe our proposed method will facilitate the growth and development of related fields.