This article reviews the past works on topological Hall effects of magnons in insulating magnets with a particular focus on ferrimagnets. First, we discuss the magnonic quantum Hall effect in ferromagnets induced by the Aharonov–Casher effect. Specifically, starting from the classical Hall effect of magnons proposed by Meier and Loss [Phys. Rev. Lett. 90, 167204 (2003)], we establish the Landau quantization of magnons with suitable electric-field configuration through the Aharonov–Casher effect in ferromagnets. Magnons are shown to form the Landau levels, giving rise to chiral edge magnon states. Second, by making use of the Néel magnetic order of insulating antiferromagnets, we show that a bosonic counterpart of electronic topological insulators can be established by the electric field gradient via the Aharonov–Casher effect, extending the notion of symmetry protected topological phases to antiferromagnetic insulators. Using magnons with opposite magnetic dipole moments associated with the Néel order, we establish a magnonic analog of the quantum spin Hall effect characterized by helical edge magnon states. We generalize this known result for antiferromagnets to ferrimagnets, leading to the discussion of the topological Hall effect of magnons in the bulk of ferrimagnets. Third, we show that the topological Hall effect is expected to exist in ferrimagnets with skyrmion crystal, by extending the magnetic texture-induced thermal Hall effects of magnons in skyrmion crystal phases of FMs proposed by Hoogdalem et al. [Phys. Rev. B 87, 024402 (2013)] into ferrimagnets. One particular feature of the topological Hall effect of ferrimagnets is that it is thermally tunable in that the direction of heat flux changes across the angular momentum compensation point at which the angular momentum changes its sign with respect to the magnetization direction. Last, by enumerating the recent development of experimental techniques, we give our perspective and provide a platform to discuss a direction for further development of topological magnonics.