We show that, after ensemble averaging, the low temperature phase of a conjugate pair of uncoupled, quantum chaotic, non-Hermitian systems such as the Sachdev-Ye-Kitaev (SYK) model or the Ginibre ensemble of random matrices is dominated by saddle points that couple replicas and conjugate replicas. This results in a nearly flat free energy that terminates in a first-order phase transition. In the case of the SYK model, we show explicitly that the spectrum of the effective replica theory has a gap. These features are strikingly similar to those induced by wormholes in the gravity path integral which suggests a close relation between both configurations. For a nonchaotic SYK, the results are qualitatively different: the spectrum is gapless in the low temperature phase and there is an infinite number of second order phase transitions unrelated to the restoration of replica symmetry.