Thermodynamic Stability and Effects of Bose–Bose Repulsion in an Ultracold Bose–Fermi Mixture with Strong Hetero-Pairing Fluctuations

We theoretically investigate the thermodynamic stability of an ultracold Bose–Fermi mixture with a tunable inter-species interaction associated with a hetero-nuclear Feshbach resonance. To include hetero-pairing fluctuations, we employ an improved T-matrix approximation (iTMA), where all Bose Green’s functions involved in the self-energy are modified so as to satisfy the required gapless Bose excitations at the BEC phase transition. Using iTMA, we evaluate the compressibility matrix in the normal state, to examine the stability of the system against an effective Bose–Bose attraction, mediated by Fermi atoms. We clarify how the mechanical collapse of a Bose–Fermi mixture can be avoided by a direct Bose–Bose repulsive interaction. We also show that this thermodynamic instability cannot be captured by the ordinary non-self-consistent TMA, where the bare Bose Green’s function is used in the self-energy.Since the thermodynamic stability is crucial for the experimental realization of a resonant Bose–Fermi mixture, our results would contribute to the research for many-body properties of this system.