Detonation combustors, such as a rotating detonation combustor and a pulse detonation combustor, have advantages in its higher thermal efficiency and possibility to configure smaller scale combustor than that of conventional internal combustion systems. In a previous study, we proposed a new detonation combustor named a reflective shuttling detonation combustor (RSDC) in which detonation waves propagate in opposite directions repeating reflection at side walls. In the present study, a rectangular combustor (53 × 45 × 5 mm) with non-premixed triplet injectors was used to clarify the effect of equivalence ratios and mass flow rates on combustion modes and propulsive performance. As a result, both detonation and deflagration modes were observed. These modes were classified into four types (Single, Double, Single strong single weak, and Deflagration modes) based on CH* images captured by a high-speed camera and a band-pass filter whose peak value is 430 nm. For mass flow rates and equivalence ratios, it is suggested that a normalized fill height h/λ, which varies depending on these parameters, affects wave number transition as a rotating detonation combustor. For propulsive performance, static pressure measured at the bottom of the combustion chamber was normalized with the theoretical value of a conventional isobaric combustor. The normalized pressure for detonation modes were lower than that of deflagration modes. This might be attributed to higher dynamic pressure caused by the wave propagation and/or insufficient combustion.