Abstract: Ruddlesden-Popper (R-P) unconventional ferroelectric Ca₃Mn₂O₇ is a new type of single-phase multiferroic material with important application prospects in new storage devices. This paper used the solid-state reaction method to prepare single-phase R-P type double-layer perovskite structure RE_0.6Ca_2.4Fe_0.6Mn_1.4O₇ (RE=Sm, Tb) ceramics. The materials’ magnetic properties, ferroelectric properties, and magnetoelectric coupling effects were studied. It is found that after Sm, Tb, and Fe replace the A/B sites of Ca₃Mn₂O₇, the samples exhibit ferromagnetic short-range order at 100 K; antiferromagnetic transition occurs near 20 K; spin glass phenomenon appears below 20 K. The samples show the special polarization properties of two pyroelectric peaks in the pyroelectric current test and show a strong magnetoelectric coupling effect. In the dielectric characterization, the samples exhibit a dielectric relaxation behavior characterized by the polaron effect of electron hopping between different valence ions. In summary, the co-doping of magnetic rare earth ion elements (Sm, Tb) and transition metal element Fe can induce a strong magnetoelectric coupling effect on R-P type unconventional ferroelectric Ca₃Mn₂O₇, providing a new direction for the exploration of new single-phase multiferroic materials.