A novel serially concatenated GMSK system is proposed for satellite communications subject to low SNR, limited power and spectrum resources. First, we design a Nonrecursive continuous-phase encoder (NRCPE) based GMSK based on the Rimoldi's decomposition. Then, a corresponding pilot-aided quasi-coherent demodulation algorithm is developed, whose basic principle is that a modified BCJR-based detection performs on the received signals with initial and ending trellisstates being determined using the very limited pilot overhead. Finally, we choose proper modulation parameters for the NRCPE based Gaussian minimum shift keying GMSK signaling according to the trade-offs between the power and spectral efficiency. The simulation results show that the LDPC coded GMSK system using the proposed algorithm can achieve excellent performance and can also work well in the presence of the large Doppler shifts and some burst errors.