In this work, we show how fiber-based terahertz systems can be robustly configured for accurate terahertz ellipsometry. To this end, we explain how our algorithms can be successfully applied to achieve accurate spectroscopic ellipsometry with a high tolerance on the imperfect polarizer extinction ratio and pulse shift errors. Highly accurate characterization of transparent, absorptive, and conductive samples comprehensively demonstrates the versatility of our algorithms. The improved accuracy we achieve is a fundamental breakthrough for reflection-based measurements and overcomes the hurdle of phase uncertainty.