In the postharvest processing of tea leaves, withering is the first indispensable manufacturing process which produces the mellow, umami and sweet taste of white tea. In this study, we aimed to determine the dynamic changes of the main metabolites and clarify the key differentially expressed genes (DEGs) involved in forming the characteristic taste of white tea during withering. Phytochemical analyses revealed that the contents of total catechins and starch decreased continuously, whereas the contents of theaflavin, γ-aminobutyric acid (GABA), maltose, and soluble sugars increased significantly during withering (from 0–48 h). Meanwhile, the elevation of α-amylase (AMY), β-amylase (BAM), total amylase, and glutamate decarboxylase (GAD) activities may be correlated with the accumulation of GABA and maltose. By transcriptome sequencing, we detected 9707, 15921, 17353, and 17538 DEGs at 12, 24, 36, and 48 h of the withering process, respectively, compared with 0 h sample (fresh leaves). The transcript levels of most of the DEGs involved in catechin biosynthesis were significantly inhibited, whereas those involved in catechin oxidation were significantly up-regulated, which could be correlated to a decrease in catechin content and an increase in theaflavin content. The DEGs involved in GABA biosynthesis were considerably up-regulated, and the down-regulation of SPMS could reduce the competition for converting spermidine to GABA. The up-regulation of the AMY and BAM genes could trigger starch degradation, resulting in the increase of soluble sugar content. These results provide new insights into the importance of the withering process to the characteristic taste of white tea.