A new hierarchical approach called bintree energy segmentation was presented for color image seg-mentation. The image features are extracted by adaptive clustering on multi-channel data at each level and used as the criteria to dynamically select the best chromatic channel, where the segmentation is carried out. In this approach, an extended direct energy computation method based on the Chan-Vese model was proposed to segment the selected channel, and the segmentation outputs are then fused with other channels into new images,from which a new channel with better features is selected for the second round segmentation. This procedure is repeated until the preset condition is met. Finally, a binary segmentation tree is formed, in which each leaf represents a class of objects with a distinctive color. To facilitate the data organization, image background is employed in segmentation and channels fusion. The bintree energy segmentation exploits color information involved in all channels data and tries to optimize the global segmentation result by choosing the "best" chan-nel for segmentation at each level. The experiments show that the method is effective in speed, accuracy and flexibility.