Abstract: Portrait semantic segmentation is one of the important research contents in the field of computer vision, but the existing portrait semantic segmentation methods are liable to ignore the small size portraits in multi-person portrait images. At the same time, the segmentation results are prone to the phenomenon of mutual adhesion between multiple portraits. Moreover, the phenomenon of mutual occlusion between portraits in the image easily leads to poor segmentation accuracy of portrait edges. Based on the above problems, a semantic segmentation method for multiple portraits with fused label semantics is propose, where multiple labels are assigned to multiple portraits in an image, and semantic labels are embedded as inputs to the encoder at the same time, and the semantic labels and the image feature representations are correlated using the cross-modal cross-attention module, and the semantically fused feature representations are obtained as outputs of the encoder at each layer of the model. The HRF attention module is proposed to generate multiple hypotheses for image based on target detection algorithm for feature extraction separately. The network is trained and tested on Supervisely augmented dataset. The experimental results show that the algorithmic model achieves 95.94%, 94.60%, and 96.02% accuracy on the three evaluation metrics of PA, MIoU, and Dice, respectively, and has higher segmentation accuracy than the semantic segmentation models U-net, PSPNet, Deeplab v3+, PortraitNet, and Swin Unet.