Abstract: An end-to-end Structure-Strategy Defect Detection Network (SSDDNet) is proposed in this research to solve the problems in surface defect detection of metallic workpieces, such as complex defect morphologies, diverse scales, ambiguous boundaries, unstable label quality, and the high cost of pixel-level annotation. A multi-scale contextual aggregation module is structurally established to fuse dynamic convolution and multi-dilation information, while a boundary enhancement module is introduced to strengthen the modeling of ambiguous boundaries. And a spatial label uncertainty modeling approach is strategically introduced to enable stable training under weakly annotated conditions. Experimental results show that: (1) SSDDNet achieves a 1.1% improvement in mean accuracy over the state-of-the-art MixSup model on the public KolektorSDD2 dataset without pixel-level annotations. (2) SSDDNet outperforms MixSup by 17.6 percentage points in mean accuracy and achieves approximately 15 percentage points improvement in SSH on the self-constructed industrial dataset BatteryBase, indicating the strong generalization capability of the proposed model. This research provides a novel approach for surface defect detection of metal workpieces.