Abstract: To improve the real-time performance of High Efficiency Video Coding (HEVC) intra-frame encoding, a method, which utilizes a lightweight convolutional network with even-length and step-size convolutional kernels and a self-attention mechanism, is proposed to predict the intra-frame partitioning structure of Coding Tree Units(CTU) , thereby reducing the encoding time required for the encoder to perform quadtree recursive traversal partitioning on CTUs. In the original encoding strategy, Rough Mode Decision accelerates the process by estimating the rate-distortion loss value in Rate Distortion Optimization based on the Sum of Absolute Transformed Difference (SATD) -based loss value, but it still consumes a certain amount of encoding time. A proposed method reduces the number of patterns calculated in the Rough Mode Decision process through a sampling search approach, reducing the number of patterns from 35 to 18, and decreasing the time required to estimate the loss value during the Rough Mode Decision process. The more favorable multiple candidate intra-frame modes obtained from the Rough Mode Decision process are used for Rate Distortion Optimization. In order to reduce the number of candidate modes that need to be calculated in Rate Distortion Optimization, an early stopping decision is implemented by filtering out some less likely candidate modes based on the differences in the estimated loss values of the intra-frame prediction angle modes in the candidate mode list, thus reducing the number of candidate modes that need to be evaluated in Rate Distortion Optimization and consequently decreasing the computation time of the Rate Distortion Optimization process. The proposed algorithm achieves an average encoding time reduction of 78.15% on the test sequences, with a BD-PSNR of -0.168dB and a BD-RATE of 3.49%.

Key words: video coding, neural network, intra-frame prediction, fast algorithm