Abstract: Using sparse sensors to capture human movements and driving virtual humans in computers to reproduce these movements is one of the key technologies in fields such as virtual reality, among which, the relevant calculations for reproducing movements must simultaneously satisfy motion constraints and physical constraints. Currently, nonlinear optimization methods are primarily employed to address the physical constraints in such computations. However, these methods suffer from several drawbacks, including long computation time, high computational complexity, and the necessity of designing dedicated optimizers. To address these issues, a method based on a deep neural network model for solving the physical constraints optimization in sparse sensor motion capture is proposed. Firstly, the model effectively combines the multi-modal feature fusion network and the multi-path refinement network to form a post-fusion multi-level structure, which serves as the basic model of this research. Secondly, through a progressive fusion method, back-projection connections between the layers of the basic model are established, enabling the iteration of the basic model. This allows the information fused at deeper layers to be utilized by the shallower layers. Thirdly, a loss function in the form of combined weighting that incorporates physical constraints is proposed, which is suitable for fine-tuning the deep model in the presence of both implicit and explicit physical constraints. The experimental results demonstrate that the proposed method not only exhibits good feasibility but also improves computational efficiency by approximately 20 percentage points. Compared with other commonly used optimization methods, the proposed method performs better in terms of four mainstream evaluation indicators. Additionally, the method yields favorable results when applied to other datasets, demonstrating its strong generalization capability. This method provides a new perspective for the formation of an end-to-end deep model for sparse sensor motion capture.