In order to reveal the mechanism of the abnormal movement (Brownian motion enhances thermal scattering) of nanoparticles on the fluid enhanced heat transfer, the two-phase model was used to study the abnormal convection and diffusion of viscous nanofluids in the flat boundary layer of porous medium. Firstly, for the two-dimensional steady boundary layer stagnation point flow of incompressible Newtonian-nanofluids, the nonlinear governing equations of the flow field and temperature field of nanofluids are established from the Oberbeck-Boussinesq approximate equations. Secondly, the modern Lie group analysis method is introduced, we give the Lie symmetry determining equation of the flow field partial differential equations and the characteristics of the solutions. Further, using the relationship between the Lie symmetries and the conserved quantities, the conservation vector form of the flow field and the group invariant solution are derived in detail, and the reduced order model of the nanofluid flat boundary layer is obtained. Finally, the correctness of the analytical results obtained by the Lie group method was verified for different values of the flow parameter Prandtl. Research has shown that the Lie group method can be used to analytically solve the velocity and temperature distribution functions of abnormal motion of nanoparticles. The fluid temperature increases with the increase of the volume fraction parameter of nanoparticles, but decreases with the increase of the Prandtl value of the base fluid, and decreases with the increase of the plate stretching speed. The Lie group analysis method in this paper provides reference value for numerical simulation solutions of various heat and mass transfer in nanofluids.