In this article computational fluid dynamics analysis of a three-dimensional compressible and turbulent flow has been carried out through a vortex tube. The standard k-ε turbulence model is utilized in order to simulate an axisymmetric computational domain. The numerical simulation has focused on the energy separation and flow field patterns of a somewhat nonconventional vortex tube, which is on the basis of creating an external hole at the end of each nozzle. According to the selected nozzles geometry, some of unfavorable phenomena such as shock wave, high pressure regions and appearing of unsymmetrical rotating flow patterns in the vortex chamber would be recovered significantly. In this way the physical parameters of flow field are derived under different both inlet mass flow rates and outlet pressures of nozzles hole (OPH). The results show that increasing OPH value enhanced the cooling capacity of machine in the most of operating conditions.