With the help of distributed Hash table, the structured P2P (peer-to-peer) network has a short routing path and good extensibility. However, the mismatch between the overlay and physical network becomes the obstacle in the way of building an effective peer-to-peer system in a large-scale environment. In this paper, a generic, protocol-independent approach is proposed to solve this problem. This method is based on the swaps of peers. By discovering and performing the potential swaps that are beneficial to the match between overlay and physical network, it can reduce the average latency and improve the performance of the system. The experimental results show that the approach can greatly reduce the average latency of overlay networks. Moreover, the cost of overhead is controllable. Besides, if combining this approach with other protocol-dependent ones, the performance can be further improved.