In the past decade or so, artificial intelligence-related services and applications have boomed, and they require high computing power, high bandwidth, and low latency. Edge computing is currently regarded as one of the most appropriate solutions for such applications, especially for video analysis-related ones. This study investigates multi-server multi-user heterogeneous video analysis task offloading, where users select appropriate edge servers and then upload their raw video data to the servers for video analysis. It models the issue of multi-server multi-user heterogeneous video analysis task offloading as a multiplayer game issue. The aim is to effectively deal with the competition for and sharing of the limited network resources among the numerous users and achieve a stable network resource allocation situation where each user has no incentive to change their task offloading decision unilaterally. With the optimization goal of minimizing the overall delay, this study successively investigates the non-distributed and distributed video analysis scenarios and proposes the game theory-based algorithms of potential optimal server selection and video unit allocation accordingly. Rigorous mathematical proof reveals that Nash equilibrium can be reached by the proposed algorithms in both of the two cases, and a low overall delay is guaranteed. Finally, extensive experiments on actual datasets show that the proposed methods reduce the overall delay by 26.3% on average, compared with that of other currently available algorithms.