Crowd intelligence is a crucial component of the next generation of artificial intelligence. Researching how to stimulate and converge the innovative power of “people” in open-source communities can significantly enhance development efficiency. Community detection, as a technical approach to studying the relationships among developers in open-source projects, plays a vital role in exploring and understanding social networks. However, current research has predominantly focused on large-scale social networks such as Facebook, while systematic studies on community detection in project-level open source software developer social networks (OSS-DSN) remain limited. This study first collects real-world data and analyzes the features of OSS-DSN. Then, it benchmarks several overlapping and non-overlapping community detection algorithms on these real datasets, comparing algorithm performance across multiple metrics and dimensions. Finally, based on synthetic OSS-DSN, it generates networks efficiently and performs algorithm evaluations using ground-truth data for comparative analysis. Differences in characteristics between small- and medium-scale social networks and large-scale networks are identified, and the influence of these differences on community detection metrics and algorithm performance is explored. The study provides a new benchmark and offers important insights into communication and collaboration in open-source software communities.