Smooth and fluent speech communication is one of the main aims of current high performance video conferencing system. For this purpose, a multipoint real-time speech mixing and scheduling algorithm based on packet network is put forward and then realized. The scheduling algorithm, which contains several strategies, such as queue model analysis, synchronous control of multipoint speech streams, data pre fetching, timely refreshing buffers and so on, effectively guarantees continuity of the mixed speech in a limited complexity, and its mixing method has proved to proved to ensure naturalness and intelligibility of the mixed speech quality. The realization of this algorithm has led to a fact that the video conferencing system has better hearing perceptibility than those available oversea H.232 products, such as Netmeeting, based on the same H.323 specification.