The existing TCP/AQM model does not consider the impact of unresponsive flows to AQM algorithms, which contributes to about 70%～80% of the Internet flows. It is important to analysze the performance of AQM algorithms taking into account of the unresponsive flows. An extended GI/M/1/N queueing system is established by means of embedding the AQM mechanism into the standard GI/M/1/N queueing system. Based on the extended GI/M/1/N queuing system and self-similar traffic of the Internet, three classical AQM algorithms (TD, RED and GRED) are evaluated for the unresponsive flows. The analytic results are consistent with those obtained from NS2 simulations, which means the extended queueing system can be used to assess the performance of AQM algorithms with the unresponsive flows.