With click-through data analysis, an automatic search engine performance evaluation method is proposed. This method generates navigational type query topics and answers automatically based on search users' querying and clicking behavior. Experimental results based on a commercial Chinese search engine's user logs show that the automatic method gets a similar evaluation result with the traditional assessor-based ones. This method can also provide timely evaluation results with little human efforts.