Human itineraries are often initiated by some general intentions and will be optimized after considering all kinds of constraints and available information. This paper proposes a category-based itinerary recommendation framework to help the user transfer from intentions to itinerary planning, which join physical trajectories and information of location based social networks. The main contributions are: (1) Build the category based activity scheduling model; (2) Design and implement the category tree based POI (point or interest) query strategy and algorithm; (3) Propose the Voronoi graph based GPS trajectory analysis method to build traffic information networks; (4) Combine social networks with traffic information networks to implement category based recommendation by ant colony algorithm. The study conducts experiments on datasets from FourSquare and GeoLife project. A test on satisfaction of recommended items is also performed. Results show that the satisfaction reaches 80% in average.