The demand for multi-speaker speech transcription and speaker attribution in applications such as meeting minutes and customer service quality inspection is increasing. Recent advances in multimodal large language models have given rise to audio-language models (ALMs) that can simultaneously interpret audio signals and natural-language prompts within a unified autoregressive decoding framework, making them a natural fit for the speaker diarization task and offering a fresh approach to end-to-end multi-speaker audio transcription. This study proposes an end-to-end speaker diarization system based on an ALM and achieves synergistic optimization of speech-recognition capability and speaker-attribution capability via a two-stage training strategy, thus generalizing the capability of ALMs to specific downstream tasks. In the first stage, supervised fine-tuning (SFT) introduces a “speaker loss” into the standard cross-entropy objective to weight and strengthen the learning signal for sparse speaker-label tokens. In the second stage, a reinforcement-learning scheme based on group relative policy optimization (GRPO) is employed, designing a reward function that jointly considers cpCER and SA-CER to break through the performance bottleneck of supervised learning. Experiments in a two-speaker setting compare with the open-source 3D-Speaker toolkit and the Diar Sortformer model, as well as the proprietary speaker diarization APIs from AssemblyAI and Microsoft Azure. Ablation studies are further conducted to validate the training methodology, and experiments are subsequently extended to a four-speaker scenario. Results demonstrate that the two-stage approach significantly improves both ASR and speaker-attribution performance in the two-speaker environment, whereas in the four-speaker setting, conventional SFT already yields substantial improvements. Challenges such as resource consumption, input-length limitations, and cross-domain adaptation are also discussed, and future enhancements are proposed, including streaming audio encoders, curriculum learning, and rejection-sampling strategies. This study shows that ALMs hold great promise for multi-speaker diarization tasks but require additional technical advances to handle more complex acoustic scenarios.