Privacy negotiation performs a pre-protective role against privacy disclosure as it can assist social network users to build a consensus on privacy protection before information sharing. Accountability is an attribute that a subject is responsible for an action or consequence, and it is an important aspect of transparent and explainable artificial intelligence applications. Accountability in the privacy negotiation process in social networks is of great significance for improving the transparency and explainability of application platforms or systems. Although Kekulluoglu et al. proposed an agent-based reciprocal privacy negotiation system, the accountability for the behaviors of agents was not discussed. For this reason, a novel system for agent behavior accountability during privacy negotiation in social networks is designed and implemented, and qualitative and quantitative accountability methods are developed. Moreover, requirements and behavior indicators are also proposed to achieve accountability. Specifically, the qualitative accountability method can accurately determine whether a privacy negotiation agent has misbehavior and pinpoint the specific location of the misbehavior. The quantitative accountability methods include simple quantification, weighted Mahalanobis distance, and improved Minhash and can quantify the severity of the agent’s misbehavior. The experimental data demonstrate the validity and rationality of the proposed system and methods.