To improve the tamper detection performance and harmonize the conflict between security and invisibility, this paper proposes a fragile watermarking algorithm for JPEG images, in which the authenticity of image blocks is determined by neighborhood comparison. This scheme divides the original image into 8×8 image blocks. For each block, four bits watermarks are generated based on the DCT coefficients to be protected. Next, the watermarks are randomly embedded in the least significant bit (LSB) of DCT coefficients with smaller quantization step in other four blocks. The authenticity of each block is determined by comparison between the number of inconsistent image blocks in the eight-neighborhood of each block and its four corresponding mapping blocks. Then, this paper derives the probability of false acceptance and false rejection under general tampering and collage attack and validates the theoretical analysis results by statistical experiments. Theoretical analysis and statistical experiments show that comparing the number of inconsistent image blocks in the eight-neighborhood of each block with its four corresponding mapping blocks improves the tamper detection performance. Embedding watermarks in the LSB of DCT coefficients with less quantization step efficiently solves the conflict between the number of DCT coefficients to be protected and invisibility.