With the development of big data and data mining technology, the access pattern becomes a risk of leaking user's privacy in the cloud computing environment. Oblivious random access memory (ORAM) is an effective way to protect the user's access pattern. The existing ORAMs mostly support a single user. The only ORAM supporting multi-user is based on the hierarchical ORAM including a reshuffling phase that may cause high computational complexity. In order to avoid reshuffling, this paper designs a new multi-user ORAM based on binary tree. First, a proxy encryption scheme is improved. Second, a proxy between users and the cloud server is introduced. The data encrypted by different users is encrypted again by the proxy to obtain the final ciphertext encrypted by the same key, and the final ciphertext is stored on the server. The security of the scheme is based on the indistinguishability of the pseudorandom function, and the worst computational complexity and the amortized computational complexity are all O(log²n), achieving higher efficiency than the existing multi-user ORAM schemes.