BOMM is a byte-oriented mixed type algorithm with memory, which is used to disorder a given byte sequence. It has been used as a main component in a new stream cipher called Loiss for having many good cryptographic properties. This paper builds an algebraic equation system with degree 5 for BOMM, and based on this equation system, discusses the complexity of algebraic attack on Loiss. In addition, the paper also discusses the statistic weakness of BOMM and gives an analysis of the security of Loiss under a specific class of weak keys.