Taking the layout problem of satellite cabins as background, the authors make a study of the optimal layout problem of circle group in a circular container with performance constraints of equilibrium etc. in this paper, which belongs to NP-complete problem. A modified genetic algorithm called decimal coded adaptive genetic algorithm (DAGA) is presented to solve the problem. In this way, the algorithm relaxes the combinatorial explode and the premature convergence of genetic algorithm. Two examples are provided (one of them is proposed by the authors, and its optimal solution is known). The numerical results show that the DAGA is effective and superior to multiplicator algorithm. The DAGA can be developed to solve other layout optimization problems.