High power consumption has become one of the top considerations in high performance computing field. Recently, there are many studies focused on optimizing the system performance within a given power budget. However, most existing solutions are explored for homogeneous system without considering the differences in power consumption and processing speed between heterogeneous processors, and therefore could not be adapted for accelerator-based heterogeneous parallel system effectively. This paper first summarizes the execution model of modern accelerator-based parallel system, and then introduces the power control framework consisting of two power management hierarchies, i.e. system-level power controller and heterogeneous processing engine power controller from top to bottom. In the lower level controller, targeted for OpenMP-like parallel loop, the paper first theoretically analyzes the conditions for the maximum performance given a power budget for heterogeneous processors. Based on this result, the paper provides a power-constrained parallel loop scheduling algorithm which coordinates parallel loop partition and voltage/frequency scaling for heterogeneous processers to achieve the optimal performance given a system power budget. In the upper level controller, the paper establishes the evaluation metrics for heterogeneous processing engine to allocate power budget, in order to keep fairness between concurrent applications and improve the whole system efficiency. Finally, the paper evaluates the proposed method in a typical CPU-GPU system.