Generally, the least mean square (LMS) adaptive algorithm is widely used in antenna array beamforming given its target tracking capability and its low computational requirements. However, the classical LMS implementation still suffers from a trade-off between convergence speed and residual error floor. Numerous variants to the classical LMS have been suggested as a solution for the previous problem at the cost of a considerable increase in the computational complexity and degraded performance in low signal to noise ratio (SNR). Thus, in this paper, we propose a multi-stage parallel LMS structure with an error feedback for accelerating the LMS convergence while maintaining a minimal steady state error and a computational complexity of order O(N), where N represents the number of antenna elements. In parallel LMS (pLMS), the second LMS stage (LMS2) error is delayed by one sample and fed-back to combine with that of the first LMS stage (LMS1) to form the total pLMS error. A transfer function approximation to the pLMS is derived in order to numerically assess the pLMS stability and to determine the approximate maximum parametric value of the step size for which the pLMS remains stable. Simulation result highlight the superior performance of the pLMS in demonstrating accelerated convergence and low steady state error compared to previous variants and for different SNR environment.