In this paper, we present an algo-rithm for capacity optimization in intelligent reflecting surface (IRS)-based multiple-input multiple-output (MIMO) communication sys-tems. To maximize the capacity of elements in IRS, we use augmented Lagrange method with the equivalent transformations on the covari-ance matrix and reflection matrix constraints.This results an adjustable phase shift on the incident signal. Furthermore, we reshape the complex-valued covariance matrix and re-flection matrix to a vector for the ease of cal-culating partial derivatives to find the search direction. Then, the quasi-Newton updates and modified Broyden-Fletcher-Goldfarb-Shano(BFGS) method in the complex domain form are used to find the local minimum. Finally,numerical simulation results demonstrate that our proposed IRS-aided system using the algo-rithm performs better than the state-of-the-art and the conventional communication systems.