Under the presence of the parameter uncertainty in the system modeling, the classical LQR or LQG controller cannot guarantee the closed loop stability. This paper investigates the modification of the classical LQR and LQG methods, which are based on Lyapunov``s second method. For the full state feedback control, a modified algebraic Riccati equation (MARE) is proposed to obtain a robust feedback control law. The concept of the design margin for parameter uncertainty in time domain, which bridges the classical quadratic stabilization technique and the $H^{\infty}$ control theory, is introduced to quantify the safety margin of the designed control system. It is shown that the controller obtained by the MARE includes that of the $H^{\infty}$ control theory as a subset. The MARE is investigated in the classical LQR, the classical quadratic stabilization and $H^{\infty}$ control perspectives. In addition, the optimization of the uncertainty sttructure is illustrated in the sense of the control performance. It gives a set of equations which are coupled but numerically solvable. For the robust dynamic output feedback control, two MAREs and a Lyapunov equation which is coupled with the one of the MAREs are developed. To deal with the stability of the closed loop system including a Kalman observer, a new Lyapunov function which is closely related to the control cost function is proposed. The result is interpreted in view of LQG and $H^{\infty}$ control theory as well.