The time dependent behavior of the neutron flux distribution which can be expressed by the diffusion equation with the xenon and iodine balance equations is studied. The partial differential reactor dynamic equation (Distributed Parameter System) is approximated into the finite dimensional ordinary differential equation by the modal expansion. The clean reactor modes and Kaplan modes are modified using the perturbed eigenvalue during transients based on the perturbation theory. The minimal number and optimal sensor locations in a nuclear system with the fixed incore detectors are determined. A scalar measure of the covariance matrix error in the optimal filter is minimized with respect to the sensor locations. the necessary conditions for optimal sensor locations are derived by using the matrix minimum principle, thus making it computaionally more attractive. The location of sensors are initially guessed through sensitivity analysis to reach the solutions of the optimal location quickly. The method to determine the minimum number of sensors is suggested based on the observability and admissible error bound. Several numerical simulations are performed to determine the minimal number and optimal sensor location for a one-dimensional slab reactor and a two-dimensional Combustion Engineering type reactor with fixed incore detectors. Through the simulations the possibility of practical implementation and the rapid convergence of the algorithm are verified. Also, to improve the reliability and safety of instrumented system, the detection and isolation of instrument failures in nuclear reactor system are studied. To isolate component failures via robust observation, a bank of detection observers based on the Kalman filter is constructed. each of which is-sensitive to only one specified component failure and minimizes the covariance matrix error. However, since the obsrved deviations may be attributed to the changes in system behavior, failure decision is confirmed by ...