Human skin plays an important role in hand manipulation by making a stable grasp with an enlarging contact area while providing a firm hold on the object. However, satisfying these two functions is contradictory in conventional single-layer artificial skin. Softer skin material would increase the contact area, which is advantageous in maintaining the stability, but it decreases the manipulability since the object tends to make uncontrollable movement within the softer skin, and vice versa for harder skin material. This paper presents a biomimetic three-layer skin structure inspired by human palm skin and shows that both stability and manipulability can be enhanced with the three-layer structure. To achieve the unique stiffness characteristics of the human palm skin, a porous latex structure, which is highly compressible but tough in tensile direction, is chosen as the subcutaneous fat layer. Through the novel experimental setup and the finite element method simulations, it is found that the porous latex structure is the key structure contributing to both stability and manipulability. Furthermore, it is demonstrated that a robotic hand with the proposed skin material shows enhanced robustness in grasping tasks. With the proposed skin material, the robotic hands would be more advantageous for challenging manipulation tasks.