We introduce a sketch-based 3D modeling system using Virtual Reality (VR) with a Head Mounted Display (HMD). The system enables users to sketch freely 3D objects in various colors, arrange and situate the objects in the environment and finally export the virtual objects into a 3D file for fabrication. We evaluated the system in a small workshop with users, and the system was found to have a possibility for rapid ideation and for spatially conceptualizing ideas.