Virtual Reality (VR) is widely used in conception, planning and design phases of a project, mainly for communication and collaboration. During the construction phase, safety demands major attention at any active construction site. Iterating among alternative safety plans in real world conditions can be dangerous, expensive, resource intensive and often not feasible. Safety is not only governed by what regulations are at place; the perception of the workers towards site safety conditions is also equally important. In other words, in addition to imposing safety rules, it is also equally important that the workers feel safe in their working environment. This paper explores the potential of leveraging VR to investigate on perceptual response of workers on site safety conditions. A VR environment is created based on real-time geometric (laser scan) and location (GPS) data collected from moving equipment and workers at an active construction site. Hazards are introduced into the VR scenes in a systematic and controlled manner. Subjects are exposed to the scenes through a head-mounted VR system. The experience of the subjects in different hazardous scenarios are recorded through Affective Sensing devices and a questionnaire. Affective sensing technology can track human physiological responses in real-time. The goal of this research is to leverage VR environment to (i) test the feasibility of using affective sensing devices for tracking human responses to hazardous situations, (ii) study the potential of exposing construction workers to virtual hazards for training purposes. The results from this study will give a better understanding of potential of VR in construction safety training and management.