Safety management is paramount because it does not only pertain to workers’ safety but also impacts productivity, profitability, and employees’ morale directly. Every construction project is unique and safety management strategies developed for one project does not necessarily cater to the needs of another project. To achieve a project-based safety management strategy, this research introduces a novel concept of Safety Frontier – the theoretical maximum level of safety that can be achieved in perfect conditions under good management and typical field conditions. The objective of this research is to identify and isolate the factors affecting worker’s safety while performing labor-intensive operations and propose a method to determine the safety frontier empirically. A Kinect camera was used to record, a sample dataset which was manually annotated to analyze sub-operations. This research only focuses on body posture as a long-term safety issue but it reports the challenges while determining the thresholds for safety and effect of other factors on the overall safety situation. The determination of the highest level of safety can act as a yardstick benchmark to evaluate effectiveness of safety management strategies. It can assist safety managers to formulate adaptive safety management strategies and help them understand the training and supervision needs at construction sites.