Maximizing safety often means operating a construction site at its highest sustainable level of safety. This sustainable level of safety is of major interest for safety managers because knowledge of such level can help them identify areas and opportunities of enhancing safety on the jobsites. Safety strategies and plans are made by the managers based on their perception of such sustainable safety. No formalized method exists to determine such level of safety for a construction site. OSHA regulations provide a general guideline but do not consider specific site conditions. The regulations also do not provide insight on what can be done beyond the mandatory requirements to maximize the level of safety and what level of safety can be attained and sustained on a site. To address this problem, this paper proposes a novel framework to identify the sustainable level of safety for a given condition at site. The method builds upon a two-way approach in which a theoretical maximum level of safety and observed level of safety govern the sustainable safety at site. The method also intends to explore the inefficiencies at the jobsite and help identify the areas of improvement. The scope of the paper is limited to labor-intensive lifting operation and relies on skeletal data collected by Kinect camera for illustration purposes. The paper outlines the method and the components of the framework and provides an illustration through a lab-based experiment. The method can potentially help the safety managers to improve their strategies based on real data collected from the actual site and set realistic goals for safety management on construction sites. The method can also be implemented to automatically analyze safety and make recommendations based on real-time data collected from the site.