Suspiciousness detection is crucial for anticipating potential security threats and facilitating timely intervention and risk mitigation. Enhanced by deep learning, vision-based systems can play a significant role in this area. This paper presents a computationally efficient vision-based suspiciousness estimation system cascading detection, classification, and analysis modules. It incorporates factors like suspicious objects, facial expressions, and abnormal body language. A suspicious object detector (SOD) is designed to precisely locate the objects invariant to the size, scale, rotation, translation, and occlusion. A deep convolutional neural network is implemented for body language and facial expression recognition with the image and landmark features as its input. To estimate the final suspiciousness, an algorithm (USE-riskometer) is proposed in this work by considering all the detected and analyzed factors. All the modules are separately trained with the corresponding object detection and facial expression datasets. In addition to this, a novel dataset for suspiciousness estimation is proposed in this work. This dataset includes various suspicious elements, such as weapons, fire, crowd presence, facial expressions, and body language in an uncontrolled environment. Each module, along with the dataset, is evaluated against state-of-the-art methods, demonstrating robustness. This work represents a significant advancement in preemptive security measures, leveraging advanced technologies for improved recognition of suspicious activities.