The paper proposes an alternative approach to improve the performance of image retrieval. In this work, a framework for image retrieval based on machine learning and semantic retrieval is proposed. In the preprocessing phase, the image is segmented objects by using Graph-cut, and the feature vectors of objects presented in the image and their visual relationships are extracted using R-CNN. The feature vectors, visual relationships, and their symbolic labels are stored in KD-Tree data structures which can be used to predict the label of objects and visual relationships later. To facilitate semantic query, the images use the RDF data model and create an ontology for the symbolic labels annotated. For each query image, after extracting their feature vectors, the KD-Tree is used to classify the objects and predict their relationship. After that, a SPARQL query is built to extract a set of similar images. The SPARQL query consists of triple statements describing the objects and their relationship which were previously predicted. The evaluation of the framework with the MS-COCO dataset and Flickr showed that the precision achieved scores of 0.9218 and 0.9370 respectively.