The problem of similarity search in high-dimensional space is a fundamental problem with numerous applications in computer science, but remains challenging due to the curse of dimensionality. To address this challenge, in this paper, DForest, a novel indexing approach, is proposed for both range and kNN queries on high-dimensional data. Unlike the previous similarity search approaches, which perform a dimensionality reduction on all objects by the same fixed value, we discover the minimal dimensionality each object requires to be under a loss threshold and attempt to reduce the dimensionality for each object individually. Furthermore, the query performance is also optimized by deriving the upper and lower bounds of retrieved blocks and calculating distances in a low-embedding space preferentially. Theoretical analysis is provided to support our search strategy. Extensive experiments demonstrate that DForest significantly outperforms all the state-of-the-art competitors in terms of query time, and exhibits good scalability.

The k-Nearest-Neighbors (kNN) search in the high-dimensional space is a fundamental problem in many applications. However, retrieving the nearest neighbors from a large-scale and high-dimensional dataset is computationally challenging, i.e, existing approaches either suffer from high construction cost or unsatisfactory search performance. In this paper, we propose a fast and light-weight learned index FLEX for kNN search in high-dimensional space. First, we develop a multi-module DNN framework that needs much less training examples than employing classical DNN models directly to reach the same accuracy level. Second, we propose a linear-time data layout algorithm which aims to maximize the accuracy under a search cost constraint. A bound of the expected approximation ratio of the proposed algorithm is also proved. Experimental evaluation shows that our index significantly outperforms the state-of-the-art indexes in terms of both time efficiency and space efficiency while maintaining competitive or better accuracy performance.