# DiskHIVF: Disk-Resident Hierarchical Inverted File Index for Billion-scale Approximate Nearest Neighbor Search
## Compile
```
sh make.sh
```

## Train model and build index
```train and build
./bin/disk_hivf_train_and_build <conf_file>
```
**conf_file:** Configuration file with the following parameters:
```
train_data_file = ./data/sift/sift_learn.dim.fvecs # Training dataset
index_data_file = ./data/sift/sift_base.dim.fvecs # Original file for index vectors
model_file = ./output/sift1m_model # Model output file
index_dir = .//output/sift1m_index # Index output directory
index_file_num = 1  # Number of files to save the index
dim = 128 # Vector dimension
kmeans_epoch = 20 # Number of kmeans training epochs
kmeans_sample_rate = 1 # Sampling ratio for the training dataset
batch_size = 128 # Batch size for kmeans training, usually no need to adjust
kmeans_centers_select_type = 3 # Initialization method for kmeans centers: 1-random selection, 2-use first k vectors, 3-kmeans++ initialization, usually no need to adjust
first_cluster_num = 300 # Number of first-level clusters
second_cluster_num = 300 # Number of second-level clusters
hierarchical_cluster_epoch = 5   # Number of epochs for hierarchical kmeans training
read_file_batch_size = 128  # Batch size for reading data
build_index_search_first_center_num = 20    # Number of first-level centers searched during training, i.e., r_building in the paper
search_first_center_num = 20    # Number of first-level centers searched during inference
search_second_center_num = 400  # Number of second-level centers searched during inference
is_disk = 1 # 0-pure memory mode, 1-disk mode enabled (default is disk mode)
search_neighbors = 100000   # Maximum number of vectors to search during inference
search_block_num = 20000    # Maximum number of disk/memory blocks to query during inference
search_top_cut = 7  # During inference, cells with distances more than search_top_cut times the current best solution are pruned
hs_mode = 1 # 0-store vector IDs on disk, 1-store vector IDs in memory. Usually hs_mode = 1 is 10% faster than hs_mode = 0 but requires more memory
thread_num = 32 # Number of training threads
read_index_file_thread_num = 5  # Number of threads for reading indexes
build_index_num = -1 # Use the first build_index_num vectors from index_data_file to build the index, -1-use all data
train_data_num = -1 # Use the first train_data_num vectors from train_data_file to train the model, -1-use all data
use_uint8_data = 0  # Data type of train_data_file and index_data_file: 0-uint8, 1-float
dynamic_prune_switch = 0 # 0-disable dynamic pruning, 1-enable dynamic pruning
dynamic_prune_a = 165.4 # Effective when dynamic_prune_switch = 1, generated by learn_dynamic_prune_hyperparameter.sh
dynamic_prune_b = -30.9 # Effective when dynamic_prune_switch = 1, generated by learn_dynamic_prune_hyperparameter.sh
dynamic_prune_c = 2.4 # Effective when dynamic_prune_switch = 1, generated by learn_dynamic_prune_hyperparameter.sh
io_thread_num = 0 # Not implemented
is_async_read = 0 # Not implemented
use_cache = 0 # Not implemented
cache_capacity = 0 # Not implemented
cache_segment = 0 # Not implemented
```

## Data Format

**.dim.fvec:** This is a binary format that begins with a `uint32` variable `dim` representing the vector dimension and a `uint64` variable `n` representing the number of vectors. Following these are `n * dim` `float`s, representing `n` `float` vectors.

**.dim.ivec:** This is a binary format that begins with a `uint32` variable `dim` representing the vector dimension and a `uint64` variable `n` representing the number of vectors. Following these are `n * dim` `int32`s, representing `n` `int32` vectors.

**.dim.bvec:** This is a binary format that begins with a `uint32` variable `dim` representing the vector dimension and a `uint64` variable `n` representing the number of vectors. Following these are `n * dim` `int8`s, representing `n` `int8` vectors.

## Dynamic Prune
Generate hyperparameters for dynamic pruning using least squares method
```
sh learn_dynamic_prune_hyperparameter.sh <conf_file> <query_file>
```
**conf_file:** Configuration file with parameters as above  
**query_file:** Query vector file

## Run Test Set
Run the test set for evaluation
```
./bin/run_test_set <conf_file> <query_file> <groundtruth_file> <topk> <at_num> <thread_num> <first_centers_num> <second_centers_num> <debug_log> <use_cache> <is_query_uint8> <use_dist> <search_neighbors> <search_blocks>
```
**conf_file:** Configuration file with parameters as above  
**query_file:** Query vector file  
**groundtruth_file:** Ground truth for query vectors  
**topk:** Evaluation metric topk-recall@at_num  
**at_num:** Evaluation metric topk-recall@at_num  
**thread_num:** Number of threads for evaluation, usually set to 1  
**first_centers_num:** Number of first-level centers searched during inference, overrides the parameter in conf_file  
**second_centers_num:** Number of second-level centers searched during inference, overrides the parameter in conf_file  
**debug_log:** 0-disable debug logging, 1-enable debug logging  
**use_cache:** Not implemented, usually set to 0  
**is_query_uint8:** 1-query vectors are uint8, 0-query vectors are float  
**use_dist:** Ground truth file contains distances for evaluation  
**search_neighbors:** Maximum number of vectors to search during inference, overrides the parameter in conf_file  
**search_blocks:** Maximum number of disk/memory blocks to query during inference, overrides the parameter in conf_file  

### Example
```
./bin/run_test_set conf/hivf.conf data/sift/sift_query.dim.fvecs data/sift/sift_groundtruth.dim.ivecs 10 10 1 70 5000 0 0 0 1
```