[Limitation] Maximum number of non-zero elements must be < Int32.Max

[michabirklbauer]

The current implementation uses the signed int32 data type to keep track of the number of non-zero elements in the matrices. Therefore the number of non-zero elements must no exceed 2 147 483 647 or otherwise unexpected behaviour will occur!

In case matrices with more than 2 * 10⁹ non-zero elements are expected, any variable keeping track of the non-zero elements needs to have its data type changed! This applies to both the C# code as well as the C++ code!

Here's an example:

CandidateVectorSearch/VectorSearch/dllmain.cpp

Lines 121 to 127 in bd84eca

	int* findTopCandidates(int* candidatesValues, int* candidatesIdx,
	int* spectraValues, int* spectraIdx,
	int cVLength, int cILength,
	int sVLength, int sILength,
	int n, float tolerance,
	bool normalize, bool gaussianTol,
	int cores, int verbose) {

cVLength as well as variables that use cVLength would need to be changed to long or int64_t. For safety it would be recommended to do the same for cILength, sVLength and sILength - even though it might not be necessary.

Wherever this function (or any of the other matrix multiplication functions) is used on the C# side, the types need to be adjusted as well (to long) - most importantly in the VectorSearchInterface.

We will not supply an implementation for matrices of this size but are happy to assist in case help is needed!