CASSGO-11 Feature Request: Support Vector Type #1734
Comments
|
We're also keen to see vector support in gocql, and I think our customers would love to see it as well. Any news on this feature? |
|
Are there plans to implement the many new features in Cassandra 5? |
|
@nkev Personally I don't plan to work on Cassandra 5 support, if anyone else wants to, feel free. See a more detailed response in the mailing list. |
|
@martin-sucha Thanks for the update. Let's hope a gopher (or few) with a deep understanding of C* puts their hand up. |
|
Hello! I will try to handle it. |
During the implementation of the vector type support I found several issues:
|
Could this be because gocql uses protocol v4, which does not have native support for the vector type, while protocol v5 does? |
|
Please open a Cassandra issue about the length issue. |
Exactly! |
|
Hello everyone, For people who are using version 4 of the protocol but need to be able to write vectors from their code base here is a possible idea: package warm
import (
"fmt"
"github.com/gocql/gocql"
)
func encInt(v int32) []byte {
return []byte{byte(v >> 24), byte(v >> 16), byte(v >> 8), byte(v)}
}
func encFloat(v float32) []byte {
return encInt(int32(math.Float32bits(v)))
}
type Float32Vector struct {
value []float32
dimensions int
}
func (m *Float32Vector) MarshalCQL(info gocql.TypeInfo) ([]byte, error) {
if len(m.value) != m.dimensions {
return nil, fmt.Errorf("float32vector expects size %d but received size %d",
m.dimensions, len(m.value))
}
var results []byte
for _, part := range m.value {
results = append(results, encFloat(part)...)
}
return results, nil
}
func (m *Float32Vector) UnmarshalCQL(info gocql.TypeInfo, data []byte) error {
panic("unmarshalling vector is not fully implemented")
}
func (m *Float32Vector) Value() []float32 {
return m.value
}
func ensureVectorDimension(values []float32, dimensions int) []float32 {
if len(values) == dimensions {
return values
}
delta := dimensions - len(values)
result := make([]float32, dimensions)
for idx, v := range values {
if idx < dimensions {
result[idx] = v
} else {
break
}
}
for idx := len(values); idx < delta+len(values); idx++ {
result[idx] = 0.0
}
return result
}
func FromVectorFloat32(value []float32, dimensions int) *Float32Vector {
return &Float32Vector{
value: ensureVectorDimension(value, dimensions),
dimensions: dimensions,
}
}You can opt to skip the padding completely and return an error if that feels more natural for your use case. The encoding functions are extracted from the existing gocql codebase. |
|
@tengu-alt, did make progress on vector support? I cannot find relevant PR opened. Do you mind if I submit my nearly completed PoC? Based on your comment from 22 May, I think you are encoding length of vector element incorrectly. |
I am currently implementing vector type support. I think the error that I mentioned is caused by the cqlsh incorrect work. |
It looks like @lukasz-antoniak already has a functioning prototype, it might be more efficient to just have him open a PR with his work depending on how much progress you have on your work |
Sounds great! I would be glad to see the @lukasz-antoniak PR. Currently I implemented a vector type unmarshal nearly the all datatypes that are supported by the driver (the vector data has a different serialization). The Marshal remains, and the test coverage also. |
|
I have opened #1828. Will continue with more unit and integration tests. |
joao-r-reis
changed the title
Since Cassandra has introduced Vector type, and it's already supported in Python driver, I hope it can be added in gocql as well, and it will resolve this issue: datastax/gocql-astra#17 (comment)
The text was updated successfully, but these errors were encountered: