Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/add big objects support for meta on chain #3063

Merged
merged 6 commits into from
Dec 28, 2024
+137 −50
Merged

Feat/add big objects support for meta on chain #3063

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b194cc2
core/object/replicate: make `validuntil` camel-cased
carpawell Dec 23, 2024
2bbea7f
pkg/core/object: drop epoch-bases comment to meta encoding func
carpawell Dec 25, 2024
a87f16d
node/object/put: allow optional fields
carpawell Dec 25, 2024
228d4ac
node/object/put: push first and previous parts to chain
carpawell Dec 25, 2024
06e7020
node/object/put: add object type to meta-on-chain
carpawell Dec 25, 2024
8ebd425
node/object/replicate_test: fix replication signature test
carpawell Dec 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 34 additions & 16 deletions pkg/core/object/replicate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,47 +5,65 @@ import (

"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
objectsdk "github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
)

const (
currentVersion = 7 // it is also a number of fields
)

const (
networkMagicKey = "network"
// required fields.
cidKey = "cid"
oidKey = "oid"
sizeKey = "size"
validUntilKey = "validUntil"
networkMagicKey = "network"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

motivated movement?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same as in #3063 (comment)

ok-ok, waste of time, more code to review, etc. but i feel better this way


// optional fields.
firstPartKey = "firstPart"
previousPartKey = "previousPart"
deletedKey = "deleted"
lockedKey = "locked"
validUntilKey = "validuntil"
typeKey = "type"
)

// EncodeReplicationMetaInfo uses NEO's map (strict order) serialized format as a raw
// representation of object's meta information.
//
// This (ordered) format is used (keys are strings):
//
// "network": network magic
// "cid": _raw_ container ID (32 bytes)
// "oid": _raw_ object ID (32 bytes)
// "size": payload size
// "deleted": array of _raw_ object IDs
// "locked": array of _raw_ object IDs
// "validuntil": last valid block number for meta information
//
// Last valid epoch is object's creation epoch + 10.
func EncodeReplicationMetaInfo(cID cid.ID, oID oid.ID, pSize uint64,
// "validUntil": last valid block number for meta information
// "network": network magic
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better keep them alphabetically sorted (in optional/non-optional groups).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in optional/non-optional groups)

👍

alphabetically sorted

👎. Excessive load to the dev brain "where to place new key?". And i like how they are grouped by type currently

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better keep them alphabetically sorted

well, my suggestion is not a random order, it was an intention. i grouped it like "logically" or as @cthulhu-rider said, currently, it is by type too. do you have any pros as to why it is better to sort them? new value will be harder to add, and alphabet sorting has no advantages (or change my mind)

// "firstPart": [OPTIONAL] _raw_ object ID (32 bytes)
// "previousPart": [OPTIONAL] _raw_ object ID (32 bytes)
// "deleted": [OPTIONAL] array of _raw_ object IDs
// "locked": [OPTIONAL] array of _raw_ object IDs
// "type": [OPTIONAL] object type enumeration
func EncodeReplicationMetaInfo(cID cid.ID, oID, firstPart, previousPart oid.ID, pSize uint64, typ objectsdk.Type,
deleted, locked []oid.ID, vub uint64, magicNumber uint32) []byte {
kvs := []stackitem.MapElement{
kv(networkMagicKey, magicNumber),
roman-khimov marked this conversation as resolved.
Show resolved Hide resolved
kv(cidKey, cID[:]),
kv(oidKey, oID[:]),
kv(sizeKey, pSize),
oidsKV(deletedKey, deleted),
oidsKV(lockedKey, locked),
kv(validUntilKey, vub),
kv(networkMagicKey, magicNumber),
}

if !firstPart.IsZero() {
kvs = append(kvs, kv(firstPartKey, firstPart[:]))
}
if !previousPart.IsZero() {
kvs = append(kvs, kv(previousPartKey, previousPart[:]))
}
if len(deleted) > 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd split unit tests into min and full cases. And include fields added in the following commits

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

kvs = append(kvs, oidsKV(deletedKey, deleted))
}
if len(locked) > 0 {
kvs = append(kvs, oidsKV(lockedKey, locked))
}
if typ != objectsdk.TypeRegular {
kvs = append(kvs, kv(typeKey, uint32(typ)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that tombstone/lock can be detected by deleted/locked already. But this covers more things, so OK.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean lock type with some non-empy deleted should be an error (panic in fact) in this func?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

although typ is uint32, lets to cast to int32. It will be int32 soon (im also unhappy bout this, but wcyd). We'll most likely forget to change conversion later

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

}

result, err := stackitem.Serialize(stackitem.NewMapWithValue(kvs))
Expand Down
98 changes: 74 additions & 24 deletions pkg/core/object/replicate_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,51 +6,101 @@ import (
"testing"

"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
cid "github.com/nspcc-dev/neofs-sdk-go/container/id"
cidtest "github.com/nspcc-dev/neofs-sdk-go/container/id/test"
"github.com/nspcc-dev/neofs-sdk-go/object"
oid "github.com/nspcc-dev/neofs-sdk-go/object/id"
oidtest "github.com/nspcc-dev/neofs-sdk-go/object/id/test"
"github.com/stretchr/testify/require"
)

type m struct {
cID cid.ID
oID oid.ID
size uint64
vub uint64
magic uint32

first oid.ID
prev oid.ID
deleted []oid.ID
locked []oid.ID
typ object.Type
}

func TestMetaInfo(t *testing.T) {
network := rand.Uint32()
oID := oidtest.ID()
cID := cidtest.ID()
size := rand.Uint64()
deleted := oidtest.IDs(10)
locked := oidtest.IDs(10)
validUntil := rand.Uint64()

raw := EncodeReplicationMetaInfo(cID, oID, size, deleted, locked, validUntil, network)
meta := m{
cID: cidtest.ID(),
oID: oidtest.ID(),
size: rand.Uint64(),
vub: rand.Uint64(),
magic: rand.Uint32(),
first: oidtest.ID(),
prev: oidtest.ID(),
deleted: oidtest.IDs(10),
locked: oidtest.IDs(10),
typ: object.TypeTombstone,
}

t.Run("full", func(t *testing.T) {
testMeta(t, meta, true)
})

t.Run("no optional", func(t *testing.T) {
meta.first = oid.ID{}
meta.prev = oid.ID{}
meta.deleted = nil
meta.deleted = nil
meta.locked = nil
meta.typ = object.TypeRegular

testMeta(t, meta, false)
})
}

func testMeta(t *testing.T, m m, full bool) {
raw := EncodeReplicationMetaInfo(m.cID, m.oID, m.first, m.prev, m.size, m.typ, m.deleted, m.locked, m.vub, m.magic)
item, err := stackitem.Deserialize(raw)
require.NoError(t, err)

require.Equal(t, stackitem.MapT, item.Type())
mm, ok := item.Value().([]stackitem.MapElement)
require.True(t, ok)

require.Len(t, mm, currentVersion)
require.Equal(t, cidKey, string(mm[0].Key.Value().([]byte)))
require.Equal(t, m.cID[:], mm[0].Value.Value().([]byte))

require.Equal(t, oidKey, string(mm[1].Key.Value().([]byte)))
require.Equal(t, m.oID[:], mm[1].Value.Value().([]byte))

require.Equal(t, networkMagicKey, string(mm[0].Key.Value().([]byte)))
require.Equal(t, network, uint32(mm[0].Value.Value().(*big.Int).Uint64()))
require.Equal(t, sizeKey, string(mm[2].Key.Value().([]byte)))
require.Equal(t, m.size, mm[2].Value.Value().(*big.Int).Uint64())

require.Equal(t, cidKey, string(mm[1].Key.Value().([]byte)))
require.Equal(t, cID[:], mm[1].Value.Value().([]byte))
require.Equal(t, validUntilKey, string(mm[3].Key.Value().([]byte)))
require.Equal(t, m.vub, mm[3].Value.Value().(*big.Int).Uint64())

require.Equal(t, networkMagicKey, string(mm[4].Key.Value().([]byte)))
require.Equal(t, m.magic, uint32(mm[4].Value.Value().(*big.Int).Uint64()))

if !full {
require.Len(t, mm, 5)
return
}

require.Equal(t, oidKey, string(mm[2].Key.Value().([]byte)))
require.Equal(t, oID[:], mm[2].Value.Value().([]byte))
require.Equal(t, firstPartKey, string(mm[5].Key.Value().([]byte)))
require.Equal(t, m.first[:], mm[5].Value.Value().([]byte))

require.Equal(t, sizeKey, string(mm[3].Key.Value().([]byte)))
require.Equal(t, size, mm[3].Value.Value().(*big.Int).Uint64())
require.Equal(t, previousPartKey, string(mm[6].Key.Value().([]byte)))
require.Equal(t, m.prev[:], mm[6].Value.Value().([]byte))

require.Equal(t, deletedKey, string(mm[4].Key.Value().([]byte)))
require.Equal(t, deleted, stackItemToOIDs(t, mm[4].Value))
require.Equal(t, deletedKey, string(mm[7].Key.Value().([]byte)))
require.Equal(t, m.deleted, stackItemToOIDs(t, mm[7].Value))

require.Equal(t, lockedKey, string(mm[5].Key.Value().([]byte)))
require.Equal(t, locked, stackItemToOIDs(t, mm[5].Value))
require.Equal(t, lockedKey, string(mm[8].Key.Value().([]byte)))
require.Equal(t, m.locked, stackItemToOIDs(t, mm[8].Value))

require.Equal(t, validUntilKey, string(mm[6].Key.Value().([]byte)))
require.Equal(t, validUntil, mm[6].Value.Value().(*big.Int).Uint64())
require.Equal(t, typeKey, string(mm[9].Key.Value().([]byte)))
require.Equal(t, int(m.typ), int(mm[9].Value.Value().(*big.Int).Uint64()))
}

func stackItemToOIDs(t *testing.T, value stackitem.Item) []oid.ID {
Expand Down
16 changes: 12 additions & 4 deletions pkg/network/transport/object/grpc/replication.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,9 +206,17 @@
}

func (s *Server) metaInfoSignature(o object.Object) ([]byte, error) {
firstObj := o.GetFirstID()
if o.HasParent() && firstObj.IsZero() {
// object itself is the first one
firstObj = o.GetID()
}

Check warning on line 213 in pkg/network/transport/object/grpc/replication.go

View check run for this annotation

Codecov / codecov/patch

pkg/network/transport/object/grpc/replication.go#L211-L213 

Added lines #L211 - L213 were not covered by tests
prevObj := o.GetPreviousID()

var deleted []oid.ID
var locked []oid.ID
switch o.Type() {
typ := o.Type()
switch typ {
case object.TypeTombstone:
var t object.Tombstone
err := t.Unmarshal(o.Payload())
Expand All @@ -235,9 +243,9 @@
secondBlock := firstBlock + currentEpochDuration
thirdBlock := secondBlock + currentEpochDuration

firstMeta := objectcore.EncodeReplicationMetaInfo(o.GetContainerID(), o.GetID(), o.PayloadSize(), deleted, locked, firstBlock, s.mNumber)
secondMeta := objectcore.EncodeReplicationMetaInfo(o.GetContainerID(), o.GetID(), o.PayloadSize(), deleted, locked, secondBlock, s.mNumber)
thirdMeta := objectcore.EncodeReplicationMetaInfo(o.GetContainerID(), o.GetID(), o.PayloadSize(), deleted, locked, thirdBlock, s.mNumber)
firstMeta := objectcore.EncodeReplicationMetaInfo(o.GetContainerID(), o.GetID(), firstObj, prevObj, o.PayloadSize(), typ, deleted, locked, firstBlock, s.mNumber)
secondMeta := objectcore.EncodeReplicationMetaInfo(o.GetContainerID(), o.GetID(), firstObj, prevObj, o.PayloadSize(), typ, deleted, locked, secondBlock, s.mNumber)
thirdMeta := objectcore.EncodeReplicationMetaInfo(o.GetContainerID(), o.GetID(), firstObj, prevObj, o.PayloadSize(), typ, deleted, locked, thirdBlock, s.mNumber)

var firstSig neofscrypto.Signature
var secondSig neofscrypto.Signature
Expand Down
11 changes: 7 additions & 4 deletions pkg/network/transport/object/grpc/replication_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"crypto/rand"
"encoding/binary"
"errors"
"fmt"
"testing"

objectV2 "github.com/nspcc-dev/neofs-api-go/v2/object"
Expand Down Expand Up @@ -135,6 +136,8 @@ func anyValidRequest(tb testing.TB, signer neofscrypto.Signer, cnr cid.ID, objID
obj.SetType(object.TypeRegular)
obj.SetContainerID(cnr)
obj.SetID(objID)
obj.SetFirstID(oidtest.ID())
obj.SetPreviousID(oidtest.ID())

sig, err := signer.Sign(objID[:])
require.NoError(tb, err)
Expand Down Expand Up @@ -397,7 +400,7 @@ func TestServer_Replicate(t *testing.T) {

sigsRaw := resp.GetObjectSignature()

for i := range 1 {
for i := range 3 {
var sigV2 refsv2.Signature
l := binary.LittleEndian.Uint32(sigsRaw)

Expand All @@ -408,10 +411,10 @@ func TestServer_Replicate(t *testing.T) {

require.Equal(t, signer.PublicKeyBytes, sig.PublicKeyBytes())
require.True(t, sig.Verify(objectcore.EncodeReplicationMetaInfo(
o.GetContainerID(), o.GetID(), o.PayloadSize(), nil, nil,
uint64((123+1+i)*240), mNumber)))
o.GetContainerID(), o.GetID(), o.GetFirstID(), o.GetPreviousID(), o.PayloadSize(), o.Type(), nil, nil,
uint64((123+1+i)*240), mNumber)), fmt.Sprintf("wrong %d signature", i+1))

sigsRaw = sigsRaw[:4+l]
sigsRaw = sigsRaw[4+l:]
}
})
})
Expand Down
12 changes: 10 additions & 2 deletions pkg/services/object/put/distributed.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,13 @@
t.encodedObject.b = nil
}()

firstObj := t.obj.GetFirstID()
if t.obj.HasParent() && firstObj.IsZero() {
// object itself is the first one
firstObj = t.obj.GetID()
}
prevObj := t.obj.GetPreviousID()

Check warning on line 130 in pkg/services/object/put/distributed.go

View check run for this annotation

Codecov / codecov/patch

pkg/services/object/put/distributed.go#L124-L130 

Added lines #L124 - L130 were not covered by tests
t.obj.SetPayload(t.encodedObject.b[t.encodedObject.pldOff:])

tombOrLink := t.obj.Type() == objectSDK.TypeLink || t.obj.Type() == objectSDK.TypeTombstone
Expand All @@ -143,7 +150,8 @@

var deletedObjs []oid.ID
var lockedObjs []oid.ID
switch t.objMeta.Type() {
typ := t.objMeta.Type()
switch typ {

Check warning on line 154 in pkg/services/object/put/distributed.go

View check run for this annotation

Codecov / codecov/patch

pkg/services/object/put/distributed.go#L153-L154 

Added lines #L153 - L154 were not covered by tests
case objectSDK.TypeTombstone:
deletedObjs = t.objMeta.Objects()
case objectSDK.TypeLock:
Expand All @@ -152,7 +160,7 @@
}

expectedVUB := (uint64(t.currentBlock)/t.currentEpochDuration + 2) * t.currentEpochDuration
t.objSharedMeta = object.EncodeReplicationMetaInfo(t.obj.GetContainerID(), t.obj.GetID(), t.obj.PayloadSize(), deletedObjs,
t.objSharedMeta = object.EncodeReplicationMetaInfo(t.obj.GetContainerID(), t.obj.GetID(), firstObj, prevObj, t.obj.PayloadSize(), typ, deletedObjs,

Check warning on line 163 in pkg/services/object/put/distributed.go

View check run for this annotation

Codecov / codecov/patch

pkg/services/object/put/distributed.go#L163 

Added line #L163 was not covered by tests
lockedObjs, expectedVUB, t.networkMagicNumber)
id := t.obj.GetID()
err := t.placementIterator.iterateNodesForObject(id, t.sendObject)
Expand Down
Loading