Can't persist Eclipse Collection's LongArrayList (Custom TypeHandler not working as expected)

[Sandared]

Hi,

I tried to store Eclipse Collection's LongArrayList (https://www.eclipse.org/collections/) with Microstream, but due to it's "items" field being transient this didn't work out of the box.

So what I did was to create a new class PersistableLongArrayList like this:

public class PersistableLongArrayList extends LongArrayList {
    static BinaryTypeHandler<PersistableLongArrayList> provideTypeHandler() {
        return Binary.TypeHandler(PersistableLongArrayList.class,
            Binary.Field_int(
                "size", 
                list -> list.size, 
                (list, value) -> list.size = value
            ),
            Binary.Field(
                long[].class, "items", 
                list -> list.items, 
                (list, value) -> list.items = value
            )
        );
    } 
}

The items field is now not null when loaded from a stored instance, however if I change the value afterwards and call storage.store(list), shutdown the db and restart it the new value is not stored, only size is stored correctly.

I've added a very simple example that shows this behavior:

public class SimpleTest {
    private PersistableLongArrayList root;

    public static void main(String[] args) {
        try {
            SimpleTest t = new SimpleTest();
            // DB erstellen
            EmbeddedStorageManager storage = t.startDB();
            // modify the list
            t.root.add(t.root.size() + 1);
            storage.store(t.root);
            // stop DB
            t.stopDB(storage);
            // restart DB
            storage = t.startDB();
            // show root element
            System.out.println(t.root);
            // Observed behavior: list "grows" from s.th. like this: [1] to s.th. like this [1, 0, 0, 0] after several runs (Probably only size grows and is stored correctly and therefore the fields up to "size" are printed but empty)
            // Expected beahvior: list starts with [1] and grows to something like this [1, 2, 3, 4] after several runs
        } catch (Throwable t) {
            t.printStackTrace();
        } finally {
            System.exit(0);
        }
    }

    private EmbeddedStorageManager startDB() {
        EmbeddedStorageManager storageManager = EmbeddedStorage.start();
        if(storageManager.root() == null) {
            PersistableLongArrayList list = new PersistableLongArrayList();
            storageManager.setRoot(list);
            storageManager.storeRoot();
            root = list;
        } else {
            root = (PersistableLongArrayList)storageManager.root();
        }
        return storageManager;
    }

    private void stopDB(EmbeddedStorageManager storageManager) {
        storageManager.shutdown();
    }
}

Probably I just misunderstood how to use the custom BinaryHandler correctly, but I have no idea what to change right now.
Any advice is appreciated :)

Kind regards,
Thomas

[hg-ms]

Hello,
You found a more complex case, the thing you missed is the “lazy Storing” behavior of Microstream (https://docs.microstream.one/manual/storage/storing-data/lazy-eager-full.html).

Here is what happen in detail:
In your PersistableLongArrayList you defined a typeHandler that stores a reference to a long-Array.
During the first store microstream detects that the referenced array has not been persisted before will persist it. Subsequent stores will detect that the array has been persisted already and skip it because of the lazy storing behavior. If you add enough elements to the LongArrayList the list will internally create new and lager array. This new array will be recognized and persisted once.
You can change that behavior using a custom PersistenceEagerStoringFieldEvaluator.
Regarding the “transient” field you could use a custom PersistenceFieldEvaluator to change the default behavior.

Using a custom PersistenceEagerStoringFieldEvaluator and PersistenceFieldEvaluator it is possible to persist the LongArrayList directly:

public class PersistenceFieldEvaluatorCustom implements PersistenceFieldEvaluator {

	@Override
	public boolean applies(final Class<?> entityType, final Field field) {

		//return true if field should be persisted, false if not
		if(entityType == org.eclipse.collections.impl.list.mutable.primitive.LongArrayList.class && field.getName().equals("items")) {
			return true;
		}

		//default: return false if field has the transient modifier
		return !XReflect.isTransient(field);
	}
}

public class PersistenceEagerStoringFieldEvaluatorCustom implements PersistenceEagerStoringFieldEvaluator {

	@Override
	public boolean isEagerStoring(final Class<?> t, final Field u) {

		//return true if field should be persisted at every store
		if(t == org.eclipse.collections.impl.list.mutable.primitive.LongArrayList.class && u.getName().equals("items"))	{
			return true;
		}
		return false;
	}
}

Setup:

final EmbeddedStorageManager s = EmbeddedStorage.Foundation()
				.onConnectionFoundation(
						c->{ c.setFieldEvaluatorPersistable(new PersistenceFieldEvaluatorCustom());
							 c.setReferenceFieldEagerEvaluator(new PersistenceEagerStoringFieldEvaluatorCustom());
						})
				.start();

[Sandared]

Thank you so much! This solution works like a charm :)
And thanks especially for the detailed explanation of the why 👍

[hg-ms]

[Sandared]

I don't think this answer belongs here. At least it is not referencing anything regarding my (answered) question about persisting LongArrayLists :)

Maybe you accidentally mixed it up with another question? Or it is wrongly displayed under my question?

Kind regards,
Thomas

[hg-ms]

Many thanks. You're right, i mixed it up with an other issue.