Urgent - Recovering a corrupted datastore

[fschoning]

I have a Quarkus app that uses Eclipsestore 1.3.1 and S3 storage blobs.

The server crashed and restarted and now the storage will not start up. For various reasons my two other recovery point data backups failed and I only have the latest version of the data, which I assume is somehow corrupted.

I know that v1.3.1 had issues with ungraceful SpringBoot restarts corrupting the data, but it seems that Quarkus may have the same error.

I URGENTLY need to recover this data asap, but I do not know where to start looking. How can I revert to my last working data? Or in worst case manually recover the data from the storage files?

The actual java data structure is very simple:

public class DataRoot {
    Map<String, String> linkMap = new HashMap<>();
    Map<String, String> reverseMap = new HashMap<>();
}

The exception at startup is:

Sep 20 09:56:22 serverb java[2525]: __  ____  __  _____   ___  __ ____  ______
Sep 20 09:56:22 serverb java[2525]:  --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
Sep 20 09:56:22 serverb java[2525]:  -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
Sep 20 09:56:22 serverb java[2525]: --\___\_\____/_/ |_/_/|_/_/|_|\____/___/
Sep 20 09:56:22 serverb java[2525]: 2024-09-20 09:56:22,082 INFO  [io.quarkus] (main) linkshortnener 1.0-SNAPSHOT on JVM (powered by Quarkus 3.9.2) started in 1.984s. Listening on: http://0.0.0.0:8054
Sep 20 09:56:22 serverb java[2525]: 2024-09-20 09:56:22,095 INFO  [io.quarkus] (main) Profile prod activated.
Sep 20 09:56:22 serverb java[2525]: 2024-09-20 09:56:22,096 INFO  [io.quarkus] (main) Installed features: [cdi, rest, smallrye-context-propagation, vertx]
Sep 20 10:02:41 serverb java[2525]: 2024-09-20 10:02:41,235 INFO  [com.coa.lin.Storage] (executor-thread-1) Storage postConstruct
Sep 20 10:02:42 serverb java[2525]: 2024-09-20 10:02:42,591 INFO  [org.ecl.sto.sto.emb.typ.EmbeddedStorageFoundation] (executor-thread-1) Creating embedded storage manager
Sep 20 10:02:42 serverb java[2525]: 2024-09-20 10:02:42,748 INFO  [org.ecl.ser.per.typ.PersistenceTypeHandlerManager] (executor-thread-1) Initializing type handler manager
Sep 20 10:02:43 serverb java[2525]: 2024-09-20 10:02:43,011 INFO  [org.ecl.sto.sto.emb.typ.EmbeddedStorageManager] (executor-thread-1) Starting embedded storage manager
Sep 20 10:02:43 serverb java[2525]: 2024-09-20 10:02:43,113 INFO  [org.ecl.sto.sto.typ.StorageSystem] (executor-thread-1) Starting storage system
Sep 20 10:02:43 serverb java[2525]: 2024-09-20 10:02:43,139 INFO  [org.ecl.sto.sto.typ.StorageStructureValidator] (executor-thread-1) Storage structure validated successfully.
Sep 20 10:02:46 serverb java[2525]: 2024-09-20 10:02:46,484 ERROR [org.ecl.sto.sto.typ.StorageChannelTask] (Eclipse Serializer-StorageChannel-0) Error occurred in storage channel#0: org.eclipse.store.storage>
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageFileManager$Default.readTransactionsFile(StorageFileManager.java:717)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageFileManager$Default.readStorage(StorageFileManager.java:682)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageChannel$Default.readStorage(StorageChannel.java:772)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageChannelTaskInitialize$Default.internalProcessBy(StorageChannelTaskInitialize.java:192)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageChannelTaskInitialize$Default.internalProcessBy(StorageChannelTaskInitialize.java:34)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:244)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageChannel$Default.work(StorageChannel.java:453)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageChannel$Default.run(StorageChannel.java:536)
Sep 20 10:02:46 serverb java[2525]:         at java.base/java.lang.Thread.run(Thread.java:1583)
Sep 20 10:02:46 serverb java[2525]: Caused by: org.eclipse.store.storage.exceptions.StorageException: 0 No file found in entries with number 3
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageTransactionsAnalysis$EntryAggregator.handleEntryFileDeletion(StorageTransactionsAnalysis.java:877)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageTransactionsAnalysis$EntryAggregator.accept(StorageTransactionsAnalysis.java:704)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageTransactionsAnalysis$Logic.processBufferedEntities(StorageTransactionsAnalysis.java:378)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageTransactionsAnalysis$Logic.processInputFile(StorageTransactionsAnalysis.java:340)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageTransactionsAnalysis$Logic.processInputFile(StorageTransactionsAnalysis.java:298)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageLiveTransactionsFile$Default.processBy(StorageLiveTransactionsFile.java:86)
Sep 20 10:02:46 serverb java[2525]:         at org.eclipse.store.storage.types.StorageFileManager$Default.readTransactionsFile(StorageFileManager.java:711)
Sep 20 10:02:46 serverb java[2525]:         ... 8 more

[fschoning]

The s3 bucket has versioning enabled. Would it be possible to use a tool like https://github.com/angeloc/s3-pit-restore to get a copy of the bucket at a previous point in time and the data will be consistent. I think that deleted s3 objects will not be recoverable. Does eclipsestore s3 blob storage delete files?

[fschoning]

I have restored several previous versions from the s3 bucket and the same exception always gets thrown!

[fschoning]

I successfully recovered the datastore. It appears that there is a problem in 1.3.1 that was fixed in 1.4.0 and after upgrading to 1.4.0 the store was successfully opened without the error.