In this chapter, you will learn how to build a basic DEC App template project from scratch, understand the project structure of the project, learn about object modeling, and implement the function of adding, deleting, modifying, and checking the message service.
- Make sure you have exported your identity to your computer via CyberChat.
-
For mac systems, there are poeple.desc and people.sec files in the ~/.cyfs_profile directory respectively
-
Windows system, there are poeple.desc and people.sec files in the C:\Users<username>\.cyfs_profile directory respectively
- Make sure you have the latest stable version of nodejs installed in your system.
- Install the cyfs-sdk, cyfs-tool, and cyfs-dapp-cli scaffolding globally.
- Use cyfs-dapp-cli to create the dec app project template in the folder you think is appropriate. The command is as follows:
npm i -g cyfs-sdk cyfs-tool cyfs-dapp-cli
cyfs-dapp-cli create message-board -t message-board-demo
cd message-board
npm i --force
cyfs-dapp-cli use note
If you change the storage path of the identity file when you use CyberChat to export your identity to the computer, you need to use -o <peoplePath>
to specify the identity file when creating a project with cyfs-dapp-cli, such as - o ~/.cyfs_profile/people
, otherwise an error will be reported because the identity file cannot be found.
Development phase: Complete the front-end and service-side development under the src folder. Publishing stage: Package the file resources required by the front-end and service-side, depending on the configuration of cyfs.config.json. Installation and running phases: DEC Service starts, runs and terminates on OOD, depending on the configuration of service_package.cfg.
- config_version: the version number of the configuration file
- app_name: The name entered in
cyfs create -n
is also the name of the app. The app names of the same owner cannot be the same. Support non-ASCII characters such as Chinese, the name will be displayed on the app store and app management interface - version: Indicates the
current version
of the App. When cyfs deploy is executed, the current version of the App will be packaged and deployed. After deployment, the third digit of the version will be automatically incremented by one as the new current version - description: The description of the App. Each time it is deployed, the description text filled in here will be updated to the App object and displayed on the App Store.
- icon: App icon, only supports cyfs link. Update every time you deploy, display on the App Store and App Admin UI
- service: Configure Dec App Service related packaging and running configuration, the details are as follows
- pack: Indicates which directories need to be packaged when packaging the Service
- type: Specifies the type of service, currently supports "node" and "rust" When specified as "node", the package directory of the service, the cyfs directory, and the package.json file will be copied to the ${dist}/service/${target} directory When it is specified as "rust", only the package directory of the service is copied to ${dist}/service, and the compilation of the service needs to be manually performed by the user in advance
- dist_targets: Specifies which platforms the service supports. The default is the same as the platforms supported by ood. If the Dec App is installed on a platform that is supported by OOD but not supported by service, an installation failure error will be returned
- app_config: configure the run configuration of the service. You can configure a different run configuration for each platform by adding the [target_name]:[config_path] field. The default default indicates the default configuration used when there is no corresponding configuration for the platform
- web: Configure the relevant packaging and runtime configuration of the Dec App Web section, as detailed below
- folder: The root directory for configuring the Dec App Web section
- entry: The file name of the configurable web home page, the current configuration modification is invalid, the home page file name must be index.html
- dist: The name of the packaging temporary directory, the default is "dist". If the default value conflicts with other local directories, it can be changed to other non-conflicting directory names
- owner_config: owner configuration file
- ext_info: Other configurations unrelated to App deployment, installation, and running, generally related to App display. Details are as follows
- medias: Configure the media information related to the app, which is currently used in the DecApp details page of the app store
- app_id: The app id calculated from owner id and app_name, cannot be modified by yourself. Anywhere in your code you need to use the Dec App Id, you need to use the ID shown here.
- id: identification
- version: version number
- install: install script
- start: start the script
- stop: stop the script
- status: run the status query script
In the project root directory, there are 4 folders and 11 files:
- .cyfs: meta information of dec app
- doc: Project related documentation
- src: the main folder of the project
- tools: tools related to the project.
- .eslintignore: eslintignore file
- .eslintrc.json: eslint configuration file
- .gitignore: gitignore file
- cyfs.config.json: The configuration file for publishing dec app to ood
- move_deploy.js: necessary file transfer before publishing dec app to ood
- package.json: project configuration file
- prettier.config.js: prettier configuration file
- README.md: README file
- service_package.cfg: dec service run configuration file in ood
- tsconfig.json: typescript configuration file
- typing.d.ts: global declaration file
Note There is an attribute of app_id in the cyfs.config.json file. app_id is the app id calculated from owner id and app_name, and cannot be modified by yourself. There are places in the code where the Dec App Id must be used, please use the ID shown here.
Under the src directory, there are 3 subfolders:
- common: code that can be reused by the front end and the service end
- service: Service engineering code
- www: front-end engineering code
Several files in the src/common/cyfs_helper folder encapsulate MetaClient, Stack and various common and complex operations, which are convenient and simple to use.
- Modify the DEC_ID_BASE58 constant value in the src/common/constant.ts source code to your own dec app id, in cyfs.config.json -> app_id
- Change the APP_NAME constant value in the src/common/constant.ts source code to your own dec app name, in cyfs.config.json -> app_name
In the CYFS network, all custom objects must inherit from the cyfs base object NamedObject, this NamedObject will provide some common properties, and each object will have these properties. NamedObject is object data that has specific meaning, cannot be tampered with, and can be confirmed. Now, let's review and learn a little bit about NamedObject.
In the CYFS network, structured data is stored in the format of "named objects", which is a general data format and an abstraction of Internet data by CYFS. Any structured data can be interpreted as a NamedObject. NamedObject is divided in structure: the immutable part Desc, the variable part Body, and the signatures of these two parts DescSign, BodySign.
Each NamedObject has a unique object Id (ObjectId) in the whole network. We use ObjectId to distinguish each object. This ObjectId is obtained by Hash calculation of the Desc part. If the Desc part changes, so does the ObjectId, which is "created a new object" instead of "modified an old one"
In contrast, the Body part of a NamedObject has changed, and its ObjectId remains unchanged. When the updated object is propagated in the CYFS network, the CYFS node will decide whether to update the object on this node according to the update_time of the Body . In order to ensure the consistency of the NamedObject Body part in the decentralized system, the object with the Body is either on the chain, and its consistency is guaranteed by the blockchain, or it is only used in the Zone, and its consistency is guaranteed by the traditional centralized method. . In development practice, we generally discourage the use of Body.
The Desc and Body parts of NamedObject are also composed of two parts: standard properties, and custom DescContent and BodyContent. When customizing a NamedObject, we can only customize its Content part. In principle, the standard property header part of NamedObject should be used as far as possible to store data, and the data not included in the standard property header should be stored in the custom Content.
The data stored in the standard attributes will be automatically recognized by the CYFS storage engine and protocol, which will bring some convenience in storage and query.
- obj_dec_type: number, must have and cannot be modified, indicating the type of a NamedObject. The obj_type of the CYFS standard type is always 0, and each type in the standard type is distinguished by obj_type_code().
- dec_id: ObjectId, optional, if set, it indicates which Dec App created this named object type. The type of a DecApp object can be uniquely determined by dec_id and obj_dec_type
- prev: ObjectId, optional, the meaning is generally "previous object of this object", if there is a definite relationship between a series of objects, you can use this field
- create_timestamp: HashValue, optional, generally indicates the Hash value of an external reference when the object is created. A common choice is to use the latest block hash of the current public chain like BTC or ETH
- create_time: u64, required, indicates the timestamp when the object was created, the default is the current timestamp. If you explicitly set the value to 0, it means that the ObjectId of this object is not related to the creation time, but only related to the content of the object.
- expired_time:u64, optional, indicating the expiration time of the object
- owner: ObjectId, optional, indicating the "owner" of this object. It is the most important standard attribute. Except for a few standard rights objects such as People and Device, any object should set an owner from the perspective of property rights and sign it with the owner's private key. It should be noted that the owner of the Device object is the People object. Therefore, the difference between being owned by a People object user and being owned by a Device object is that the Device object is a secondary identity object.
- area: ObjectId, optional, indicating the area to which the object was created. The filling of the region can be filled in according to the real situation. CYFS is designed around the jurisdiction of the region with data property rights.
- author: ObjectId, optional, indicating the author of this content. For example, an old book whose owner is the publisher, but the author must be an old friend.
- public_key: PublicKeyValue, optional, only authorized objects have this value
- prev_version: HashValue, optional, indicating the pre-body Hash of this Body
- update_time: u64, required, indicating the modification time of the Body content. By default, the CYFS infrastructure will determine whether the object needs to be updated according to update_time. If the update_time of the new object is newer than the current one, update the locally stored object, otherwise do not update
In the cyfs system, we divide objects into 3 categories:
- Standard Object: There are 15 built-in objects in cyfs sdk, occupying bits 1-15 of ObjectTypeCode, and the interval range is [1, 15]. These objects represent our basic abstraction of CYFS Network. usage.
- Core (semantic) object CoreObject: cyfs sdk built-in object, uniformly occupies the 16th bit of ObjectTypeCode, the interval range [32, 32767], and the object is distinguished by the obj_type field. These objects are objects that will have common semantics between different applications in the future and are maintained by standardization organizations. Usually a DECObject defined by a popular application will be upgraded to a CoreObject after it becomes the de facto standard.
- Application object DECObject: The object that is extended by the App developer and has dec_id set, also occupies the 16th bit of ObjectTypeCode (greater than 32767), and distinguishes objects through the obj_type field. Note: An object will be treated as a DecObject only after the dec_id field is set. If not set, it will be mistakenly treated as a CoreObject
Due to the relationship between the Desc part of NamedObject and ObjectId, this requires us to introduce stable coding technology. protobuf is a widely used protocol for serializing structured data, and its stable encoding properties just meet our needs. Use the proto description file to define the structure of the data, then use the protoc compiler to generate source code, and use various languages to write and read structured data in various data streams. Even data structures can be updated without breaking deployed programs compiled from old data structures. 2 important concepts about protobuf:
- Serialization: Converting structured data or objects into a format that can be used for storage and transmission.
- Deserialization: In other computing environments, the serialized data is restored to structured data and objects.
In NamedObject, the desc length is currently limited to 64KB, but the data size of many scenarios today is difficult to exceed this limit. With this restriction, we hope to help developers seriously distinguish structured data from unstructured data. Many times you don't need a larger NamedObject, but should use unstructured data (Chunk). For example, the content of a book or the base64 encoding of a picture, you can consider using the following reference methods to solve:
- Save the data object itself as a chunk, and the extended EBookObject saves the corresponding chunkid.
- See src/common/objs/obj_proto.proto for the complete code
Each message in the message board is a NamedObject, and we call the NameObject representing the message a message object. The message object contains two properties: key and content. key is the unique identifier of the message object, and content is the text content of the message. According to this design, a message will not be particularly long due to the previous 64KB limit. The definition in the porto file is as follows:
message Message {
string key = 1;
string content = 2;
}
Different systems operate differently when using protobuf to generate static stubs.
- For mac system, in the project root directory, execute the command as follows:
npm run proto-mac-pre
npm run proto-mac
Explain these 2 commands:
- proto-mac-pre: Install
ts-protoc-gen
globally, which can generate the corresponding declaration file for us. After that, modify the executable permissions of theprotoc
program in thetools
directory. - proto-mac: Use
protoc
to generate static stub code.
Since it is a directly executed protoc executable, a pop-up window may prompt cannot open "protoc" because the developer cannot be verified, and the developer needs to set it according to the following path: System Preferences -> Security & Privacy -> Allow Apps Downloaded from -> Select App Store and Approved Developers -> Click Allow Still Follow this path to set it up and execute the command again.
- Windows system, in the project root directory, run the following command:
npm run proto-windows
After running, two files, obj_proto_pb.d.ts and obj_proto_pb.js, are generated in the src/common/objs folder. In the obj_proto_pb.d.ts declaration file, we can see the stub code of the Message object as follows:
export class Message extends jspb.Message {
getKey(): string;
setKey(value: string): void;
getContent(): string;
setContent(value: string): void;
serializeBinary(): Uint8Array;
toObject(includeInstance?: boolean): Message.AsObject;
static toObject(includeInstance: boolean, msg: Message): Message.AsObject;
static extensions: { [key: number]: jspb.ExtensionFieldInfo<jspb.Message> };
static extensionsBinary: {
[key: number]: jspb.ExtensionFieldBinaryInfo<jspb.Message>;
};
static serializeBinaryToWriter(
message: Message,
writer: jspb.BinaryWriter
): void;
static deserializeBinary(bytes: Uint8Array): Message;
static deserializeBinaryFromReader(
message: Message,
reader: jspb.BinaryReader
): Message;
}
The four important contents in the stub code of the above Message object are explained:
- Read and set key values using getKey and setKey respectively
- Use ge to read and set the content value respectively tContent and setContent
- Encode the Message object as Uint8Array, use serializeBinary
- Decode the Uint8Array format of the Message object into a Message object, use deserializeBinary
Earlier, we have used protobuf to generate static stub code, but if the object can be used in CYFS, it is also necessary to create its own definition file for each object. We can use cyfs-objs-generator to easily achieve this purpose. cyfs-objs-generator can be used to convert a defined .proto file into a .ts file of one or more custom objects. Currently supported .proto types are: double, float, int32, int64, uint32, uint64, bool, string, bytes and enum.
cyfs-objs-generator does not support nested definitions temporarily. If nested definitions are required, please modify the .ts file of the custom object by yourself.
npm i -g cyfs-objs-generator
Use cyfs-objs-generator -i <proto_file_path> -o <output_dir_path>
, for example:
cyfs-objs-generator -i ./obj_proto.proto -o ./test_out
You can use the cyfs-objs-generator tool to automatically generate all the code for custom object files. Still, there is no harm in mastering the principles. The process of defining object files is relatively mechanical and cumbersome. Basically, just fill in according to the template. Here is a brief explanation:
- Define the immutable part Desc
- Definition object ExampleDescTypeInfo extends cyfs.DescTypeInfo object_type: application object type sub_desc_type: the existence form of the desc system field (disable-none, option-optional, other)
- Define specific ExampleDescContent extends cyfs.ProtobufDescContent
- Define the decoder ExampleDescContentDecoder extends cyfs.ProtobufDescContentDecoder<ExampleDescContent, protos.Example>
- Define ExampleDesc extends cyfs.NamedObjectDesc
- Define ExampleDescDecoder extends cyfs.ProtobufBodyContentDecoder
- Define the variable part Body
- Define ExampleBodyContent extends cyfs.ProtobufBodyContent
- Define ExampleBodyContentDecoder extends cyfs.BodyContentDecoder<ExampleBodyContent, protos.NoneObject>
- Define the ID helper class
- ID auxiliary conversion ExampleId extends cyfs.NamedObjectId<ExampleDescContent, ExampleBodyContent>
- Binary ID Decoder ExampleIdDecoder extends cyfs.NamedObjectIdDecoder<ExampleDescContent, ExampleBodyContent>
- Integrate immutable desc and mutable body
- Define the object constructor ExampleBuilder extends cyfs.NamedObjectBuilder<ExampleDescContent, ExampleBodyContent>
- Define the object itself Example extends cyfs.NamedObject<ExampleDescContent, ExampleBodyContent>
- Define object decoder ExampleDecoder extends cyfs.NamedObjectDecoder<ExampleDescContent, ExampleBodyContent, Example>
So far, the object definition part is fully realized, but have you found that there are as many as 12 classes defined in order to define an object, which is a huge psychological pressure. Let's classify these classes from a macro perspective:
-
According to the processing target, it can be divided into Desc, DescContent, Body, ID, and integration objects.
-
By function, it can be divided into:
- Codec rule description, here tells the codec how to encode and decode specific fields
DescTypeInfo
- Object content definition, define which fields the object contains, how to encode and decode, DescContent/Body/ID have corresponding codec classes
// Desc DescContent DescContentDecoder // Body BodyContent BodyContentDecoder // ID NamedObjectId NamedObjectIdDecoder
- The category name, after defining the above types, can basically be used. The system provides a series of templates to complete the remaining integration work, but the system provides some common names, which are very difficult to read and lengthy, so it is generally recommended to They all define an alias to make the code more readable. These classes can usually use the default interface directly without adding other content; of course, in order to achieve other effects, you can define some auxiliary interfaces for them
// Desc NamedObjectDesc NamedObjectDescDecoder // complete Object NamedObjectBuilder NamedObject NamedObjectDecoder
To sum up, these types mainly provide field definitions and encoding and decoding rules for each part of the object (Desc/DescContent/Body/ID/complete object). The basic method is to inherit the corresponding base class provided by the system, implement the corresponding virtual interface, and add some Fields defined by yourself do not need to completely define their content and encoding and decoding methods from scratch.
Simply put, it is actually a fill-in-the-blank question. For the complete source code, see src/common/objs/message_object.ts For more custom object modeling, you can refer to this document for fill-in-the-blank questions.
Through the study in the previous Introduction to RootState based on ObjectMap
, we know that in CYFS, it includes standard naming container objects
such as ObjectMap, ObjectSet..., and these standard naming container objects
are used to store named objects. Each named object is represented by a unique path of "/key1/key2/key3".
Message objects are also an instance of named objects. These objects should be stored in a name container object
that can represent a message object list
. This name container object
also has its own path, and we name it messages_list
.
That is to say, only the MessageObject stored in a specific ObjectMap<MessageObject>
is the message to be displayed in the message object list.
Therefore, the storage structure of the title corresponding object on RootState is as follows:
|--messages_list
| |--message1.key => Message object
| |--message2.key => Message object
- Full storage path design including other objects, see src/doc/object_model.md
Through the previous study, you have understood the basic principles of RootState, completed the modeling of message objects, and designed the storage path of message objects on RootState. However, you may ask, how to add, delete, modify, and check the objects on RootState? Answer: Use PathOpEnv (Root Path Operation Env) In CYFS, for the state of any RootState, such as /a/b/c, except for leaf nodes, each level is an ObjectMap, and leaf nodes can be containers in any ObjectId or ObjectMap, ObjectSet format, etc. PathOpEnv implements path-based operations on the DEC's RootState.
In order to solve the data consistency problem, PathOpEnv adopts the transaction
mechanism based on path lock
to realize the operation of the object data on the RootState.
A transaction is a sequence of operations performed by a single logical unit of work, either completely or not at all. For a logical unit of work to be a transaction, it must satisfy the ACID (Atomicity, Consistency, Isolation, and Durability) properties.
To perform CRUD operations on objects on RootState, a complete PathOpEnv transaction consists of 3 steps:
- Lock path (lock): lock a group of paths, only the current PathOpEnv can operate. After a path is locked, the parent path and child path of the path will be locked, other PathOpEnv cannot be locked repeatedly, but the sibling path will not be locked, such as locking the path /a/b/c, then /a, /a/ b, /a/d/c/d cannot be locked again, but the paths of /a/c, /a/b/d can be locked.
- Transaction operation (CURD): create an object at the specified path (insert_with_path), update the object at the specified path (set_with_key), read the object from the specified path (get_by_path), and delete the object from the specified path (remove_with_key)
- Transaction commit (commit): After using PathOpEnv to complete the operation, commit will be required to update all modification operations to RootState. If it's just a read operation, then commit doesn't do anything. A PathOpEnv can only commit/abort once, and cannot be used again after that. After successful submission, the corresponding DEC and Global RootState status will be returned.
Now, let's try it out and use RootState to implement the function of adding, deleting, modifying and checking messages.
- See src/service/routers/publish_message.ts for the complete code
To add a message, the core is to need 5 steps:
- Receive new message objects
const { object, object_raw } = req.request.object;
// ...ignore
const decoder = new MessageDecoder();
const dr = decoder.from_raw(object_raw);
// ...ignore
const messageObject: PublishMessageRequestParam = dr.unwrap();
- Use PathOpEnv to lock the new path according to the key of the message object
const createMessagePath = `/messages_list/${messageObject.key}`;
const paths = [createMessagePath];
// ...ignore
const lockR = await pathOpEnv.lock(paths, cyfs.JSBI.BigInt(30000));
- Save the message object to NON
// ...ignore
const nonObj = new cyfs.NONObjectInfo(
messageObject.desc().object_id(),
messageObject.encode_to_buf().unwrap()
);
const putR = await stack.non_service().put_object({
common: {
dec_id: decId,
level: cyfs.NONAPILevel.NOC, // Local operation only, no network
flags: 0, // operation will be initiated
},
object: nonObj,
});
// ...ignore
- Use PathOpEnv to perform the insert_with_path operation to add the new message object to message_list
const objectId = nonObj.object_id;
const rp = await pathOpEnv.insert_with_path(createMessagePath, objectId);
// ...ignore
- Commit the transaction
const ret = await pathOpEnv.commit();
// ...ignore
- See src/service/routers/update_message.ts for the complete code
To modify a message, the core is to need 5 steps:
- Receive a new message object (specify the key value to be modified)
const { object, object_raw } = req.request.object;
// ...ignore
const decoder = new MessageDecoder();
const dr = decoder.from_raw(object_raw);
// ...ignore
const messageObject: UpdateMessageRequestParam = r.unwrap();
- Use PathOpEnv to lock the object path according to the key of the message object
const updateMessagePath = `/messages_list/${messageObject.key}`;
const paths = [updateMessagePath];
// ...ignore
const lockR = await pathOpEnv.lock(paths, cyfs.JSBI.BigInt(30000));
- Save the new message object to NON
const nonObj = new cyfs.NONObjectInfo(
messageObject.desc().object_id(),
messageObject.encode_to_buf().unwrap()
);
// ...ignore
const putR = await stack.non_service().put_object({
common: {
dec_id: decId,
level: cyfs.NONAPILevel.NOC,
flags: 0,
},
object: nonObj,
});
// ...ignore
- Use PathOpEnv to execute set_with_path to perform update operation
const newObjectId = nonObj.object_id;
const rs = await pathOpEnv.set_with_path(updateMessagePath, newObjectId!);
// ...ignore
- Commit the transaction
const ret = await pathOpEnv.commit();
// ...ignore
- See src/service/routers/delete_message.ts for the complete code
To delete a message, the core is to need 4 steps:
- Receive the message object to be deleted (specify the key value)
const { object, object_raw } = req.request.object;
// ...ignore
const decoder = new MessageDecoder();
const dr = decoder.from_raw(object_raw);
// ...ignore
const messageObject: DeleteMessageRequestParam = r.unwrap();
- Use PathOpEnv to lock the object path according to the key of the message object
const deleteMessagePath = `/messages_list/${messageObject.key}`;
const paths = [deleteMessagePath];
// ...ignore
const lockR = await pathOpEnv.lock(paths, cyfs.JSBI.BigInt(30000));
- Use PathOpEnv to perform remove_with_path operation
const rm = await pathOpEnv.remove_with_path(deleteMessagePath);
// ...ignore
- Commit the transaction
const ret = await pathOpEnv.commit();
// ...ignore
- See src/service/routers/retrieve_message.ts for the complete code
To add a message, the core is to need 3 steps:
- Receive the message object to be queried (specify the key value)
const { object, object_raw } = req.request.object;
// ...ignore
const decoder = new Mess();
ageDecoder();
const dr = decoder.from_raw(object_raw);
// ...ignore
const messageObject: RetrieveMessageRequestParam = dr.unwrap();
- Use PathOpEnv to read the message object objectId to be queried
const queryMessagePath = `/messages_list/${messageObject.key}`;
const idR = await pathOpEnv.get_by_path(queryMessagePath);
// ...ignore
const objectId = idR.unwrap();
// ...ignore
- Get the message object on non
const gr = await stack.non_service().get_object({
common: { level: cyfs.NONAPILevel.NOC, flags: 0 },
object_id: objectId,
});
// ...ignore
There is a lot of content in this chapter. We learned the principles of CYFS named object modeling, and practiced the methods of saving and reading states in DEC Service. Learned about the use of the RootState protocol API. But this is not yet a working DEC Service. To get our DEC Service up and running and perform the necessary debugging, read on to the next chapter.