RFC: Object based design for Hilla data models

[platosha]

After some extensive prototyping, I would like to propose the new design for Hilla models that replaces TypeScript class based definitions with objects defined using a builder API.

In this discussion, let me explain the purpose and some key design choices. Please give your feedback, and feel free to ask questions.

At the same time, here is my prototype branch for those who want to play with this idea themselves: https://github.com/vaadin/hilla/tree/proto/model-object-design/packages/ts/model. The test file has some usage examples.

Motivation

Hilla models were originally introduced to support form binding use cases. But recently we started to add more high-level data-oriented frontend developer productivity features to Hilla, such as AutoGrid and the upcoming AutoForm/AutoCrud, where we also take the data structure and metadata from the models. Some limitations of the current class-based models design became apparent:

• Accessing object properties is hard. It is expected that the described properties are easily available: for example, NamedModel.name references the name property of a NamedModel, and Object.keys(SomeModel) could iterate the keys. The model classes do not meet this expectation, and require an extra step for these use cases: either instantiation or a prototype access of some sort.
• The aforementioned model instantiation is hard. The model class constructor historically requires a value container in a form of either a form binder or the “parent” container model.
• Creating models manually is hard. Ideally an object model should use getters with a lazy initialization, and the generated models do use getters implemented with a helper.

Usage

Type description

Models are primarily used as runtime values that describe the underlying type. To illustrate, let us consider the following generated entity structure:

interface Named {
  name: string;
}

The name property of this type could be referenced by indexing:

type NamedNameType = Named["name"];

However, TypeScript types do not work as values. You cannot, for example, pass them to some React component props:

function MyView() {
  return <AutoGrid
    model={Named}
    //     ^: Error: 'Named' only refers to a type, but is being used as a value here.ts(2693)
    visibleColumns={[Named["name"]]}
    //               ^: Error: same as above
  />;
}

For this we need some value that describes the type and has equivalent structure. This is what the models are.

Let us assume that Hilla provides an additional NamedModel object for our use case, which simplified structure is:

const NamedModel = {
  name: StringModel,
};

Now you can use both NamedModel and NamedModel.name values as descriptions of their respective types:

function MyView() {
  return <AutoGrid
    model={NamedModel}
    visibleColumns={[NamedModel.name]}
  />;
}

The key difference here from the existing class-based Hilla models is that you can directly reference a property (NamedModel.name), which is not supported with the current class-based design.

Creating models

Hilla generates both interfaces and their models for Java entities in endpoints, but sometimes you may not have a Java type. One common example is coming up with a test model for testing a frontend component integrated with Hilla models.

As a base, Hilla provides builtin model values for primitives (BooleanModel, NumberModel, StringModel), and empty objects (ObjectModel).

You can create an object model using the Hilla-provided m.from builder API.

Models of optional value (T | undefined) and array models (T[]) can be created by calling m.optional(model) and m.array(model). Models of TypeScript enums can be created with m.enum(enum);.

Object models with primitive properties

interface OrderRow {
  product: string;
  qty: number;
}

const OrderRowModel = m
  .from(
    ObjectModel, // The base 
    toObject<OrderRow> // The underlying type
  ) 
  .name('OrderRow') // A string name for debug output
  .property('product', () => StringModel)
  .property('number', () => NumberModel)
  .build();

Inheritance

interface Address extends Named {
  street: string;
}

const AddressModel = m
  .from(NamedModel, toObject<Address>)
  .name('Address')
  .property('street', () => StringModel)
  .build();

Composition

interface Customer extends Named {
  address: Address;
}

const CustomerModel = m
  .from(NamedModel, toObject<Customer>)
  .name('Customer')
  .property('address', () => AddressModel)
  .build();

Optionals, arrays, enums

enum OrderStatus {
  New = 'New',
  Shipped = 'Shipped',
}

interface Order {
  notes?: string;
  rows: OrderRow[];
  status: OrderStatus;
}

const OrderModel = m
  .from(ObjectModel, toObject<Order>)
  .name('Order')
  .property('notes', () => m.optional(StringModel))
  .property('rows', () => m.array(OrderRowModel))
  .property('status', () => m.enum(OrderStatus))
  .build();

Self-references in objects

interface Employee extends Named {
  supervisor?: Employee;
}

const EmployeeModel = m
  .from(NamedModel, toObject<Employee>)
  .name('Employee')
  .property('supervisor', function() {
    // `this` refers to the resulting model object (EmployeeModel)
    return m.optional(this);
  })
  .build();

Hierarchy location

When you define a property using some model as a value, the builder internally “attaches” it. The resulting property will hold a copy of the given model value with altered metadata, so that the information about the container and the key.

In the end, every model value could tell about its location in the hierarchy: it is either detached (by default) or attached some object property or array item.

In above examples, AddressModel and CustomerModel.address describe the same type (Address object), but their values are different: the latter is attached by nesting in the "address" property of CustomerModel. The same is also true for the deep nested models (AddressModel.street and CustomerModel.address.street).

Reference

Now let us take a look at the model object internals.

Base types and values

First, there is a default container for all the detached models:

/**
 * The model hierarchy root type
 */
export interface ModelOwner {
  model?: IModel;
}

/**
 * The default container (hierarchy root) for detached models
 */
export const detachedModelOwner: ModelOwner = {
  model: undefined,
};

All the models implement the base interface IModel<T>:

/**
 * The base interface for Hilla data models
 */
export interface IModel<T = unknown> {
  /**
   * String name for debug output
   */
  readonly [_name]: string;

  /**
   * Contaniner model or hierarchy root
   */
  readonly [_owner]: IModel | ModelOwner;
  
  /**
   * The key in the container (property name for object, or index number for arrays).
   */
  readonly [_key]: keyof any;

  /**
   * Value getter and type marker
   */
  readonly [_value]: T;

  /**
   * Optional marker
   */
  readonly [_optional]: boolean;

  /**
   * Other metadata (validation rules, JVM type and annotations, etc) in JSON-like structure
   */
  readonly [_meta]: ModelMetadata;
}

There is AbstractModel value that describes unknown type following the above interface.

Readonly pattern

The model properties are strictly read-only to reject mutation attempts, that are likely to cause errors down the road. Users are expected to create copies or wrap existing models using the Hilla model APIs.

Internal and public properties

To avoid naming conflicts with the user's entity properties, the internal properties are defined using symbols, as illustrated in IModel<T>.

String properties only occur in object models. They are used to resemble the structure of the user's entity.

The internal properties are non-enumerable, whereas object properties can be enumerated using the regular JS workflow: for...in loop or Object.keys() / Object.entries() in combination with Object.getPrototypeOf() for hierarchy traversal.

// If you just need to list properties in any order
for (const key in OrderModel) {
  const propertyName = key as string & keyof typeof OrderModel;
  const propertyValueModel: IModel = OrderModel[propertyName];
  console.log(propertyName, '\n ', propertyValueModel);
}

// For more complex cases, for example, if we want inherited properties first
function getObjectProperties(model: typeof ObjectModel): string[] {
  const properties = new Set();
  const collectPropertyName = (name: string) => properties.add(name);
  
  const proto = Object.getPrototypeOf(model);
  if (isModel(proto, ObjectModel)) {
    const inheritedProperties = getObjectProperties(proto);
    inheritedProperties.forEach(collectPropertyName);
  }
  
  Object.keys(model as unknown as Record<string, IModel>).forEach(collectPropertyName);
  return Array.from(properties);
}

Implementation notes

The m model APIs use Object.create internally to retain the hierarchy in the prototype chain.

Here is a rough illustration of the low-level implementation behind the APIs:

const AddressModel = Object.create(
  NamedModel,
  {
    // Internal properties are non-enumerable and use symbols. In this example,
    // the new model describes a different type from the base, so it replaces
    //  the internal `[_value]` property as follows:
    [_value]: {
      enumerable: false,
      get() {
        return { name: '' } as Named;
      }
    },
    
    // String properties that follow the structure of the underlying type (Address).
    // Their getters return a model, which is attached to `this` current model:
    street: {
      enumerable: true,
      get() {
        // Input: the `.property('street', () => StringModel)` builder method call arguments
        const propertyName = 'street'; 
        const getter = () => StringModel;

        // The original detached model is obtained from the getter, to which
        // we pass `this` to enable self-referencing.
        const originalValueModel = getter.call(this);

        // The value of `this` is determined in runtime from the user invoking
        // `.street` on some `AddressModel` somewhere, either the original
        // detached model being created, or some altered copy of it. 
        const owner: IModel<Address> = this;

        // To attach a model as a property, we create a modified copy of
        // the original value model with changes for the internal properties
        // describing the hierarchy location. Additionally, we memoize
        // the attached copies based on the value of `this` and
        // the property name. I omit the memoize cache implementation
        // here, assume that `useMemo` from React is available.
        return useMemo((() => Object.create(originalValueModel, {
          [_owner]: {
            enumerable: false,
            get: () => {
              return this;
            },
          },
          [_key]: {
            enumerable: false,
            get: () => {
              return propertyName;
            },
          },
        })), [this, propertyName]);
      },
    },
  }
) as TypeModel<Named>;

[Legioth]

A couple of thoughts on the shape of the API:

• Using m as the starting point for building models gives the impression that you're starting from an existing instance (which I guess is kind of true but still a bit confusing). What about having a convention where it's named e.g. Model, BaseModel, or ModelBuilder instead?
  • Alternatively, could we make inheritance clearer by starting from the base type, e.g. NamedModel.subType(toObject<Address>) instead of m.from(NamedModel, toObject<Address>)?
• See the point of doing model initialization lazily for complex cases but I wonder if it would make sense to have a non-lazy option for simple cases? someModel.property('notes', StringModel)
• Does the toObject<OrderRow> syntax have any other purpose than to tie down the type parameter? Could that instead be satisfied along the lines of m.from<OrderRow>(ObjectModel)?

With all my suggestions taken together, the first example would be simplified to this format:

const OrderRowModel = ObjectModel
  .subType<OrderRow>()
  .name('OrderRow') // A string name for debug output
  .property('product', StringModel)
  .property('number', NumberModel)
  .build();

[platosha]

Thanks a lot for the feedback!

Naming suggestions are welcome. m is exactly the model builder, not an instance, and it provides a DSL for building several kinds of models from built-ins. Not only objects, but also m.optional and m.array, m.enum, and maybe more. ModelBuilder as a good sounding name, but ergonomics of repeating it again and again in a namespace fashion is not that great. Let's keep in mind that cases of several layers of wrapping like ModelBuilder.optional(ModelBuilder.array(ModelBuilder.optional(StringModel))) are common in Hilla.

I would still strongly prefer some form of external ModelBuilder over attaching API methods to model instances, as with NamedModel.subType(). The latter shares the same namespace with the entity properties defined by the user, so there is always a risk of naming conflicts there.

The non-lazy definition option is reasonable. The builder can still keep doing all the work behind the scenes with creating the getter that attaches models. To my knowledge, the only use case that strictly must be lazy is the self-reference of the model that is being created.

Regarding toObject, its internal double-duty is to provide the empty value of the given type. In case of the object type, the empty value could be always drawn after the structure of the model, therefore the implementation of toObject empty value getter is shared between all object types. This means, yes, we could theoretically replace the toObject argument value with a generic type parameter. However, in practice, it seems to be only working well if we attach the builder API to the model itself, as in your example. How could then the API for an external object model builder that takes an optional value argument (parent object model) and a required type argument for the final model's value object type look like?

[Lodin]

• I would suggest to get rid of m at all. Let's just use a simple import from the library or a specific module in the library, so we don't need any prefix at all.

  import {from, optional, array, enum} from '@hilla/form/Model.js';
  
  const OrderModel = from(ObjectModel, toObject<Order>)
    .name('Order')
    .property('notes', () => optional(StringModel))
    .property('rows', () => array(OrderRowModel))
    .property('status', () => enum(OrderStatus))
    .build();

• I would also suggest to rename from to extend, so it will be similar to well-known JS approaches existed before ES2015.

  const OrderModel = extend(ObjectModel, toObject<Order>);

[platosha]

Some notes from discussing F2F with @Lodin:

Explicit self argument for self-reference is nicer than this, because it allows arrow functions:

.property('supervisor', (self) => m.optional(self))

One nice discovery is that the user can omit the call entirely:

.property('supervisor', m.optional)

When couldn't get the m.from<OrderRow>(ObjectModel) form to work according to our expectations in TypeScript. The challenge here that we use both types, typeof ObjectModel and OrderRow, as generic arguments down the road, so we need both two as generic parameters here. Then, we cannot constraint the explicit OrderRow argument using the inferred typeof ObjectModel type from the value.

One idea about this we had is to receive two generic type arguments in separate builder steps. @Lodin suggested to combine defining the new name and the target object type into a single method replacing .name:

const AddressModel = m.extend(NamedModel)
  .define<Address>('Address')
  .property('street', StringModel)
  ...
  .build();

[platosha]

I have prototyped separate builder steps. The basic shape is m.extends(SuperEntityModel).object<Entity>('Entity'), and for objects extending Object the .extends step can be omitted, like so:

const NamedModel = m
  .object<Named>('Named')
  .property('name', StringModel)
  .build();

const AddressModel = m
  .extend(NamedModel)
  .object<Address>('Address')
  .property('street', StringModel)
  .property('premise', StringModel)
  .property('city', StringModel)
  .property('postalCode', StringModel)
  .property('country', StringModel)
  .build();

This works with the prototype use cases and seems to be a reasonable compromise. Any objections?

[platosha]

BTW this reminds me that it would be nice to receive names for enum models as well. So, adding the name argument to the mix, the enum model definition would look like so:

const AccountTypeModel = m.enum(AccountType, 'AccountType');

... which is similar to .object<Address>('Address').

[tarekoraby]

For dummies like me, can someone please explain how the above proposals would change the following example?

<AutoGrid
  service={ProductService}
  model={ProductModel}
  visibleColumns={['category', 'name', 'supplier.supplierName', 'price']}
/>

Also, to clarify, would the above proposal(s) work equally well for all components? For example, can one use the proposed approaches to declare a ComboBox's item-label-path and item-value-path properties?

<ComboBox item-label-path="name" item-value-path="id" items={items} />

[tarekoraby]

Okay, upon a second look, I understand this one

function MyView() {
  return <AutoGrid
    model={NamedModel}
    visibleColumns={[NamedModel.name]}
  />;
}

Can the same approach be used for other components? For example, can one do the following?

<ComboBox item-label-path=NamedModel.name item-value-path=NamedModel.id items={items} />

[platosha]

It still requires adding support from the component's side. For <AutoGrid>, I assume that we add support in #1496. We could consider adding <ComboBox> support also, but it's not planned yet.

[jojule]

Dumb questions:

• Why users would benefit from this: Why should they want to iterate properties of a Model?
• In what situations users need to instantiate models - wont Hilla do it for them?
• When users would like to define new models? Can we support them to do so with the current class based models?

[Legioth]

Why users would benefit from this: Why should they want to iterate properties of a Model?

Iterating properties is more of a need for generic components like a form binder or a grid column configurator. It might not be directly used from typical application code but many enterprisey applications tend to gradually morph in that direction as they standardize the patterns used for implementing e.g. generic CRUD functionality.

In what situations users need to instantiate models - wont Hilla do it for them?

Hilla does that for typical usage through e.g. the form binder but that functionality is not easily reusable for e.g. configuring grid columns.

When users would like to define new models? Can we support them to do so with the current class based models?

There have been requests to be able to use Hilla's form binder also with types that don't have a direct Java counterpart but are instead further processed in the client before being passed to the server. Another use case is to just make some customization to an existing model to include something that isn't possible to express in the Java type system.