Use macros to define messages rather than eval

[mchitre]

What does this PR do?

• Adds ability to defined strongly typed messages through a new @message macro
• Removes the MessageClass() and AbstractMessageClass() API (BREAKING!!!)
• Redefines performatives as symbols instead of strings (as long as users used the public API of Performative.AGREE, etc, this is non-breaking)
• Removes support for msgID and perf deprecated properties in favor of the official messageID and performative properties (non-breaking provided users avoided the deprecated properties)

What was the problem?

The MessageClass() based API defines new data types dynamically. These data types are defined through eval. During package compilation data types in the module are cached, and so eval into the module later can break it. While we were careful to only dynamically add types to non-cached project modules, this scales poorly and prevents pre-compilation of downstream packages.

The message types defined using MessageClass() were dynamic, i.e., properties were dynamically added to them. This prevented any type inference on the messages and hence required dynamic dispatch on downstream code using the messages, making that code non-performant.

What is the solution?

The solution is to allow concretely typed structs as messages, and to be able to define them statically in a way compatible with pre-compilation.

We now define messages with a macro:

@message "org.arl.fjage.demo.MyMessage" struct MyMessage
  a::Int = 0
  b::Union{Nothing,String} = nothing
  c::Vector{Float64} = Float64[]
end

All message structs are mutable. The @message macro also automatically adds a few fields to the struct:

• performative::Symbol
• messageID::String
• inReplyTo::String
• sender::AgentID
• recipient::AgentID
• sentAt::Int64

Messages can be constructed easily:

julia> msg = MyMessage()
MyMessage:INFORM[a:0 c:[]]

julia> classname(msg)
"org.arl.fjage.demo.MyMessage"

julia> msg.messageID
"835f5bc5-7d76-4285-aee9-656a910e22ac"

julia> msg = MyMessage(a=21, b="hello")
MyMessage:INFORM[a:21 b:"hello" c:[]]

and properties can be accessed as per normal:

julia> @show msg.a
msg.a = 21
21

julia> push!(msg.c, 27.0)
1-element Vector{Float64}:
 27.0

Access to non-existent properties throw errors unless ignore_missingfields=true when setting the property. This is used in inflation to ensure forward-compatibility with remote messages where new fields may be added without breaking old code.

We can also define abstract message types that can be used as base types for other messages:

abstract type MyAbstractMessage <: Message end

@message "org.arl.fjage.demo.MyConcreteMessage" struct MyConcreteMessage <: MyAbstractMessage
  a::Int
end

julia> MyConcreteMessage(a=1)
MyConcreteMessage:INFORM[a:1]

julia> MyConcreteMessage(a=1) isa MyAbstractMessage
true

Java allows concrete messages to be extended. To enable the same functionality in Julia, we allow concrete message to be defined with the same name as an abstract base class:

abstract type SomeMessage <: Message end

@message "org.arl.fjage.demo.SomeMessage" struct SomeMessage <: SomeMessage
  a::Int
end

@message "org.arl.fjage.demo.SomeExtMessage" struct SomeExtMessage <: SomeMessage
  a::Int
  b::Int
end

julia> SomeMessage(a=1) isa SomeMessage
true

julia> SomeExtMessage(a=1, b=2) isa SomeMessage
true

Performatives are guessed automatically based on message classname. By default, the performative is Performative.INFORM. If a message classname ends with a Req, the default performative changes to Performative.REQUEST. Performatives may be overridden at declaration or at construction (and are mutable):

@message "org.arl.fjage.demo.SomeReq" struct SomeReq end
@message "org.arl.fjage.demo.SomeRsp" Performative.AGREE struct SomeRsp end

julia> SomeReq().performative
:REQUEST

julia> SomeRsp().performative
:AGREE

julia> SomeRsp(performative=Performative.INFORM).performative
:INFORM

Finally, it is always possible that we receive a message from a remote container with a message class that we do not have a definition for. These are handled by generating a parametric GenericMessage that can contain any properties:

julia> msg = GenericMessage("org.arl.fjage.demo.DynamicMessage")
DynamicMessage:INFORM

julia> msg.a = 1
1

julia> msg.b = "xyz"
"xyz"

julia> msg
DynamicMessage:INFORM[a:1 b:"xyz"]

julia> classname(msg)
"org.arl.fjage.demo.DynamicMessage"

julia> msg isa GenericMessage
true

If no classname is specified, GenericMessage defaults to the Java GenericMessage class:

julia> GenericMessage() |> classname
"org.arl.fjage.GenericMessage"

Reminder

This isn't new, but easy to forget. Any package that defines strongly typed messages should call registermessages() during package initialization:

function __init__()
  registermessages()
end

This ensures that the messages are registered with Fjage.jl, and can be constructed on demand when the classname is encountered in JSON coming in from a remote container. Without this registration, a GenericMessage with that classname will be constructed instread of the correct message type.

[ettersi]

Looks good to me! Always good to move away from eval 😊

I have just two minor nitpicks / questions.

access to non-existent properties does not throw errors:
...
This is necessary to ensure forward-compatibility with remote messages where new fields may be added without breaking old code.

I'm not sure I understand this correctly, but I don't think this is true? We could have message.non_existing_property throw an error but then use some special setproperty_nothrow!(message, property, value) in _inflate() to allow unknown fields in incoming messages. I'm not saying message.non_existing_property should throw (that's a discussion to be had elsewhere), but I just wanted to make sure we're not dismissing this option for the wrong reasons.

[mchitre]

access to non-existent properties does not throw errors:
...
This is necessary to ensure forward-compatibility with remote messages where new fields may be added without breaking old code.

I'm not sure I understand this correctly, but I don't think this is true? We could have message.non_existing_property throw an error but then use some special setproperty_nothrow!(message, property, value) in _inflate() to allow unknown fields in incoming messages. I'm not saying message.non_existing_property should throw (that's a discussion to be had elsewhere), but I just wanted to make sure we're not dismissing this option for the wrong reasons.

Sure, yes, we could limit the non-throwing behavior to just inflation, and still throw for use in direct use of the property. I am happy with that option.

[mchitre]

@ettersi updated code so that non-existent property access throws errors unless a keyword argument ignore_missingfields=true in setproperty!(). Updated writeup above to reflect that.

[ettersi]

I think we can avoid a lot of the scoping complications by not blanket-escaping the entire macro return value. I can have a look at this tomorrow.

[mchitre]

I think we can avoid a lot of the scoping complications by not blanket-escaping the entire macro return value. I can have a look at this tomorrow.

Agree. And I tried doing it several times and failed. The problem is that the macro call for @kwdef fails if the whole quote is not escaped. Welcome to try and see if you find a solution.

The current version assumes that Fjage is imported in the target environment. As long as this is true, it works well.

[ettersi]

The problem is that the macro call for @kwdef fails if the whole quote is not escaped.

This test in Base.@kwdef indeed does not account for escaped type names T. See here for patch of @kwdef which allows proper escaping in @message. Let's see if I can get this patch into base Julia 🥁🥁🥁

[mchitre]

The right place to fix this is base Julia. Couple of things to think through to decide if we want a temporary fix here ... see my comments on #46.

[mchitre]

All ready to merge now, having addressed all of @ettersi's comments. Pending @notthetup approval.

[ettersi]

Sorry 🙈

[ettersi]

xref JuliaLang/julia#53230