Support MeTTa expressions instead of code strings in compile! function

[patham9]

As Vitaly and Alexey pointed out, it would be good if !(compile ...) would accept a MeTTa expression instead of a code string. That way it becomes possible to derive program code at runtime and then to pass it to the compiler.

Example, current:

!(compile! "
(= (facF $n)
   (If (== $n 0)
       1
       (* $n (facF (- $n 1)))))
")

Example desired:

!(compile! 
(= (facF $n)
   (If (== $n 0)
       1
       (* $n (facF (- $n 1)))))
)

Hereby, the optional possibility to compile a MeTTa file can be preserved:
!(compile! test.metta)

[Necr0x0Der]

All the mentioned options can be useful:

• Compiling a code string
• Compiling a MeTTa expression
• Compiling a file (either via loading it to the code string or parsing it into MeTTa expressions)

The use cases are somewhat different, and we don't have enough practice in using metta-morph. Thus, I'd not replace one option with another, but rather augment them with each other until we see that something is really redundant.
But I'd like to mention one more option: compiling spaces. We can fill a space at runtime either with generated expressions or with a file content, e.g.

!(import! &test test.metta)
!(compile-space! &test)

I'd like to note that going beyond individual expressions can be tricky, because MeTTa can have definitions like

(= (Add $x Z) $x)
(= (Add $x (S $y)) (Add (S $x) $y))

which is a pretty ordinary tail-recursive function, but it is defined in a not Scheme-ish way. Working with such cases should be covered after #15 (and after introducing skipped unit tests in #16 ).
Thus, I propose to consider this issue in a simplistic way - just make a version of compile!, which translated a given expression[s] (even if they are provided in a space) to a code string. Let's leave all additional features for later.

[patham9]

I agree, this should suffice to handle the compilation of a MeTTa expression in addition.
Regarding:

(= (Add $x Z) $x)
(= (Add $x (S $y)) (Add (S $x) $y))

No worries that's actually no issue for Mettamorph already! :)

[patham9]

Now I see, even though it is fine within Mettamorph itself, such a function can actually cause issues when called from MeTTa due to limitation 4:
"For non-deterministic execution, backtracking cannot continue across the MeTTa-MeTTamorph boundary, a collapse before returning is necessary in such cases."
So there needs to be a function in the compiled code which can call the (also compiled) non-deterministic one but delivers a deterministic result, this deterministic function can then be called from MeTTa.

[Necr0x0Der]

Yes, basically, this function is not non-deterministic, but converting it to an ordinary Scheme function is not straightforward and requires some code and type analysis.
Also I wonder how metta-morph handles patterns in function definitions. I didn't check this.

[patham9]

Non-determinism and patterns in = based function definitions are already fully handled. = is converted to a Scheme function by a macro which maintains a hashmap with function name as key and as value a list of functions (match-lambda*'s which do respect argument patterns!) of same name, whereby for each (= (f Xargs) Body) expansion a new entry is added in the list of functions for f, and the actual Scheme function body for f is the ambivalence operator being applied over the entire list for f.
A testcase which tests for this: https://github.com/trueagi-io/metta-morph/blob/main/tests/multifunction.metta

[patham9]

Also, I have added your testcase (which already passes): https://github.com/trueagi-io/metta-morph/blob/main/tests/peano.metta

[TeamSPoon]

The underlying issue is whether or not Rust MeTTa can return the source code of a space? (Return the source code without trying to reduce or evaluate it on accident).. if this is possible, then it could be much easier to compile it

[patham9]

@Necr0x0Der: This example might be of interest to you, as your Add example is now fully supported even when compiled and called from MeTTa: https://github.com/trueagi-io/metta-morph/blob/f21765f211a72f2a0218be9bf2100714325d77c4/extend/example5.metta
However there is an additional example underneath which shows limitation 4 does still exist for truly nondeterministic functions. Maybe the collapse idea can actually be applied automatically, I will try.

[patham9]

This idea worked out, calling non-deterministic code is fine now and the example has been cleaned up accordingly: https://github.com/trueagi-io/metta-morph/blob/main/extend/example5.metta
(meaning above-mentioned limitation 4 has been resolved by automatically applying the (Scheme-)collapse within the Scheme boundary and applying a (Metta-)superpose at its return)

[Necr0x0Der]

@patham9 , a side remark: please, don't commit directly to the main branch. These changes are not seen by others unless you refer to them explicitly. Please, create a branch for the change, introduce it there, push this branch to the origin, and raise a PR. Then, changes will be observable (and everyone subscribed will get a notification) and trackable.

[patham9]

Makes sense, I will commit into PR's from now-on.
The last commit I did resolves this issue #18 and allows expressions to be compiled: 8948b50
However it has known/expected limitations (like with superpose), but I guess we can close this issue as these limitations are not caused by Mettamorph. Though with superpose it should be fine if we can resolve the new related issue #20