Allow passing arguments to superclass constructor in static proxy

[mhsmith]

As it says in the documentation, "Supplying arguments to the superclass constructor is not currently possible" in a static_proxy. This prevents them inheriting directly from Java classes which don't have a zero-argument constructor, such as View and IntentService.

We may fix this in the future, but it's a difficult technical problem, and it seems that relatively few users are interested in the static_proxy feature anyway.

If this limitation affects anyone else, please click the thumbs up button below, and subscribe to this issue.

[visopsys]

I need this feature. I am subclassing IntentService and need to call super constructor of IntentService

@mhsmith

[mhsmith]

For IntentService your constructor parameter is probably always the same, in which case you can work around this by inheriting indirectly.

For example, create the following Java class:

public abstract class JMyService extends IntentService {
    public JMyService() {
        super("MyService");
    }
}

And then inherit that in Python:

class MyService(static_proxy(JMyService)):
    @Override(jvoid, [Intent])
    def onHandleIntent(self, intent):
        ...

[mhsmith]

Very few people are using the static proxy feature, so we won't be doing this in the foreseeable future.

[mhsmith]

Old notes from 2018:

---

We cannot simply generate forwarding constructors for each public constructor in the base class. IntentService is an example of where a 1-argument superclass constructor must be called by a zero-argument subclass constructor. (Though in this case you can probably work around by inheriting via an intermediate Java class: see example here.)

Java syntax is very restrictive:

• super or this, if present, must be very first statement in constructor. If neither is provided, compiler auto-generates a zero-argument super call, causing an error if no such constructor is accessible.
• "A [super or this] statement in a constructor body may not refer to any instance variables or instance methods or inner classes declared in this class or any superclass, or use this or super in any expression."

The JVM spec is a bit more flexible:

• Multiple super and this calls may appear in the constructor. JVM uses control flow analysis to ensure a call takes place exactly once in all possible paths.
• The call does not need to be the first code in the constructor, but it must happen before any methods or fields are accessed on the superclass.
• Violating any of these rules is a bytecode verification error.

Jython takes advantage of this to allow standard Python syntax. It ensures the bytecode passes the verifier by generating a zero-argument "super()" call at the end of all paths which don't otherwise contain one.

However, Jython works by translating the Python source into bytecode, which will never be our approach. We could bypass the bytecode verifier if we could generate a native constructor, and then call the base class constructor from Cython code using CallNonVirtual. However, native constructors aren't allowed in Java source code, so this could only be done using bytecode generation. I don't see anything in the JVM or DEX specs prohibiting native constructors, but I can't find any reference to anyone attempting them either, so I have no confidence that they'll work.

For standard Java bytecode, commonly used generator tools include cglib and javassist. This question also mentions Byte Buddy as having DEX support, and there's a DEX-only tool called dexmaker.

Dynamic loading of .dex files does appear to work:

• Does Android ART support runtime dynamic class loading just like Dalvik?
• Load DEX file dynamically on Android 5.0

But to make this work for classes loaded outside of our control (e.g. Activity or View subclasses), we'd have to modify or replace the ClassLoader used by that code. This is platform and application-specific, and can only be done by modifying internal data structures. For example, to change the ClassLoader used to load Activities, you'd have do something like this, which uses reflection to do the equivalent of:

context.mMainThread.mPackages.get(context.getPackageName()).mClassLoader = <new loader>

But using reflection to access private members of Android platform classes is now mostly blocked.

---

To support base classes like IntentService, I was initially thinking of something like this:

@constructor(super="arg1-expression, arg2-expression")
def __init__(self):
    ...

where the quoted string would be copied directly into the generated Java constructor with no further interpretation. We could also support a "this" keyword which would work the same way.

VOC used a similar approach: https://github.com/beeware/voc/blob/90fc07386bae942b7eb499ba98792fb5c75d8121/voc/python/ast.py#L1008-L1025

---

However, we really want to use the standard Python syntax if at all possible. The difficulty with this comes with Java-initiated construction. Even with bytecode generation, JVM verification means that the constructor must contain a super() call; we can't do that via JNI unless we can make a native constructor (notes above). And if using source code generation, the super constructor signature must be known at compile time: we can't use run-time overload resolution here.

The best approach I've thought of so far is:

• Extend the @constructor decorator to specify super constructor signature for each "this-class" constructor signature.
  • e.g. @constructor([jint, Object], super=[String], throws=[Whatever, Etc])
  • Default is still zero arguments for both signatures, and either signature, or the whole decorator, can be omitted.
• Generated Java constructor for each signature calls super() with correct number of arguments:
  • Expression for first argument initiates the Python constructor, which will be terminated with an exception when it calls superclass __init__. Exception object contains the parameters passed to superclass __init__, which must be compatible with the superclass signature of the "this-class" signature actually called.
  • Expression for all arguments takes values for super() parameters out of this exception and converts to required type.
  • What if there are no arguments? Then it's impossible in Java source code (but not bytecode: see below) to run any other code before it. Unless we generated two Java classes, one inheriting from the other, and run the code in the superclass constructor?
  • In Java bytecode we have more flexibility, and we can run any code before the super call as long as it doesn't access any instance variables or methods. We can even make super calls within conditional blocks, as long as every possible path through the constructor makes exactly one super call. This could eliminate any need to specify the super signature in the decorator, as it can now be determined at runtime.
• Python constructor is then called again. This time, calling superclass __init__ does not throw an exception.

The exception is necessary because If the Java constructor waited for the Python constructor to return normally before doing the Java super() call, the remainder of the Python constructor wouldn't have access to a valid Java object, so it couldn't do much useful. (If the super() call is zero-argument, which is the default, then the initial __init__ call doesn't need to happen.)

How do we identify this object if we can't access any local variables including this? Maybe something like:

public Foo(String arg0, String arg1) {
    this(new InternalClass(), arg0, arg1);
}
private Foo(InternalClass ic, String arg0, String arg1) {
    super(<expression-with-ic>, <expression-with-ic>)
}

The double call to __init__ will only cause a problem if __init__ has side-effects which happen before it calls super().__init__. This places some constraints on the location of the super call, but still milder ones than those imposed by Java.

Other possible ways of achieving this without the double call (haven't thought these through):

• Using "yield" to suspend the Python constructor execution
• http://pyjs.org/Translator.html: "The yield support is implemented as a state machine, rewriting the code so that it is re-entrant, and can resume on each yield."
• Could maybe use the @constructor decorator to transform the function in some way.

For Python-initiated construction the Python constructor would only be called once, and the Java object would be created when super().__init__ is called, by means of direct non-virtual constructor calls via JNI, and/or generated helper constructors.

• In fact, we already have forwarding constructors which take a leading argument of a private class to avoid any clashes. So explicitly specifying the super signature is unnecessary, because it can be resolved using the existing overload algorithm.

The result is that we get almost normal Python syntax and semantics from the Python super().__init__ call, whether construction is initiated by Python or Java.

[mhsmith]

Toga only requires Python-intiated construction, which is the easier case to solve, so we should look at that first. However, the current workaround of putting a Canvas helper class in the Briefcase Gradle template works fine, so this isn't urgent.

[t-arn]

I also need this feature for extending ArrayAdapter (see beeware/toga#1393) or EditText (for adding line numbering).

I know that we could add these extended classes to the briefcase Android template, but we will end up with lots of those classes and the dependencies between the 2 projects will increase the complexity unnecessarily

[t-arn]

Toga only requires Python-intiated construction, which is the easier case to solve, so we should look at that first.

@mhsmith Sounds good to me! I currently only need a solution for Toga.

However, the current workaround of putting a Canvas helper class in the Briefcase Gradle template works fine, so this isn't urgent.

Well, Russel just closed my PR beeware/briefcase-android-gradle-template#61 which used this workaround. So, the workaround is not really a solution, not even an intermediate one :-(

I would really appreciate if you could make this subclassing work for Toga. I would love to offer my help, but this stuff is beyond my knowledge.

[mhsmith]

Like Russell said, our current priority is to complete and test the existing Toga APIs, so unless this issue turns out to block something more critical, I probably won't have time to look at it in the next few months. Sorry about that.

[mhsmith]

In Toga, this would currently be useful for:

• Android WebView lacks several features due to inability to subclass beeware/toga#1020
• The Canvas, which currently depends on a helper class in the template
• Probably any styling of the Selector Worked around in Implemented Selection.set_font() for Android beeware/toga#2130
• Reworking the ScrollContainer to use a single native container which allows diagonal scrolling. Though it turns out macOS doesn't natively support that either.