Android calls python through chaquo to get the ui window built by toga, strange error “Failed to create image decoder with message 'unimplemented'”

[andy4lee]

Describe the bug

I need a permanent floating window, so I need to use a service to implement it, not an activity (toga can be called normally，so to prove that the software environment is fine)

FloatingWindowService.java:

package org.beeware.android;

import static android.content.ContentValues.TAG;

import android.app.Notification;
import android.app.NotificationChannel;
import android.app.NotificationManager;
import android.app.PendingIntent;
import android.app.Service;
import android.content.Context;
import android.content.Intent;
import android.content.SharedPreferences;
import android.content.res.Configuration;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.PixelFormat;
import android.hardware.display.DisplayManager;
import android.hardware.display.VirtualDisplay;
import android.media.Image;
import android.media.ImageReader;
import android.media.projection.MediaProjection;
import android.media.projection.MediaProjectionManager;
import android.os.Build;
import android.os.Handler;
import android.os.IBinder;
import android.util.DisplayMetrics;
import android.util.Log;
import android.view.Gravity;
import android.view.LayoutInflater;
import android.view.MotionEvent;
import android.view.View;
import android.view.ViewGroup;
import android.view.MenuItem;
import android.view.Menu;

import android.view.WindowManager;
import android.widget.Button;
import android.widget.TextView;

import androidx.annotation.Nullable;

import java.util.Arrays;
import java.util.Objects;

import com.chaquo.python.Kwarg;
import com.chaquo.python.PyException;
import com.chaquo.python.Python;
import com.chaquo.python.PyObject;
import com.chaquo.python.android.AndroidPlatform;
import com.example.uiapp.R;

public class FloatingWindowService extends Service {
	
	public static final String TAG = "FloatingWindowService";
    private WindowManager windowManager;
    private ViewGroup floatView;
    private int LAYOUT_TYPE;
    private WindowManager.LayoutParams floatWindowLayoutParam;
	private static PyObject pythonApp;
		
	@SuppressWarnings("unused")
    public static void setPythonApp(IPythonApp app) {
        pythonApp = PyObject.fromJava(app);
    }

    public void onCreate() {
        super.onCreate();
        Log.d(TAG, "FloatingWindowService onCreate()");
        updateFloatingWindowContent(null);
		//createFloatingWindow();
		//startTogaActivity();
		userCode("onCreate");
    }
	private void updateFloatingWindowContent(TextView textView) {
        // Make sure Python is initialized
        if (!Python.isStarted()) {
            AndroidPlatform platform = new AndroidPlatform(this);
            platform.redirectStdioToLogcat();
            Python.start(platform);
        }
        Python py = Python.getInstance();
        Log.d(TAG, "Running Python code to update floating window content");

        // Call Python function to get content
        py.getModule("runpy").callAttr(
                "run_module",
                "uiapp.__main__",
                new Kwarg("run_name", "__main__"),
                new Kwarg("alter_sys", true)
        );
        // Update the view (run on UI thread)
        //textView.post(() -> textView.setText(content));
    }
	
	private void startTogaActivity() {
		Intent intent = new Intent(this, TogaActivity.class);
		intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
		startActivity(intent);
    }
	private int mScreenWidth;
    private int mScreenHeight;

    private void createFloatingWindow() {
		LayoutInflater inflater = (LayoutInflater) getBaseContext().getSystemService(LAYOUT_INFLATER_SERVICE);
        floatView = (ViewGroup) inflater.inflate(R.layout.activity_main, null);
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            LAYOUT_TYPE = WindowManager.LayoutParams.TYPE_APPLICATION_OVERLAY;
        } else {
            LAYOUT_TYPE = WindowManager.LayoutParams.TYPE_TOAST;
        }
		
        WindowManager windowManager = (WindowManager) getSystemService(WINDOW_SERVICE);
		
		DisplayMetrics metrics = new DisplayMetrics();
        windowManager.getDefaultDisplay().getMetrics(metrics);
        mScreenWidth = metrics.widthPixels;
        mScreenHeight = 200;

        // Determine the correct window type
        WindowManager.LayoutParams layoutParams;
        if (android.os.Build.VERSION.SDK_INT >= 26) {
            // For Android 8.0 and above
            layoutParams = new WindowManager.LayoutParams(
                    mScreenWidth,
                    mScreenHeight,
                    LAYOUT_TYPE,
                    WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE | WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL,
                    android.graphics.PixelFormat.TRANSLUCENT
            );
        } else {
            // For below Android 8.0
            layoutParams = new WindowManager.LayoutParams(
                    WindowManager.LayoutParams.WRAP_CONTENT,
                    WindowManager.LayoutParams.WRAP_CONTENT,
                    WindowManager.LayoutParams.TYPE_PHONE,
                    WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE | WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL,
                    android.graphics.PixelFormat.TRANSLUCENT
            );
        }

		layoutParams.gravity = Gravity.TOP | Gravity.CENTER;
        layoutParams.x = 0;
        layoutParams.y = 0;

        // Create a placeholder view
        TextView textView = new TextView(this);
        textView.setText("Loading...");

        // Add the view to the window
        windowManager.addView(textView, layoutParams);
		Log.d(TAG, "Add the view to the window");
        // Initialize Python and request content
        updateFloatingWindowContent(textView);
    }

    

    MediaProjectionManager mediaProjectionManager;
    MediaProjection mMediaProjection;

    @Override
    public int onStartCommand(final Intent intent, int flags, int startId) {
        createNotificationChannel();
        int mResultCode = intent.getIntExtra("code", -1);
        Intent mResultData = intent.getParcelableExtra("data");
        mediaProjectionManager = (MediaProjectionManager)getSystemService(Context.MEDIA_PROJECTION_SERVICE);
        mMediaProjection = mediaProjectionManager.getMediaProjection(mResultCode, Objects.requireNonNull(mResultData));
        Log.e(TAG, "mMediaProjection created: " + mMediaProjection);
        return super.onStartCommand(intent, flags, startId);
    }
	
	private void createNotificationChannel() {
        Notification.Builder builder = new Notification.Builder(this.getApplicationContext()); 
        Intent nfIntent = new Intent(this, MainActivity.class); 
        builder.setContentIntent(PendingIntent.getActivity(this, 0, nfIntent, PendingIntent.FLAG_IMMUTABLE)) 
                .setLargeIcon(BitmapFactory.decodeResource(this.getResources(), R.mipmap.ic_launcher)) 
                .setSmallIcon(R.mipmap.ic_launcher) 
                .setContentText("is running......") 
                .setWhen(System.currentTimeMillis()); 
 
        if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            builder.setChannelId("notification_id");
        }

        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            NotificationManager notificationManager = (NotificationManager)getSystemService(NOTIFICATION_SERVICE);
            NotificationChannel channel = new NotificationChannel("notification_id", "notification_name", NotificationManager.IMPORTANCE_LOW);
            notificationManager.createNotificationChannel(channel);
        }
        Notification notification = builder.build(); 
        notification.defaults = Notification.DEFAULT_SOUND; 
        startForeground(110, notification);
    }
	

    public void onConfigurationChanged(Configuration newConfig) {
        Log.d(TAG, "onConfigurationChanged() start");
        super.onConfigurationChanged(newConfig);
        userCode("onConfigurationChanged", newConfig);
        Log.d(TAG, "onConfigurationChanged() complete");
    }

    public boolean onOptionsItemSelected(MenuItem menuitem) {
        Log.d(TAG, "onOptionsItemSelected() start");
        PyObject pyResult = userCode("onOptionsItemSelected", menuitem);
        boolean result = (pyResult == null) ? false : pyResult.toBoolean();
        Log.d(TAG, "onOptionsItemSelected() complete");
        return result;
    }

    public boolean onPrepareOptionsMenu(Menu menu) {
        Log.d(TAG, "onPrepareOptionsMenu() start");
        PyObject pyResult = userCode("onPrepareOptionsMenu", menu);
        boolean result = (pyResult == null) ? false : pyResult.toBoolean();
        Log.d(TAG, "onPrepareOptionsMenu() complete");
        return result;
    }


    private PyObject userCode(String methodName, Object... args) {
        if (pythonApp == null) {
            // Could be a non-graphical app such as Python-support-testbed.
            return null;
        }
        try {
            if (pythonApp.containsKey(methodName)) {
                return pythonApp.callAttr(methodName, args);
            } else {
                // Handle the case where the method doesn't exist
                return null;
            }
        } catch (PyException e) {
            if (e.getMessage().startsWith("NotImplementedError")) {
                return null;
            }
            throw e;
        }
    }


    @Nullable
    @Override
    public IBinder onBind(Intent intent) {
        return null;
    }
}

Steps to reproduce

1. briefcase run android -d emulator-5554
2. Go to app
3. Click on 'button'
4. See error on console

Expected behavior

a permanent floating window by toga

Screenshots

Environment

• Operating System: windows 10.0.17763.316
• Python version: 3.11.2
• Software versions:
  • Briefcase: 0.3.18
  • Toga: 0.4.5
  • ...

Logs

�[38;5;75mD/FloatingWindowService: Running Python code to update floating window content�[0m
�[38;5;40mI/python.stdout: WARNING: Can't find app icon; falling back to default icon�[0m
�[38;5;166mW/m.example.uiap: Accessing hidden method Ljava/io/FileDescriptor;->setInt$(I)V (greylist,core-platform-api, JNI, allowed)�[0m
�[38;5;166mW/m.example.uiap: Accessing hidden method Landroid/os/Handler;->(Landroid/os/Looper;Landroid/os/Handler$Callback;Z)V (greylist, JNI, allowed)�[0m
�[38;5;166mW/m.example.uiap: Accessing hidden method Landroid/os/Handler;->(Z)V (greylist, JNI, allowed)�[0m
�[38;5;40mI/python.stdout: Python app launched & stored in Android Activity class�[0m
�[38;5;75mD/skia : --- Failed to create image decoder with message 'unimplemented'�[0m

Additional context

No response

[freakboy3742]

Converted to a discussion since this doesn't appear to be a bug in Toga, but a question about Chaquopy usage.

It's not even clear to me that this is a Chaquopy error, as you haven't provided any Python code to demonstrate what is going on, nor have you provided any guidance on how this FloatingWindowService is being used.

You're clearly doing something with skia, as that is the line in the logs that is raising an error, but there's no mention of skia anywhere in your ticket description. My guess is that you've added skia to your Python app and you're assuming it will "just work". This won't be the case, as there will inevitably be a binary component involved to integrate with the Android GUI layer.

However, it's almost impossible to provide debugging advice beyond this when you don't provide a set of instructions that allows us to reproduce the problem.

[freakboy3742]

If you're having difficulty with an Android API, you'll need to take that up with an Android support forum. We're not in a position to provide assistance with using the full Android API.

If you believe the issue is Chaquopy, you'll need to narrow down your request to the specific Chaquopy call that you're having a problem with.

[andy4lee]

As Toga cannot achieve a permanent floating window, we followed the briefcase official example to use it, but it still cannot be implemented by android service. （android activity is normal），then i want to use kivy achieve, However, thank you for your enthusiastic response

[andy4lee]

https://github.com/beeware/briefcase-android-gradle-template/blob/main/%7B%7B%20cookiecutter.format%20%7D%7D/app/src/main/java/org/beeware/android/MainActivity.java

[andy4lee]

https://github.com/beeware/briefcase-android-gradle-template/blob/main/%7B%7B%20cookiecutter.format%20%7D%7D/app/src/main/java/org/beeware/android/MainActivity.java

[freakboy3742]

I still have no idea what it is that you are asking for. Linking to BeeWare's own source code without context does not provide any additional detail that we can use to help you.

You clearly want to build a very specific thing, in a very specific way. However, I have no idea what that thing is. Until you are able to provide details on exactly what it is you are trying to achieve - not just repeating the words "permanent floating window" as if that's a standard Android API concept - then we won't be able to provide any assistance.