Merge pull request #1434 from hanyin-arm/hanyin-selfie-app

New Learning Path: "Build a Hands-Free Selfie app with Modern Android Development and MediaPipe Multimodal AI"

assets/contributors.csv

@@ -48,3 +48,4 @@ Chen Zhang,Zilliz,,,,
 Tianyu Li,Arm,,,,
 Georgios Mermigkis,VectorCamp,gMerm,georgios-mermigkis,,https://vectorcamp.gr/ 
 Ben Clark,Arm,,,,
+Han Yin,Arm,hanyin-arm,nacosiren,,

content/learning-paths/smartphones-and-mobile/build-android-selfie-app-using-mediapipe-multimodality/2-app-scaffolding.md

@@ -0,0 +1,197 @@
+---
+title: Scaffold a new Android project
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+This learning path will teach you to architect an app following [modern Android architecture](https://developer.android.com/courses/pathways/android-architecture) design with a focus on the [UI layer](https://developer.android.com/topic/architecture/ui-layer).
+
+## Development environment setup
+
+Download and install the latest version of [Android Studio](https://developer.android.com/studio/) on your host machine.
+
+This learning path's instructions and screenshots are taken on macOS with Apple Silicon, but you may choose any of the supported hardware systems as described [here](https://developer.android.com/studio/install).
+
+Upon first installation, open Android Studio and proceed with the default or recommended settings. Accept license agreements and let Android Studio download all the required assets.
+
+Before you proceed to coding, here are some tips that might come handy:
+
+{{% notice Tip %}}
+1. To navigate to a file, simply double-tap `Shift` key and input the file name, then select the correct result using `Up` & `Down` arrow keys and then tap `Enter`.
+
+2. Every time after you copy-paste a code block from this learning path, make sure you **import the correct classes** and resolved the errors. Refer to [this doc](https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html) to learn more.
+{{% /notice %}}
+
+## Create a new Android project
+
+1. Navigate to **File > New > New Project...**.
+
+2. Select **Empty Views Activity** in **Phone and Tablet** galary as shown below, then click **Next**.
+![Empty Views Activity](images/2/empty%20project.png)
+
+3. Proceed with a cool project name and default configurations as shown below. Make sure that **Language** is set to **Kotlin**, and that **Build configuration language** is set to **Kotlin DSL**.
+![Project configuration](images/2/project%20config.png)
+
+### Introduce CameraX dependencies
+
+[CameraX](https://developer.android.com/media/camera/camerax) is a Jetpack library, built to help make camera app development easier. It provides a consistent, easy-to-use API that works across the vast majority of Android devices with a great backward-compatibility.
+
+1. Wait for Android Studio to sync project with Gradle files, this make take up to several minutes.
+
+2. Once project is synced, navigate to `libs.versions.toml` in your project's root directory as shown below. This file serves as the version catalog for all dependencies used in the project.
+
+![version catalog](images/2/dependency%20version%20catalog.png)
+
+{{% notice Info %}}
+
+For more information on version catalogs, please refer to [this doc](https://developer.android.com/build/migrate-to-catalogs).
+
+{{% /notice %}}
+
+3. Append the following line to the end of `[versions]` section. This defines the version of CameraX libraries we will be using.
+```toml
+camerax = "1.4.0"
+```
+
+4. Append the following lines to the end of `[libraries]` section. This declares the group, name and version of CameraX dependencies.
+
+```toml
+camera-core = { group = "androidx.camera", name = "camera-core", version.ref = "camerax" }
+camera-camera2 = { group = "androidx.camera", name = "camera-camera2", version.ref = "camerax" }
+camera-lifecycle = { group = "androidx.camera", name = "camera-lifecycle", version.ref = "camerax" }
+camera-view = { group = "androidx.camera", name = "camera-view", version.ref = "camerax" }
+```
+
+5. Navigate to `build.gradle.kts` in your project's `app` directory, then insert the following lines into `dependencies` block. This introduces the above dependencies into the `app` subproject.
+
+```kotlin
+    implementation(libs.camera.core)
+    implementation(libs.camera.camera2)
+    implementation(libs.camera.lifecycle)
+    implementation(libs.camera.view)
+```
+
+## Enable view binding
+
+1. Within the above `build.gradle.kts` file, append the following lines to the end of `android` block to enable view binding feature.
+
+```kotlin
+    buildFeatures {
+        viewBinding = true
+    }
+```
+
+2. You should be seeing a notification shows up, as shown below. Click **"Sync Now"** to sync your project.
+
+![Gradle sync](images/2/gradle%20sync.png)
+
+{{% notice Tip %}}
+
+You may also click the __"Sync Project with Gradle Files"__ button in the toolbar or pressing the corresponding shorcut to start a sync.
+
+![Sync Project with Gradle Files](images/2/sync%20project%20with%20gradle%20files.png)
+{{% /notice %}}
+
+3. Navigate to `MainActivity.kt` source file and make following changes. This inflates the layout file into a view binding object and stores it in a member variable within the view controller for easier access later.
+
+![view binding](images/2/view%20binding.png)
+
+## Configure CameraX preview
+
+1. **Replace** the placeholder "Hello World!" `TextView` within the layout file `activity_main.xml` with a camera preview view:
+
+```xml
+    <androidx.camera.view.PreviewView
+        android:id="@+id/view_finder"
+        android:layout_width="match_parent"
+        android:layout_height="match_parent"
+        app:scaleType="fillStart" />
+```
+
+
+2. Add the following member variables to `MainActivity.kt` to store camera related objects:
+
+```kotlin
+    // Camera
+    private var camera: Camera? = null
+    private var cameraProvider: ProcessCameraProvider? = null
+    private var preview: Preview? = null
+```
+
+3. Add two new private methods named `setupCamera()` and `bindCameraUseCases()` within `MainActivity.kt`:
+
+```kotlin
+    private fun setupCamera() {
+        viewBinding.viewFinder.post {
+            cameraProvider?.unbindAll()
+
+            ProcessCameraProvider.getInstance(baseContext).let {
+                it.addListener(
+                    {
+                        cameraProvider = it.get()
+
+                        bindCameraUseCases()
+                    },
+                    Dispatchers.Main.asExecutor()
+                )
+            }
+        }
+    }
+
+    private fun bindCameraUseCases() {
+        // TODO: TO BE IMPLEMENTED
+    }
+```
+
+4. Implement the above `bindCameraUseCases()` method:
+
+```kotlin
+private fun bindCameraUseCases() {
+        val cameraProvider = cameraProvider
+            ?: throw IllegalStateException("Camera initialization failed.")
+
+        val cameraSelector =
+            CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_FRONT).build()
+
+        // Only using the 4:3 ratio because this is the closest to MediaPipe models
+        val resolutionSelector =
+            ResolutionSelector.Builder()
+                .setAspectRatioStrategy(AspectRatioStrategy.RATIO_4_3_FALLBACK_AUTO_STRATEGY)
+                .build()
+        val targetRotation = viewBinding.viewFinder.display.rotation
+
+        // Preview usecase.
+        preview = Preview.Builder()
+            .setResolutionSelector(resolutionSelector)
+            .setTargetRotation(targetRotation)
+            .build()
+
+        // Must unbind the use-cases before rebinding them
+        cameraProvider.unbindAll()
+
+        try {
+            // A variable number of use-cases can be passed here -
+            // camera provides access to CameraControl & CameraInfo
+            camera = cameraProvider.bindToLifecycle(
+                this, cameraSelector, preview,
+            )
+
+            // Attach the viewfinder's surface provider to preview use case
+            preview?.surfaceProvider = viewBinding.viewFinder.surfaceProvider
+        } catch (exc: Exception) {
+            Log.e(TAG, "Use case binding failed", exc)
+        }
+    }
+```
+
+5. Add a [companion object](https://kotlinlang.org/docs/object-declarations.html#companion-objects) to `MainActivity.kt` and declare a `TAG` constant value for `Log` calls to work correctly. This companion object comes handy for us to define all the constants and shared values accessible across the entire class.
+
+```kotlin
+    companion object {
+        private const val TAG = "MainActivity"
+    }
+```
+
+In the next chapter, we will build and run the app to make sure the camera works well.

content/learning-paths/smartphones-and-mobile/build-android-selfie-app-using-mediapipe-multimodality/3-camera-permission.md

@@ -0,0 +1,118 @@
+---
+title: Handle camera permission
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Run the app on your device
+
+1. Connect your Android device to your computer via a USB **data** cable. If this is your first time running and debugging Android apps, follow [this guide](https://developer.android.com/studio/run/device#setting-up) and double check this checklist:
+
+    1. You have enabled **USB debugging** on your Android device following [this doc](https://developer.android.com/studio/debug/dev-options#Enable-debugging). 
+
+    2. You have confirmed by tapping "OK" on your Android device when an **"Allow USB debugging"** dialog pops up, and checked "Always allow from this computer".
+
+    ![Allow USB debugging dialog](https://ftc-docs.firstinspires.org/en/latest/_images/AllowUSBDebugging.jpg)
+
+
+2. Make sure your device model name and SDK version correctly show up on the top right toolbar. Click the **"Run"** button to build and run, as described [here](https://developer.android.com/studio/run).
+
+3. After waiting for a while, you should be seeing success notification in Android Studio and the app showing up on your Android device. 
+
+4. However, the app shows only a black screen while printing error messages in your [Logcat](https://developer.android.com/tools/logcat) which looks like this:
+
+```
+2024-11-20 11:15:00.398 18782-18818 Camera2CameraImpl       com.example.holisticselfiedemo      E  Camera reopening attempted for 10000ms without success.
+2024-11-20 11:30:13.560   667-707   BufferQueueProducer     pid-667                              E  [SurfaceView - com.example.holisticselfiedemo/com.example.holisticselfiedemo.MainActivity#0](id:29b00000283,api:4,p:2657,c:667) queueBuffer: BufferQueue has been abandoned
+2024-11-20 11:36:13.100 20487-20499 isticselfiedem          com.example.holisticselfiedemo      E  Failed to read message from agent control socket! Retrying: Bad file descriptor
+2024-11-20 11:43:03.408  2709-3807  PackageManager          pid-2709                             E  Permission android.permission.CAMERA isn't requested by package com.example.holisticselfiedemo
+```
+
+5. Worry not. This is expected behavior because we haven't correctly configured this app's [permissions](https://developer.android.com/guide/topics/permissions/overview) yet, therefore Android OS restricts this app's access to camera features due to privacy reasons. 
+
+## Request camera permission at runtime
+
+1. Navigate to `manifest.xml` in your `app` subproject's `src/main` path. Declare camera hardware and permission by inserting the following lines into the `<manifest>` element. Make sure it's **outside** and **above** `<application>` element.
+
+```xml
+    <uses-feature
+        android:name="android.hardware.camera"
+        android:required="true" />
+    <uses-permission android:name="android.permission.CAMERA" />
+```
+
+2. Navigate to `strings.xml` in your `app` subproject's `src/main/res/values` path. Insert the following lines of text resources, which will be used later.
+
+```xml
+    <string name="permission_request_camera_message">Camera permission is required to recognize face and hands</string>
+    <string name="permission_request_camera_rationale">To grant Camera permission to this app, please go to system settings</string>
+```
+
+3. Navigate to `MainActivity.kt` and add the following permission related values to companion object:
+
+```kotlin
+        // Permissions
+        private val PERMISSIONS_REQUIRED = arrayOf(Manifest.permission.CAMERA)
+        private const val REQUEST_CODE_CAMERA_PERMISSION = 233
+```
+
+4. Add a new method named `hasPermissions()` to check on runtime whether camera permission has been granted:
+
+```kotlin
+    private fun hasPermissions(context: Context) = PERMISSIONS_REQUIRED.all {
+        ContextCompat.checkSelfPermission(context, it) == PackageManager.PERMISSION_GRANTED
+    }
+```
+
+5. Add a condition check in `onCreate()` wrapping `setupCamera()` method, to request camera permission on runtime.
+
+```kotlin
+        if (!hasPermissions(baseContext)) {
+            requestPermissions(
+                arrayOf(Manifest.permission.CAMERA),
+                REQUEST_CODE_CAMERA_PERMISSION
+            )
+        } else {
+            setupCamera()
+        }
+```
+
+6. Override `onRequestPermissionsResult` method to handle permission request results:
+
+```kotlin
+    override fun onRequestPermissionsResult(
+        requestCode: Int,
+        permissions: Array<out String>,
+        grantResults: IntArray
+    ) {
+        when (requestCode) {
+            REQUEST_CODE_CAMERA_PERMISSION -> {
+                if (PackageManager.PERMISSION_GRANTED == grantResults.getOrNull(0)) {
+                    setupCamera()
+                } else {
+                    val messageResId =
+                        if (shouldShowRequestPermissionRationale(Manifest.permission.CAMERA))
+                            R.string.permission_request_camera_rationale
+                        else
+                            R.string.permission_request_camera_message
+                    Toast.makeText(baseContext, getString(messageResId), Toast.LENGTH_LONG).show()
+                }
+            }
+            else -> super.onRequestPermissionsResult(requestCode, permissions, grantResults)
+        }
+    }
+```
+
+## Verify camera permission
+
+1. Rebuild and run the app. Now you should be seeing a dialog pops up requesting camera permissions! 
+
+2. Tap `Allow` or `While using the app` (depending on your Android OS versions), then you should be seeing your own face in the camera preview. Good job!  
+
+{{% notice Tip %}}
+Sometimes you might need to restart the app to observe the permission change take effect.
+{{% /notice %}}
+
+In the next chapter, we will introduce MediaPipe vision solutions.