dapr Actor runtime issue in Kubernetes

[tvikure]

Ask your question here

I am trying to run a nodejs express app to create actors on demand in kubernetes. I have below services running in dapr-system namespace.

dapr-operator-559ff985fd-7s4wj
dapr-placement-server-0
dapr-scheduler-server-0
dapr-sentry-7f6966fd59-mg8jt
dapr-sidecar-injector-65fd7d94c5-qks5q

in the default namespace

my-node-app and redis is running.

When trying to create actors on demand facing the below error. Please suggest. Trying to fix this from 48 hours. this is only issue when running kubernetes cluster. When I try to run in local using dapr run it is working fine.

using below versions.

"@dapr/dapr": "^3.3.1",
"dapr-client": "^3.3.1",
"express": "^4.19.2"

Daprd failure (attached detailed log file)

time="2024-09-29T12:01:17.372807304Z" level=warning msg="Error processing operation DaprBuiltInActorNotFoundRetries. Retrying in 1s…" app_id=my-node-app instance=my-node-app-7b84959966-m4whv scope=dapr.runtime type=log ver=1.14.4

my-node-app logs

node dist/index.js
server created v6
Running API server on 3000
registered actors [ 'MyActor' ]
2024-09-29T12:00:32.803Z INFO [HTTPServer, Actors] Initializing Actors
Starting Dapr server
2024-09-29T12:00:32.805Z INFO [HTTPServer, HTTPServer] Listening on 50004
2024-09-29T12:00:32.805Z INFO [HTTPServer, HTTPServer] Registering 0 PubSub Subscriptions
2024-09-29T12:00:32.810Z INFO [HTTPClient, HTTPClient] Sidecar Started
Server started successfully
builder
actor [class MyActor extends AbstractActor]
2024-09-29T12:01:17.371Z INFO [HTTPClient, HTTPClient] Sidecar Started
error creating actors Error: {"error":"Internal Server Error","error_msg":"{"errorCode":"ERR_ACTOR_INVOKE_METHOD","message":"error invoke actor method: did not find address for actor MyActor/asset-0"}","status":500}
at HTTPClient.execute (/usr/src/app/node_modules/@dapr/dapr/implementation/Client/HTTPClient/HTTPClient.js:172:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async ActorClientHTTP.invoke (/usr/src/app/node_modules/@dapr/dapr/actors/client/ActorClient/ActorClientHTTP.js:21:24)
at async Proxy. (/usr/src/app/node_modules/@dapr/dapr/actors/client/ActorProxyBuilder.js:38:33)

import express from 'express';
import { ActorProxyBuilder, DaprServer, DaprClient, ActorId } from "@dapr/dapr";
import { MyActor, IMyActor } from './actors/MyActor';

const app = express();
app.use(express.json());

const daprHost = "127.0.0.1";
const daprPort = "3500";
const serverHost = "127.0.0.1";
const serverPort = "50004";

const server = new DaprServer({
  serverHost,
  serverPort,
  clientOptions: {
  daprHost: daprHost,
  daprPort: daprPort,
  actor: {
    actorIdleTimeout: "1h",
    actorScanInterval: "30s",
    drainRebalancedActors: true,
    reentrancy: {
      enabled: true,
      maxStackDepth: 32,
    },
    remindersStoragePartitions: 0,
  },
}});

console.log('server created');

const client = new DaprClient({ daprHost, daprPort });

async function start() { 
server.actor.registerActor(MyActor)
await new Promise(resolve => setTimeout(resolve, 5000));
const actors = await server.actor.getRegisteredActors();
console.log('registered actors ', actors);
await server.actor.init(); 

console.log("Starting Dapr server ");
await server.start();
console.log("Server started successfully");
console.log("builder  ");
}

app.post('/api/createActor', async (req, res) => {
  const { actorType } = req.body;
  const builder = new ActorProxyBuilder<IMyActor>(MyActor, client);
  const actor = builder.build(new ActorId(`asset-1`));
  console.log('actor', actor);

  await actor.start() 
});`


app.listen(3000, () => {
  console.log('Running API server on 3000');
});


start().catch(e => console.error(e));

```

Deployment yaml

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-node-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-node-app
  template:
    metadata:
      labels:
        app: my-node-app
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "my-node-app"
        dapr.io/app-port: "3000"
        dapr.io/app-protocol: "http"
        dapr.io/actor-port: "50004"
    spec:
      imagePullSecrets:
        - name: myregistrykey
      containers:
        - name: my-node-app
          image: mydockerrepo/my-node-app:v1.11
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: my-node-app 
spec:
  selector:
    app: my-node-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
  type: LoadBalancer
```
  

[daprd_log.txt](https://github.com/user-attachments/files/17179948/daprd_log.txt)