Can't get it to work ? (Home Assistant MQTT & SMAhub)

[CarlssonmedC]

Hi - and help! :)

Got my Solar panels installed end of January, with a TriPower X 12 inverter. Long story short, found HomeAssistant and this project and it looked "easy enough"...

Used to consider myself somewhat capable when it comes to configuring stuff like this - but I am about to give up... :) I just can't get my HA-installed MQTT Broker to receive anything from the SMAhub.

Setup:

HA OS installed in KVM environment, bridged network (192.168.50.0/24 network). Ubuntu 22.04 LTS as host.
MQTT Broker & integration installed via Add-On, been following various youtube and written guides and it seems to be installed properly. Have created user and set password in HA and entered this in the config.
Enable Discovery is currently disabled the MQTT options in HA, but have tried with it enabled as well and could not see any difference,

I can connect via MQTT Explorer (running on my windows pc) and publish something, and it pops up in the "Listen to a Topic" from the HA Integration MQTT Settings.

For testing purpose, I have only added a few sensors to HA, in a mqtt.yaml file that is included into main configuration.yaml
sensor. Sensors do show up in HA under entities. (have also tried adding the mqtt lines directly in the configuration.yaml), but when I add them to a card on the dashboard I only get "unknown" as values.

  - name: Device Name
    state_topic: "TriPowerX/SERIAL_NO_IS_ENTERED_HERE/device_info/name"
    icon: "mid:border-all"
  - name: WModCfg
    state_topic: "TriPowerX/SERIAL_NO_IS_ENTERED_HERE/Setpoint/PlantControl/Inverter/WModCfg/WCtlComCfg/W"
    unit_of_measurement: "W"
    device_class: "power"
    state_class: "measurement"

So far, so good, I believe... ?

SMAhub is running in docker container hosted on same Ubuntu-host as mentioned above, Start up with simple

docker-compose up -d

Compose-file basically copy-pasted from the example in the ReadMe:

version: '3'
services:
  smahub:
    image: ghcr.io/anotherdaniel/smahub:latest
    container_name: smahub
    network_mode: host
    restart: unless-stopped
    container_name: smahub
 environment:
      -  DEMOSOURCE_ENABLED=false
      -  DEMOSINK_ENABLED=false
      -  SMAHUB_VERBOSER=true
      -  TRIPOWERX_ENABLED=true
      -  TRIPOWERX_ADDRESS=ip-of-my-inverter
      -  TRIPOWERX_PROTOCOL=https
      -  TRIPOWERX_VERIFYTLS=false
      -  TRIPOWERX_USER=username
      -  TRIPOWERX_PASSWORD=password
      -  TRIPOWERX_PREFIX=TriPowerX
      -  MQTT_ENABLED=false
      -  SHM2_ENABLED=false
      -  HA_MQTT_ENABLED=true
      -  HA_MQTT_ADDRESS=ip-of-my-haos-instance
      -  HA_MQTT_PORT=1883
      -  HA_MQTT_USER=username
      -  HA_MQTT_PASSWORD=password
      -  HA_MQTT_PREFIX=homeassistant
      -  HA_MQTT_UPDATEFREQ=60

Starts without errors, and I can see via

docker logs smahub

that it seems to be able to fetch data from the inverter: (edited for space saving purpose). So the TRIPOWERX-part seems to be working.

2024-02-07 20:00:48 DEBUG    TriPowerXnnnnnnnnnn.device_info.name: Sunny Tripower X 12
2024-02-07 20:00:48 DEBUG    TriPowerXnnnnnnnnnn.device_info.configuration_url: https://192.168.50."X"
...
...
2024-02-07 20:00:49 DEBUG    value of DcMs.Amp is currently not availably (nighttime?)
2024-02-07 20:00:49 DEBUG    value of DcMs.Vol is currently not availably (nighttime?)

(and yes, it is nighttime :D )
But after this... I've hit a wall. I am unable to find any reference as to where to being to look for errors... MQTT is also somewhat new to me, but I feel I grasp the idea of it, and the basics aren't really rocket science...

Any pointers would be highly appreciated! Like, how can I find if the SMAhub actually tries to send towards the HA installed MQTT broker?
Docker & docker networks are also new-ish topics for me - setting SMAhub as "host" doesn't give it it's own ip, but that is perhaps not necessary?
I'm quite certain it is something really simple I've overlooked, but after 3 days of struggling I need help 🗡️

Thanks in advance!

Problem Exists Between Keyboard and Chair?

[CarlssonmedC]

Update, if someone having similar issues and stumbles across this:

Some additional hours of youtube videos, written guides/blogposts reading and sifting through official documentation to learn a bit more of the components...

like a container running
network-mode: host
will simply use the hosts network, and, to the "rest of the world" be seen as simply a service running on the hosts, with all ports exposed per default. Likely common knowledge to most who are using docker etc, but new to me

This led me to play around a bit more, and for testing purposes set up the following:

• a separate "stand alone" Eclipse Mosquitto docker container (running network-mode: host), with default 1883 and 9001 ports set for mqtt and ws.

docker-compose.yaml for reference

version: "3.7"
services:
  # mqtt5 eclipse-mosquitto
  mqtt5:
    image: eclipse-mosquitto
    container_name: mqtt5
    network_mode: host
    volumes:
      - ./config:/mosquitto/config:rw
      - ./data:/mosquitto/data:rw
      - ./log:/mosquitto/log:rw
    restart: unless-stopped

• changed my smahub container from HA-MQTT to MQTT and set the hosts ip address as "target" (MQTT_ADDRESS)

With this setup, I can easily access the mosquitto broker container with MQTT Explorer,where I can see that smahub is pouring data onto it, with the TriPowerXXXXXXXX (where XXXXXX is the serial of my inverter) as Topic.

And while this confirms that I got the SMA hub to work, it still puts me back to square 1 in terms of getting the data to the HA-MQTT broker...

Sidenote: I keep running into the following error message when trying the mosquitto_pub or _sub commands... and it doesn't matter if I run them in side the "new" mosquitto broker, from my linux host or from the HAOS terminal. Think I've run out of google-replies to read on this, but no anwer seems to match my setup/situation.
if anyone have any idea, I'm all ears... 😄

/ # mosquitto_sub -h MQTT_BROKER-HOST-IP -u USERNAME -p PASSWORD -t '#'
Error: No such file or directory

[AnotherDaniel]

Hi @CarlssonmedC - first, I have to offer an apology for not reacting to your question earlier; I saw the notification email fly by a couple weeks ago, but have been somewhat ops-blind and only looked into the issue threads since then.

Ok, just for my benefit: can you connect to (with MQTT Explorer or any other client) the mosquitto instance that is running with your home assistant? I have never run HA-OS, so we need to step through this together...

[CarlssonmedC]

No worries @AnotherDaniel 😄

But yes -
I can connect with MQTT Explorer from my windows PC to the MQTT-broker running as an addon-on HA.
I am able to generate topics from the MQTT Explorer and publish so that I, from the HA GUI in the MQTT integration page, am able to see the topics pop up - as per image below

From the smahub containers logs, I can see that the script is actually connecting to the inverter and is showing values as I would expect is correct. But from here, I am sort of lost on how to trace where the traffic goes 😄

The MQTT logs on HA side doesn't show the smahub even connecting - I do see the mqtt explorer connecting though. (MQTT Explorer from my win-PC, smahub running on my linux "homelab" machine, so different IP. (HAOS runs as VM in KVM on the same linux host)
No local firewalls enabled on either.

The test I did earlier, was to spin up a stand-alone Eclipse Mosquitto container and enabled the "normal" MQTT env variables in the smahub docker-compose file. Connected MQTT Explorer to that instances and started smahub again (with new config pointing it towards the "normal" mqtt instance) - and I could then see the TriPowerX topic and subtopics in MQTT Explorer, along with all regular updates it pushed.

Have now tried to reinstall all MQTT related addons etc on HA, but I get the exact same situation as described in the beginning....

[AnotherDaniel]

Hm... sounds tricky. One thing on your mosqitto_sub command: I think you're using a small p ('-p') to set the password - but small p is for passing the network port to connect to. For password, use a capital P ('-P')

Fyi: the smahub container config uses the 'host' network to enable smahub to receive SHM2 broadcast data frames - when you're not using SHM2, it should be possible to use smahub without network: host.

I don't really have a good idea how to pursue this; with the mosquitto_sub command 'fix' above, can you subscribe/publish to the HA MQTT instance from a shell on your linux system?

[CarlssonmedC]

thanks for the info on the -p vs -P

Re-read the docs for mosquitto_sub/pub and yes, it is -p for port and -P for password. Changed that and can now publish from the command line of my linux host to the mqtt broker on the HA-OS

mosquitto_pub -h 192.168.50.160 -u **redacted** --P **redacted** -d -t testing/from/scarif/pub -m 'test1'
Client (null) sending CONNECT
Client (null) received CONNACK (0)
Client (null) sending PUBLISH (d0, q0, r0, m1, 'testing/from/scarif/pub', ... (5 bytes))
Client (null) sending DISCONNECT

[AnotherDaniel]

Ok, and just to be sure: the exact same config does not work with smahub?
Another question: your standalone mqtt broker, did that use user/password authentication?

[CarlssonmedC]

Same user and pwd from mqtt explorer as in the smahub config and from the mosquitto_pub command, with the same ip as host (ie the HAOS vm's IP) And yes, the stand alone used user & pwd as well. Den lör 10 feb. 2024 22:03Daniel ***@***.***> skrev:

…

Ok, and just to be sure: the exact same config does not work with smahub? Another question: your standalone mqtt broker, did that use user/password authentication? — Reply to this email directly, view it on GitHub <#38 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFYXGSPHJCXVEVRROZOBTUTYS7OAZAVCNFSM6AAAAABC6Q7EA2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIMRZGY4DE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AnotherDaniel]

Strange.

On the machine where the smahub container runs (while it is running), can you do

docker exec -it smahub sh

(smahub is the container name)
and then, in the in-container shell, ping the ip of your mqtt broker?

[CarlssonmedC]

Yes, works fine Den lör 10 feb. 2024 22:27Daniel ***@***.***> skrev:

…

Strange. On the machine where the smahub container runs (while it is running), can you do docker exec -it smahub sh (smahub is the container name) and then, in the in-container shell, ping the ip of your mqtt broker? — Reply to this email directly, view it on GitHub <#38 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFYXGSIC2TAQKTXEVRUJ4ZTYS7Q3ZAVCNFSM6AAAAABC6Q7EA2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIMRZG44DK> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AnotherDaniel]

Pretty much out of ideas, then. Phew...

[CarlssonmedC]

Yeah, me too... :D Will see if I can find more detailed logs on the HA MQTT side that might show if the smahub traffic reaches it at all... or simply switch to a pure container setup with stand alone broker. Getting the info from the inverter is sort of the main purpose of my HA "journey" :) Thans for the help and effort! If I stumble across a solution I'll update the thread again. Cheers & Have a nice weekend! Den lör 10 feb. 2024 22:56Daniel ***@***.***> skrev:

…

Pretty much out of ideas, then. Phew... — Reply to this email directly, view it on GitHub <#38 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFYXGSO4GIADW55JFSIYHATYS7UJJAVCNFSM6AAAAABC6Q7EA2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIMRZHEYDQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AnotherDaniel]

One of the not-very-pretty things you could do while looking for the actual root cause: set up mqtt bridging between the standalone mosquitto and the ha one. If that works, you'll have a extraneous mosquitto instance in your system, but you'll get the data flowing...

[CarlssonmedC]

Yeah, I was thinking about that as well - but as you say, it is sort of ugly 😆 Will tweak around a bit and see what I end up with.

I was already looking at going "pure docker" anyway, so will try and spin up a HA docker instance and test smahub towards that with a stand-alone broker as well.. One of the upisdes I have found since starting out with docker, it's quite quick and easy to set up stuff for this sort of testing... (I left/drifted out of the IT world back when VM Ware was the No 1 topic and companies started going from huge datacenters with tens of thousands of physical servers to just using a room or two filled with ESXi's and virtualizing everything, and installing a new VM was about as much work as installing a physical one - so the quick n easy work with docker is just a fantastic new experience 😁 )

Btw, Is there any way to set the MQTT QoS from smahub? Or is set somewhere in the code already? haven't seen anything in the docs here related to QoS. (From what I understand, the QoS level is set from the publishing client?) Was thinking if a high QoS might generate some output in any log somehwere...

[AnotherDaniel]

smahub doesn't do anything with QoS at the moment - never any need, yet, as the environment it's in doesn't seem to be anywhere close to any limit.

[CarlssonmedC]

Thanks for the answer regarding the QoS!

However, after some additional tinkering, I simply tried the following in the smahub docker-compose - as I realized this was one thing I haven't tried yet...

Enabled the "normal" MQTT and disabled the HA-specific one. Same IP, same username and password.

      -  MQTT_ENABLED=true
      -  MQTT_ADDRESS=192.168.50.160
      -  MQTT_PORT=1883
      -  MQTT_USER=username
      -  MQTT_PASSWORD=password

      -  HA_MQTT_ENABLED=false
      -  HA_MQTT_ADDRESS=192.168.50.160
      -  HA_MQTT_PORT=1883
      -  HA_MQTT_USER=username
      -  HA_MQTT_PASSWORD=password
      -  HA_MQTT_PREFIX=homeassistant
      -  HA_MQTT_UPDATEFREQ=60

And violá!

Both the broker log on the HAOS reports new client connected as 'smahub' and I can from MQTT Explorer see that my HAOS MQTT Broker is now showing the TriPower topics published from smahub.

From the code-perspective, what is different between the two? (HA-MQTT and MQTT?) Perhaps there has been some change on the HA MQTT Broker add-on... (or there is simply just trolls running in my setup 😆 )

[AnotherDaniel]

Ah! The difference is that mqtt directly uses the paho mqtt library to publish 'raw' content to the target broker.
Ha-mqtt uses a library that generates topics and messages in a the ha autodiscovery format, so that no additional sensor configurations are needed on ha side.

You can connect to a broker with either, should see very different topic structures.

[CarlssonmedC]

Ok! Would be sweet to have that available perhaps... is a bit tedious to manually config all sensors on HA side 😆
Will hopefully get some time later tonight to play around with that, as I still have that mqtt stand-alone container configured and available, just not spun up.

[CarlssonmedC]

Hello again, @AnotherDaniel ! 😄

Had some more time to spare, and dug into trying to get the HA_MQTT to work, but alas...

However, there seems like something happens with the HA_MQTT "thingy" once you put it in a docker.

So what I've done:

Using the same docker image and compose file, I first ran with regular MQTT while capturing traffic with tcpdump.
I then stopped (docker-compose down) the smahub container (and the active tcpdump), altered the compose file to use HA_MQTT sink instead, started a new tcpdump and then ran docker-compose up.

Test with MQTT-sink works fine, as I've confirmed earlier

The HA_MQTT however, after the first initial setup, it starts sending over UDP ?

Next step was to remove everything from docker related to samhub, before following the guide on this page to download the source and run smahub as a python script only.

And this works flawlessly. I can see both in HA and in MQTT Explorer that the python script successfully publishes with the HA_MQTT sink as enabled (and the the 'regular' MQTT sink disabled).

Went on an build new image from the code just downloaded.... aaaaand we're back at square one again, with smahub as a docker-container, once initial MQTT ping request and response is done it swaps over to pushing UDP traffic.

Not sure if this is something you can or even want to look at (or if it's even something that is related to the code you've done or if this is something more upstream in python vs docker compose level... ? ), but thought I'd report back my findings at least 😄 Will continue to run the normal MQTT sink for now, as I really like having smahub in a docker.... if you ever get around to finding and fixing this, would love a ping back 👍

Have a nice weekend!

[AnotherDaniel]

This is really a bummer - also, I (only) run smahub in Docker, and this definitely can work... don't know yet what exactly I will be able to do here, but I'll keep this on the investigation list.

[CarlssonmedC]

I take it you use the HA-MQTT then as well?

Really weird that my installation acts the way it does.... 😄

A bit short on time to play around the coming weeks, but will continue to test. And as long as this winter keeps pouring snow on us, the panels won't get used much anyway....

But if you stumble across something, it'd be interesting to hear about it. (Likewise, I'll post any progress I might make). 😄

[AnotherDaniel]

Yes, I'm using ha-mqtt, mosquitto and ha, all in containers. What I'm doing is, I'm operating these three using one docker-compose configuration. No special network fanciness involved, except for abovementioned reason (receiving SHM2 datagrams) I'm using network mode 'host' for smahub.