Successfully migrated from ZHA to Z2M without re-pairing

[seidtgeist]

@Koenkk First of all, love the project! Thank you! I'm so glad to be able to set light gradients on my Signe again, and the software updates are a welcome plus (for no reason other than feeling up to date).

So, I've had a fun night and automated the device migration part with a script:

1. Stop Z2M
2. Get list of devices and manufacturer name from ZHA zigbee.db
3. Find device by IEEE to get friendly name from HA config
4. For every device, create the minimum viable JSON database.db entry, e.g.:

   {
     "id": 3,                            // next highest id
     "type": "EndDevice",                // this will be updated by interview process
     "ieeeAddr": "0x0017880110409015",   // reformatted ieee from zigbee.db
     "nwkAddr": 19243,                   // nwk from zigbee.db
     "manufId": 4107                     // endpoint attribute mapped from zigbee.db, mapped via Z2M's cluster.ts
   }

5. Append one entry per line for every device in zigbee2mqtt/database.db
6. For friendly device names, create this YAML snippet for zigbee2mqtt/configuration.yaml:

   devices:
     '0x0017880110409015':
       friendly_name: Office Floor
     # ...

7. Start Z2M
8. Start interview process for each device
9. Done. No need to enable permit join, re-pair or factory-reset anything in my case. Z2M will fill in the device endpoints, cluster data, etc.

This could be automated further for a full ZHA import:

1. Extract PAN, Extended PAN, Network Key and Channel from ZHA network config
2. Follow the above steps for the device database
3. Trigger interview process for all devices via MQTT

I guess you know all this already, but I've read quite a few forum discussion and watched youtube videos where people crawl through their homes, re-pairing all of their devices manually 🤭

I could write this as a standalone TS script if you're interested?

[Koenkk]

Interesting! Had a chat with @Nerivec about this, it should be easy to add an automatic interview when an unknown device sends a message.

So the procedure would be:

• Make sure network settings are the same (key, channel, panid and ext panid). Can these be retrieved easily?
• Start z2m, trigger every device such that it sends a message -> device gets interviewed

[seidtgeist]

• Make sure network settings are the same (key, channel, panid and ext panid). Can these be retrieved easily?

Yes. Want me to write a function which reads the relevant ZHA settings and returns them?

• Start z2m, trigger every device such that it sends a message -> device gets interviewed

That's right.

No more pair-crawling :D

[Koenkk]

Yes. Want me to write a function which reads the relevant ZHA settings and returns them?

Yes, would be cooler if it could also generate a part of the z2m configuration.yaml!

[Nerivec]

key, channel, panid and ext panid

That's the stuff from the coordinator_backup.json, so, should already be fine on that point. Just need to copy it over from ZHA (I think you can even download it from ZHA?).

The newer coordinator_backup.json format (once finalized) should have an independent list of ieee/nwk pairs, which would allow to fill in the database with the minimal info, from Controller, if the result of adapter start was restore and the database is empty. Then we'd have to trigger a round of (throttled) interviews for routers (easy enough). Then trigger interviews for end devices on reception of a message (should prevent too many failures because of sleep, keep manual interactions to a minimum). Since the database would be filled with minimal info, we can simply use the interviewed/interviewing fields to dictate behavior on this.

Just a quick thought, but something along these lines should be doable.
Definitely on the TODO list 😉

[JayGoldberg]

In my case the downloaded ZHA backup 2024-11-08T05-25-59.735Z.json (Download Backup in the ZHA Configure window) contains keys and IDs but not friendly names, etc. It seems to be a radio backup?

As I understand, all of this is already on the radio so you should be able to just stop ZHA, start Z2M and point it at the same radio? As @thargy alludes to in this discussion.

[thargy]

Only the IEEE and Network address is held on each radio to confirm it is registered with the network. The device's manufacturer ID, model, clusters, etc. can be interviewed. What isn't available is a friendly name/entity ID, so either that data needs to be grabbed from ZHA's config, or supplied manually.

There's an argument for Z2m to populate the IEEEAddress as an attribute against the entity in HA as it would be useful for restoring an association. I've also asked about populating a list of child entities for groups, or at least supply a GroupID.

[seidtgeist]

What to do after coming home from a nice party? Feed the cats and deliver the code, of course!

Note: The most straightforward way would be to instruct users to download a backup from ZHA and then disable it. However, you could probably also just curl the backup from the other extension.

example.ts

import { convertZHAtoZ2M } from './slurpbee.ts'

// Excerpt of actual ZHA backup.json
const zhaBackup = {
    "version": 1,
    "network_info": {
        "extended_pan_id": "04:08:15:16:23:42:13:37",
        "pan_id": "78B3",
        "channel": 25,
        "network_key": {
            "key": "04:08:15:16:23:42:13:37:04:08:15:16:23:42:13:37"
        }
    }
};

// There might be a better way of getting this data and I don't know it
const haRegistry = JSON.parse(await Deno.readTextFile(
    'path/to/.storage/core.device_registry'
));

// Convert to Z2M format and then to YAML
const z2mConfig = convertZHAtoZ2M(zhaBackup, haRegistry);
const yamlOutput = jsyaml.dump(z2mConfig, {
    indent: 2,
    lineWidth: -1,
    noRefs: true,
});

console.log(yamlOutput);

advanced:
  pan_id: 30899
  ext_pan_id:
    - 4
    - 8
    - 21
    - 22
    - 35
    - 66
    - 19
    - 55
  channel: 25
  network_key:
    - 4
    - 8
    - 21
    - 22
    - 35
    - 66
    - 19
    - 55
    - 4
    - 8
    - 21
    - 22
    - 35
    - 66
    - 19
    - 55
devices:
  0xe0:79:8d:ff:fe:cf:c3:09:
    friendly_name: 'Generic Zigbee Coordinator (EZSP)' # note: not sure the coordinator should be imported, probably not :) i didn't do it.
  0xb4:35:22:ff:fe:56:c2:ef:
    friendly_name: Boiler Room Mirror Right
  0xb4:35:22:ff:fe:c6:b1:96:
    friendly_name: Boiler Room Mirror Left
  0x00:17:88:01:0b:7f:38:26:
    friendly_name: Boiler Room Ceiling

slurpbee.ts

import yaml from 'js-yaml';

interface ZHABackup {
    version: number;
    network_info: {
        extended_pan_id: string;
        pan_id: string;
        channel: number;
        network_key: {
            key: string;
        };
    };
}

interface HADevice {
    identifiers: [string, string][];
    name_by_user: string | null;
    name: string;
    area_id?: string | null;
}

interface HARegistry {
    data: {
        devices: HADevice[];
    };
}

interface Z2MConfig {
    advanced: {
        pan_id: number;
        ext_pan_id: number[];
        channel: number;
        network_key: number[];
    };
    devices: {
        [key: string]: {
            friendly_name: string;
        };
    };
}

function hexStringToNumbers(hexString: string): number[] {
    return hexString.split(':').map(hex => parseInt(hex, 16));
}

/**
 * Extracts ZHA devices from Home Assistant's device registry
 */
function extractZHADevices(registry: HARegistry): Record<string, string> {
    const devices: Record<string, string> = {};

    registry.data.devices
        .filter(device =>
            device.identifiers?.length > 0 &&
            device.identifiers[0][0] === 'zha'
        )
        .forEach(device => {
            const ieee = device.identifiers[0][1];
            // Prefer user-defined name, fall back to default name
            const name = device.name_by_user || device.name;
            devices[ieee] = name;
        });

    return devices;
}

function convertZHAtoZ2M(zhaBackup: ZHABackup, registry: HARegistry): Z2MConfig {
    const zhaDevices = extractZHADevices(registry);

    return {
        advanced: {
            pan_id: parseInt(zhaBackup.network_info.pan_id, 16),
            ext_pan_id: hexStringToNumbers(zhaBackup.network_info.extended_pan_id),
            channel: zhaBackup.network_info.channel,
            network_key: hexStringToNumbers(zhaBackup.network_info.network_key.key),
        },
        devices: Object.fromEntries(
            Object.entries(zhaDevices).map(([ieee, name]) => [
                `0x${ieee}`,  // Ensure IEEE has 0x prefix
                { friendly_name: name }
            ])
        )
    };
}

export { convertZHAtoZ2M, type ZHABackup, type Z2MConfig };

[thargy]

Note you only need the IEEE and Network addresses. You can set the "type" to "Unknown". The interview will grab the correct type and everything else (including the Manufacturer ID).

For coordinator's that support backups, the IEEE and Network addresses should be retrieved from the coordinator as that is the definitive source of truth. For a ZHA migration, you only need to grab the IEEE and friendly name mapping.

I believe all the relevant state should also be queried from the device.

Most importantly, you have to batch the interviews otherwise you are likely to crash the network/coordinator. In my testing a maximum of 3/4 concurrent interviews is safe.

[JayGoldberg]

Does Z2H mean ZHA or Z2M? I think this was a typo of disabling ZHA, since the next step is to copy the ZHA zigbee.db.

[seidtgeist]

Z2M, thanks for catching this! We want it to be stopped before editing the database on disk.

[JayGoldberg]

I was able to download zigbee.db on a HAOS OVA install by using the File editor addon.

Then dumping some attributes from it there are quite a few tables and they are versioned:

$ echo ".schema" | sqlite3 zigbee.db |grep "CREATE TABLE"
...
CREATE TABLE sqlite_sequence(name,seq);
CREATE TABLE devices_v13 (
CREATE TABLE endpoints_v13 (
CREATE TABLE clusters_v13 (
CREATE TABLE attributes_cache_v13 (
CREATE TABLE neighbors_v13 (
CREATE TABLE routes_v13 (
CREATE TABLE node_descriptors_v13 (
CREATE TABLE groups_v13 (
CREATE TABLE group_members_v13 (
CREATE TABLE relays_v13 (
CREATE TABLE unsupported_attributes_v13 (
CREATE TABLE network_backups_v13 (

I think we're only interested in devices

$ sqlite3 -header -column zigbee.db "SELECT * FROM devices_v13;"
ieee                     nwk    status  last_seen       
-----------------------  -----  ------  ----------------
00:12:4b:00:25:8c:2e:01  0      2       1731087968.03853
a4:c1:38:ff:f0:68:ee:84  36015  2       1727419863.37843
a4:c1:38:99:1f:8f:b1:74  38073  2       1731085873.37684
28:2c:02:bf:ff:ed:7f:50  19380  2       1731088878.55636
28:2c:02:bf:ff:ed:ff:39  36867  2       1731088891.67052
28:2c:02:bf:ff:ed:3a:ff  54497  2       1731088883.11546
a4:c1:38:03:8b:85:e2:fa  9725   2       1726708778.38752
28:2c:02:bf:ff:ed:a7:ab  38581  2       1731088855.53571
28:2c:02:bf:ff:e3:11:45  47669  2       1730996825.1379 
a4:c1:38:4e:36:1d:2c:90  17650  2       1731080608.45794
a4:c1:38:92:34:e5:72:f5  58464  2       1731084464.99849
28:2c:02:bf:ff:e8:fd:45  10249  2       1731087959.9287 
a4:c1:38:12:0d:3a:64:93  46338  2       1731088896.81854
a4:c1:38:a6:ff:b9:d3:b8  55145  2       1731084162.01582
a4:c1:38:1b:37:9f:a2:3d  53758  2       1731085470.95818

There's not a good way of exporting devices from HA and I don't have access to .storage. I'm looking at https://community.home-assistant.io/t/export-devices-and-entities/259465 to get the friendly names to build a basic JSON entry with ieee & nwk.

[koying]

Thanks for this.

Note that the ext_pan_id is in reverse order in the Z2M configuration vs. a ZHA backup for me (no clue who is right)

ZHA backup

    "network_info": {
        "extended_pan_id": "7e:51:d3:4f:64:50:5a:6e",

Z2M config

  ext_pan_id:
    - 110
    - 90
    - 80
    - 100
    - 79
    - 211
    - 81
    - 126

[Nerivec]

Big-endian, little-endian, it's not a matter of "right", but "right where it's used". Z2M adapts it as necessary throughout the code, so does ZHA. 😉

[koying]

They might, but my coordinator (yellow) was not happy to see a different ext_pan_id. Something in the like of "was configured with", with the ZHA ext_pan_id in reverse order if I specify the ext_pan_id in config the same way as ZHA.

[Nerivec]

ZHA stores them one way, and Z2M the other, so you indeed need to reverse them when switching from one to the other, in the file that is. The various systems in Z2M usually take care of this conversion when importing a coordinator_backup.json into Z2M, but the process here is a bit more "hands-on", so some conversions have to be made manually.
Hopefully we can build in support for most of this soon 😉

[ofcaah]

ZHA stores them one way, and Z2M the other, so you indeed need to reverse them when switching from one to the other, in the file that is. The various systems in Z2M usually take care of this conversion when importing a coordinator_backup.json into Z2M, but the process here is a bit more "hands-on", so some conversions have to be made manually. Hopefully we can build in support for most of this soon 😉

Hey. First of all, thank you for all your contributions. I'm also in the "ZHA continues its progress down the drain, and I don't want my network to sink with the ship" boat. I was thinking about the "hands-on way" migration this weekend, but after reading your post, maybe I should postpone it for some time, if not only to make things easier for myself, then to provide a "it worked/it borked" datapoint on the less "hands-on" approach. Question is, in your estimation, how much longer would I have to suffer ZHA restarting itself/doing other unwanted things before even a beta-quality procedure is implemented? ;)

PS. Looking at things, I think implementing such procedure will result in influx of new Z2M users :>

[Nerivec]

ZHA currently does not follow the open coordinator backup format, which makes it hard to derive a proper logic from it (would have to walk every single version from the moment they deviated from the format to adapt to every change, a monstruous task).
It should be easier once the new format is put in place, Z2M can add support for it, and then migrating from Z2M<>ZHA should be simpler. The new format will also provide all the necessary information, whereas now, it would require parsing databases (same problem as above: past/future changes would have to be supported...).
No ETA can be given at this time.

[bobobo1618]

Tried following this but something is unhappy:

error: 	z2m: Error: network commissioning timed out - most likely network with the same panId or extendedPanId already exists nearby (Error: AREQ - ZDO - stateChangeInd after 60000ms
    at Object.start (/app/node_modules/zigbee-herdsman/src/utils/waitress.ts:59:23)
    at ZnpAdapterManager.beginCommissioning (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:370:31)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at ZnpAdapterManager.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:91:21)
    at ZStackAdapter.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:158:16)
    at Controller.start (/app/node_modules/zigbee-herdsman/src/controller/controller.ts:137:29)
    at Zigbee.start (/app/lib/zigbee.ts:69:27)
    at Controller.start (/app/lib/controller.ts:161:27)
    at start (/app/index.js:154:5))
    at ZnpAdapterManager.beginCommissioning (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:372:23)
    at ZnpAdapterManager.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:91:21)
    at ZStackAdapter.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:158:16)
    at Controller.start (/app/node_modules/zigbee-herdsman/src/controller/controller.ts:137:29)
    at Zigbee.start (/app/lib/zigbee.ts:69:27)
    at Controller.start (/app/lib/controller.ts:161:27)
    at start (/app/index.js:154:5)

A quick web search suggests that maybe the problem is that there are other routers around that are still running the network and that's causing some kind of conflict?

[bobobo1618]

Figured out the issue was the missing coordinator backup. Adding it got that error to go away.