Creating a new pipeline for all commands that result in "Sorry I couldn't understand that"

[jazzmonger]

So in thinking through this new Willow eco system now that I generally understand the basics of WIS, ha Voice assistants, etc, many (over 50%) of the voice commands I use every day aren't implemented in HA via its Assistant. As a stop gap, is there a way to intercept (or know programmatically) when a command sent to HA via WIS fails with "Sorry I couldn't understand that" And then perform another action?

Use cases (there are dozens, especially for integrations that only Alexa supports)
Alexa play x on y (Pandora and Apple music especially, not supported on Music Assistant with HA Supervised installs)
Alexa set a timer called t on z
Alexa what's the name of that song
Alexa add carrots to my Walmart list. (AnyList integration)
Alexa how many ounces in 1 cup
Alexa what is 2050 divided by pie
Alexa what is a recipe to substitute buttermilk

And on and on an on and on. Dozens (hundreds) more...

On such a failure (better yet, a new Trigger in an automation), I would send the same STT result from WIS on to Amazon's Alexa using this service:

This would allow the commands that HA understands to execute within the HA system, and pass the rest of them that fail on to Alexa to do something with. I know the whole point is to replace these nasty, cloud based spy boxes, but we just aren't there yet.

I would change Alexas wake word to Echo, and shim the Box3/WIS solution squarely between HA and Amazon's Alexa and slowly send fewer and fewer requests on to Amazon as time goes by and HA matures in the Voice arena.

Using the Alexa custom command service I've noticed that since you bypass the Amazon wake word and Amazons spotty STT, the result is pretty snappy.

How might I do this?

Jeff

[kovrom]

If you are willing to use WAC you can do it now.
https://github.com/kovrom/willow-autocorrect

[jazzmonger]

Absolutely brilliant! I'm 1/2 way there installing WAC, but aging parents demanded I abandon that effort, for a few days anyway. Looks like a perfect solution!

[jazzmonger]

Wait a sec... is this in your fork or the main branch? Are you romanKO on Discord?

Jeff

[kovrom]

It's in my fork. And yes, that would be me.

[jazzmonger]

Ok, I just figured that out. This is like a game changer. Now I want to drive the 10 hours to get home so I can play with this, but... I'll have to wait until after Xmas! How do your changes get folded back into the main branch or do they? I assume this is something everyone with Alexa or Google will absolutely want... at least initially.

so... does this work well? What kind of delay is there to ultimately fire off the automation after it's triggered?

[kovrom]

I see it as a stopgap, until HA intents are matured and local AI is more affordable hardware-wise. Trigger time on WAC side about 80ms on average. And then it is however long it takes your amazon to respond, I would say maybe 4 sec total.
Also, Merry Xmas :)

[jazzmonger]

I see it as a stopgap, until HA intents are matured

EXACTLY!! Which is the same as how I envisioned it. It's sort of the best of both worlds. As time goes on, fewer and fewer phrases would get forwarded to the Evil Empires and processed locally.

so, see if they can include this in the main branch so we can keep up with their updated releases!

you do realize that the combination of all of this completely leapfrogs ALL of the HA voice stuff by like a year... or more.

[mikey60]

Here is a brute force method that I am using. It typically responds within one or two seconds for me when playing local media and internet stations to the Music Assistant player. I created multiple sentence automations. One for each area player. I have also created a sentence automation that changes the music that is currently playing. My sentence automation for the four season porch is shown below. The first condition of the automation is a special case for SiriusXM and and Pandora which are played on the basement Yamaha receiver, then re-sampled and sent out with an Icecast streaming server (This takes a longer time.) The second sequence goes to the Music Assistant player in the four season porch: 'media_player.four_season_porch_2'. 'media_player.four_season_porch_2' is an audio Chromecast with its output connected to a small stereo amplifier with two speakers that is controlled by Tasmota WiFi smart plug. The playlists, tracks and artists are played from my local Music Assistant media and the radio stations I have specifically created and named in the Music Assistant Radio. Sorry, the formatting of the templates is ugly because the automation was edited in the Home Assistant Automation UI.

description: ""
trigger:
  - platform: conversation
    command:
      - Play [the] {name} in [the] four season[s] porch[s]
      - Play [the] {name} in [the] four-season[s] porch[s]
      - Play [the] {name} in [the] porch[s]
      - Play [the] {name} in [the] four season[s]
action:
  - choose:
      - conditions:
          - condition: template
            value_template: >
              {{ trigger.slots.name is search('pandora', ignorecase=True) or
              trigger.slots.name is search('sirius', ignorecase=True) }}
        sequence:
          - service: input_select.select_option
            data:
              entity_id: input_select.music_selection
              option: >-
                {%- if trigger.slots.name is search('pandora', ignorecase=True)
                %} Pandora {%- elif  trigger.slots.name is search('sirius',
                ignorecase=True) %} SiriusXM  {%- else %} Pandora {% endif %}
          - service: input_select.select_option
            data:
              entity_id: input_select.music_to_play
              option: Porch Music
          - service: script.play_music
            data: {}
      - conditions:
          - condition: template
            value_template: |
              {{ states('media_player.basement_zone2') != 'playing' }}
        sequence:
          - service: media_player.play_media
            target:
              entity_id: media_player.four_season_porch_2
            data:
              media_content_id: >-
                {%- if trigger.slots.name is search('radio', ignorecase=True) %}
                {{trigger.slots.name}}   {%- elif trigger.slots.name is
                search('artist', ignorecase=True) %}
                {{trigger.slots.name|regex_replace(find='artist', replace='',
                ignorecase=True)}}   {%- elif trigger.slots.name is
                search('playlist', ignorecase=True) %}
                {{trigger.slots.name|regex_replace(find='playlist', replace='',
                ignorecase=True)}}   {%- elif trigger.slots.name is
                search('track', ignorecase=True) %}
                {{trigger.slots.name|regex_replace(find='track',
                replace='',ignorecase=True)}}   {%- endif %}
              media_content_type: >-
                {%- if trigger.slots.name is search('radio', ignorecase=True) %}
                radio  {%- elif trigger.slots.name is search('playlist',
                ignorecase=True) %} playlist {%- elif trigger.slots.name is
                search('track', ignorecase=True) %} track {%- elif
                trigger.slots.name is search('artist', ignorecase=True) %}
                artist  {%- endif %}
              enqueue: replace
          - service: switch.turn_on
            target:
              entity_id: switch.four_season_amp
            data: {}
mode: single

[janstadt]

thanks for the reply. i have lots of speakers exposed to HA/MASS so i will probs hold off on a custom automation to do this all and hope things get better. I've been working with the MASS OpenAI integration using home-llm or ollama to connect to a local llm instance and its working alright. still a bit confused as to how it all ties together though.

[mikey60]

I did some experimenting with MASS OpenAI integration using home-llm with ollama and the llama3:8b model with no success. I will have to spend some quality time with the approach later and see if I can get it working.

[janstadt]

Im having luck with gemma 7b and the one “home” something something recommended/tested in the home-llm GitHub repo.

[mikey60]

Thanks for the info. I will have to give it a try again in the near future.