Merge pull request #8913 from RasaHQ/intent-ted

`UnexpecTEDIntentPolicy` final merge

changelog/8913.feature.md

@@ -0,0 +1,46 @@
+Introduces a new policy called [`UnexpecTEDIntentPolicy`](./policies.mdx#unexpected-intent-policy).
+
+`UnexpecTEDIntentPolicy` helps you [review conversations](./conversation-driven-development.mdx#review)
+ and also allows your bot to react to unexpected user turns in conversations.
+ It is an auxiliary policy that should only be used in conjunction with
+ at least one other policy, as the only action that it can trigger
+ is the special and newly introduced
+ [`action_unlikely_intent`](./default-actions.mdx#action_unlikely_intent) action.
+
+ The auto-configuration will include `UnexpecTEDIntentPolicy` in your
+ configuration automatically, but you can also include it yourself
+ in the `policies` section of the configuration:
+
+ ```
+ policies:
+   - name: UnexpecTEDIntentPolicy
+     epochs: 200
+     max_history: 5
+ ```
+
+As part of the feature, it also introduces:
+
+- [`IntentMaxHistoryTrackerFeaturizer`](./policies.mdx#3-intent-max-history)
+  to featurize the trackers for `UnexpecTEDIntentPolicy`.
+- `MultiLabelDotProductLoss` to support `UnexpecTEDIntentPolicy`'s multi-label training objective.
+- A new default action called [`action_unlikely_intent`](./default-actions.mdx#action_unlikely_intent).
+
+
+`rasa test` command has also been adapted to `UnexpecTEDIntentPolicy`:
+
+- If a test story contains `action_unlikely_intent` and the policy ensemble does not trigger it, this leads to
+  a test error (wrongly predicted action) and the corresponding story will be logged in `failed_test_stories.yml`.
+- If the story does not contain `action_unlikely_intent` and Rasa Open Source does predict it then
+  the prediction of `action_unlikely_intent` will be ignored for the evaluation (and hence not lead
+  to a prediction error) but the story will be logged in a file called `stories_with_warnings.yml`.
+
+
+The `rasa data validate` command will warn if `action_unlikely_intent` is
+included in the training stories. Accordingly, `YAMLStoryWriter` and `MarkdownStoryWriter` have been updated to not dump `action_unlikely_intent` when writing stories to a file.
+
+:::caution
+The introduction of a new default action **breaks backward compatibility of previously trained models**.
+It is not possible to load models trained with previous versions of Rasa Open Source. Please re-train
+your assistant before trying to use this version.
+
+:::

data/test_config/config_empty_en_after_dumping.yml

@@ -27,6 +27,9 @@ policies:
 # # See https://rasa.com/docs/rasa/policies for more information.
 #   - name: MemoizationPolicy
 #   - name: RulePolicy
+#   - name: UnexpecTEDIntentPolicy
+#     max_history: 5
+#     epochs: 100
 #   - name: TEDPolicy
 #     max_history: 5
 #     epochs: 100

data/test_config/config_empty_en_after_dumping_core.yml

@@ -6,6 +6,9 @@ policies:
 # # See https://rasa.com/docs/rasa/policies for more information.
 #   - name: MemoizationPolicy
 #   - name: RulePolicy
+#   - name: UnexpecTEDIntentPolicy
+#     max_history: 5
+#     epochs: 100
 #   - name: TEDPolicy
 #     max_history: 5
 #     epochs: 100

data/test_config/config_empty_fr_after_dumping.yml

@@ -27,6 +27,9 @@ policies:
 # # See https://rasa.com/docs/rasa/policies for more information.
 #   - name: MemoizationPolicy
 #   - name: RulePolicy
+#   - name: UnexpecTEDIntentPolicy
+#     max_history: 5
+#     epochs: 100
 #   - name: TEDPolicy
 #     max_history: 5
 #     epochs: 100

data/test_config/config_with_comment_between_suggestions.yml

@@ -25,6 +25,9 @@ policies:
 # # If you'd like to customize them, uncomment and adjust the policies.
 # # See https://rasa.com/docs/rasa/policies for more information.
 #   - name: MemoizationPolicy
+#   - name: UnexpecTEDIntentPolicy
+#     max_history: 5
+#     epochs: 100
 #   - name: TEDPolicy
 #     max_history: 5
 #     epochs: 100

data/test_config/config_with_comments_after_dumping.yml

@@ -25,6 +25,9 @@ policies: # even here
 # # See https://rasa.com/docs/rasa/policies for more information.
 #   - name: MemoizationPolicy
 #   - name: RulePolicy
+#   - name: UnexpecTEDIntentPolicy
+#     max_history: 5
+#     epochs: 100
 #   - name: TEDPolicy
 #     max_history: 5
 #     epochs: 100

data/test_moodbot/unexpected_intent_policy_config.yml

@@ -0,0 +1,14 @@
+language: en
+
+pipeline:
+
+policies:
+  - name: MemoizationPolicy
+  - name: UnexpecTEDIntentPolicy
+    max_history: 5
+    epochs: 1
+  - name: TEDPolicy
+    max_history: 5
+    epochs: 1
+    constrain_similarities: true
+  - name: RulePolicy

data/test_yaml_stories/stories_unexpected_intent_unlearnable.yml

@@ -0,0 +1,12 @@
+version: "2.0"
+stories:
+- story: "story 1"
+  steps:
+  - intent: greet
+  - action: action_unlikely_intent
+
+- story: "story 2"
+  steps:
+  - intent: greet
+  - action: utter_greet
+  - action: action_unlikely_intent

data/test_yaml_stories/test_multiple_action_unlikely_intent_warnings.yml

@@ -0,0 +1,28 @@
+version: "2.0"
+stories:
+  - story: path 1
+    steps:
+      - user: |
+          hello there!
+        intent: greet
+      - action: utter_greet
+      - user: |
+          amazing
+        intent: mood_great
+      - action: utter_happy
+
+  - story: path 2
+    steps:
+      - user: |
+          hello there!
+        intent: greet
+      - action: utter_greet
+      - user: |
+          very sad
+        intent: mood_unhappy
+      - action: utter_cheer_up
+      - action: utter_did_that_help
+      - user: |
+          yes
+        intent: affirm
+      - action: utter_happy

docs/docs/command-line-interface.mdx

@@ -143,6 +143,20 @@ rasa interactive
 
 This will first train a model and then start an interactive shell session.
 You can then correct your assistants predictions as you talk to it.
+If [`UnexpecTEDIntentPolicy`](./policies.mdx#unexpected-intent-policy) is
+included in the pipeline, [`action_unlikely_intent`](./default-actions.mdx#action_unlikely_intent)
+can be triggered at any conversation turn. Subsequently, the following message will be displayed:
+
+```
+ The bot wants to run 'action_unlikely_intent' to indicate that the last user message was unexpected
+ at this point in the conversation. Check out UnexpecTEDIntentPolicy docs to learn more.
+```
+
+ As the message states, this is an indication that you have explored a conversation path
+ which is unexpected according to the current set of training stories and hence adding this
+ path to training stories is recommended. Like other bot actions, you can choose to confirm
+ or deny running this action.
+
 
 If you provide a trained model using the `--model` argument, training is skipped
 and that model will be loaded instead.

docs/docs/conversation-driven-development.mdx

@@ -59,6 +59,23 @@ Your test users had at least some instruction about what the bot was intended to
 
 *   Look at conversations where an “out_of_scope” intent or fallback behavior occurred. These could indicate a potential new skill, or just a misclassified user utterance.
 *   Look for user frustration, such as requests for transfer to a human.
+*   If the assistant was trained with [`UnexpecTEDIntentPolicy`](./policies.mdx#unexpected-intent-policy) included in the pipeline,
+    you can look for conversations where `action_unlikely_intent` is predicted at any conversation turn.
+    An `action_unlikely_intent` is predicted when the last intent expressed by the user is
+    unexpected in the current conversation context. You can also filter out such conversations by
+    running a [standalone script](https://gist.github.com/alwx/b426b7b573ff963c85c65ea6466528d7) which does the following:
+
+    - Fetch real conversations from a tracker store.
+    - Run `rasa test` on the fetched conversations and filter conversations containing `action_unlikely_intent`
+      in a separate warnings file. You can read more on [how to interpret these warnings](./testing-your-assistant.mdx#interpreting-the-generated-warnings).
+
+    Reviewing this subset of conversations can help you understand if real users have taken a
+    conversation path which is not present in the training data and hence "surprising"
+    for machine learning policies like `TEDPolicy`. Adding these conversation paths (with potential
+    corrections if `TEDPolicy` subsequently failed) as training stories will result in more robust action prediction
+    by policies such as `TEDPolicy`. Users are encouraged to [adjust the `tolerance` parameter of
+    `UnexpecTEDIntentPolicy`](./policies.mdx#tuning-the-tolerance-parameter) to control how
+    "surprising" a conversation should be to be included in the warnings file.
 
 ### Annotate
 

docs/docs/default-actions.mdx

@@ -187,3 +187,53 @@ Overriding the default action for forms should **only** be used during the proce
 migrating from Rasa Open Source 1 to 2. In this case you can override the default
 action to instruct Rasa Open Source to use the deprecated `FormAction` which is part of
 the Rasa SDK.
+
+## action_unlikely_intent
+
+Rasa Open Source triggers `action_unlikely_intent` via [`UnexpecTEDIntentPolicy`](./policies.mdx#unexpected-intent-policy).
+You can control how often this action is predicted by tuning the [`tolerance`](./policies.mdx#unexpected-intent-policy)
+parameter of `UnexpecTEDIntentPolicy`.
+
+### Customization
+
+You can customize your assistant's behaviour to configure what should happen once `action_unlikely_intent`
+is triggered. For example, as a follow up you can trigger a hand-off to a human agent with a rule:
+
+```yaml
+- rule: trigger human handoff with action_unlikely_intent
+  steps:
+    - action: action_unlikely_intent
+    - action: ask_human_handoff
+    - intent: affirm
+    - action: trigger_human_handoff
+```
+
+Alternatively, you can also override it's behaviour as a [`custom action`](./custom-actions.mdx) by
+adding `action_unlikely_intent` to the list of actions in the domain and implementing the custom behaviour:
+
+```python
+class ActionUnlikelyIntent(Action):
+
+    def name(self) -> Text:
+        return "action_unlikely_intent"
+
+    async def run(
+        self, dispatcher, tracker: Tracker, domain: Dict[Text, Any],
+    ) -> List[Dict[Text, Any]]:
+
+        # Implement custom logic here
+        return []
+```
+
+:::note
+Since `action_unlikely_intent` can be triggered at any conversation step during inference,
+all policies which are trained on only story data, for example - `TEDPolicy`, `UnexpecTEDIntentPolicy`,
+`MemoizationPolicy` ignore its presence in the tracker when making a prediction. However, `RulePolicy`
+takes its presence into account so that [conversation behaviour is customizable](./default-actions.mdx#customization-1).
+
+:::
+
+:::note
+`action_unlikely_intent` cannot be included in the training stories. It can **only** be added to rules.
+
+:::

docs/docs/migration-guide.mdx

@@ -12,6 +12,33 @@ how you can migrate from one version to another.
 
 ## Rasa 2.7 to 2.8
 
+:::caution
+This release **breaks backward compatibility of machine learning models**.
+It is not possible to load models trained with previous versions of Rasa Open Source. Please re-train
+your assistant before using this version.
+
+:::
+
+### Deprecations
+
+#### Tracker Featurizers
+
+`training_states_actions_and_entities` method of `TrackerFeaturizer`, `FullDialogueTrackerFeaturizer` and
+`MaxHistoryTrackerFeaturizer` classes is deprecated and will be removed in Rasa Open Source 3.0 .
+If you had a custom tracker featurizer which relied on this method from any of the above classes, please use
+`training_states_labels_and_entities` instead.
+
+`training_states_actions` method of `TrackerFeaturizer`, `FullDialogueTrackerFeaturizer` and
+`MaxHistoryTrackerFeaturizer` classes is deprecated and will be removed in Rasa Open Source 3.0 .
+If you had a custom tracker featurizer which relied on this method from any of the above classes, please use
+`training_states_labels` instead.
+
+#### State Featurizer
+
+`encode_all_actions` method of `SingleStateFeaturizer` class is deprecated and will be removed in Rasa Open Source 3.0 .
+It is recommended to use the method `encode_all_labels` instead.
+
+
 ### Incremental Training
 
 Users don't need to specify an additional buffer size for sparse featurizers anymore during incremental training.