1_create_luis.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: LUIS App\n",
    "\n",
    "**Content**\n",
    "\n",
    "* Create Luis App\n",
    "* Add Intents\n",
    "* Add Entities\n",
    "* Add Utterances\n",
    "* Train the LUIS model\n",
    "* Publish the LUIS model\n",
    "* Test\n"
   ]
  },
  {
   "source": [
    "References:\n",
    "* https://docs.microsoft.com/en-us/azure/cognitive-services/luis/client-libraries-rest-api?tabs=windows&pivots=programming-language-python\n",
    "* https://docs.microsoft.com/en-us/azure/cognitive-services/LUIS/luis-concept-entity-types\n",
    "* https://docs.microsoft.com/en-us/python/api/azure-cognitiveservices-language-luis/azure.cognitiveservices.language.luis.authoring.operations.modeloperations?view=azure-python\n",
    "* https://github.com/Azure-Samples/cognitive-services-quickstart-code/blob/master/python/LUIS/python-sdk-authoring-prediction/application_quickstart.py"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Requirement already satisfied: azure-cognitiveservices-language-luis in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (0.7.0)\n",
      "Requirement already satisfied: azure-common~=1.1 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from azure-cognitiveservices-language-luis) (1.1.26)\n",
      "Requirement already satisfied: msrestazure<2.0.0,>=0.4.32 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from azure-cognitiveservices-language-luis) (0.6.4)\n",
      "Requirement already satisfied: msrest>=0.5.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from azure-cognitiveservices-language-luis) (0.6.21)\n",
      "Requirement already satisfied: adal<2.0.0,>=0.6.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from msrestazure<2.0.0,>=0.4.32->azure-cognitiveservices-language-luis) (1.2.6)\n",
      "Requirement already satisfied: six in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from msrestazure<2.0.0,>=0.4.32->azure-cognitiveservices-language-luis) (1.15.0)\n",
      "Requirement already satisfied: isodate>=0.6.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from msrest>=0.5.0->azure-cognitiveservices-language-luis) (0.6.0)\n",
      "Requirement already satisfied: requests-oauthlib>=0.5.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from msrest>=0.5.0->azure-cognitiveservices-language-luis) (1.3.0)\n",
      "Requirement already satisfied: requests~=2.16 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from msrest>=0.5.0->azure-cognitiveservices-language-luis) (2.24.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from msrest>=0.5.0->azure-cognitiveservices-language-luis) (2020.6.20)\n",
      "Requirement already satisfied: PyJWT<3,>=1.0.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.32->azure-cognitiveservices-language-luis) (2.0.1)\n",
      "Requirement already satisfied: cryptography>=1.1.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.32->azure-cognitiveservices-language-luis) (3.1.1)\n",
      "Requirement already satisfied: python-dateutil<3,>=2.1.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.32->azure-cognitiveservices-language-luis) (2.8.1)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from requests-oauthlib>=0.5.0->msrest>=0.5.0->azure-cognitiveservices-language-luis) (3.1.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from requests~=2.16->msrest>=0.5.0->azure-cognitiveservices-language-luis) (2.10)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from requests~=2.16->msrest>=0.5.0->azure-cognitiveservices-language-luis) (1.25.11)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from requests~=2.16->msrest>=0.5.0->azure-cognitiveservices-language-luis) (3.0.4)\n",
      "Requirement already satisfied: cffi!=1.11.3,>=1.8 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from cryptography>=1.1.0->adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.32->azure-cognitiveservices-language-luis) (1.14.3)\n",
      "Requirement already satisfied: pycparser in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.1.0->adal<2.0.0,>=0.6.0->msrestazure<2.0.0,>=0.4.32->azure-cognitiveservices-language-luis) (2.20)\n",
      "Requirement already satisfied: ipykernel in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (5.3.4)\n",
      "Requirement already satisfied: jupyter-client in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipykernel) (6.1.7)\n",
      "Requirement already satisfied: traitlets>=4.1.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipykernel) (5.0.5)\n",
      "Requirement already satisfied: ipython>=5.0.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipykernel) (7.19.0)\n",
      "Requirement already satisfied: tornado>=4.2 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipykernel) (6.0.4)\n",
      "Requirement already satisfied: jupyter-core>=4.6.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from jupyter-client->ipykernel) (4.6.3)\n",
      "Requirement already satisfied: pyzmq>=13 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from jupyter-client->ipykernel) (19.0.2)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from jupyter-client->ipykernel) (2.8.1)\n",
      "Requirement already satisfied: ipython-genutils in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from traitlets>=4.1.0->ipykernel) (0.2.0)\n",
      "Requirement already satisfied: setuptools>=18.5 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel) (50.3.1.post20201107)\n",
      "Requirement already satisfied: pickleshare in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel) (0.7.5)\n",
      "Requirement already satisfied: backcall in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel) (0.2.0)\n",
      "Requirement already satisfied: colorama; sys_platform == \"win32\" in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel) (0.4.4)\n",
      "Requirement already satisfied: jedi>=0.10 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel) (0.17.1)\n",
      "Requirement already satisfied: pygments in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel) (2.7.2)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel) (3.0.8)\n",
      "Requirement already satisfied: decorator in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from ipython>=5.0.0->ipykernel) (4.4.2)\n",
      "Requirement already satisfied: pywin32>=1.0; sys_platform == \"win32\" in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from jupyter-core>=4.6.0->jupyter-client->ipykernel) (227)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from python-dateutil>=2.1->jupyter-client->ipykernel) (1.15.0)\n",
      "Requirement already satisfied: parso<0.8.0,>=0.7.0 in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in c:\\users\\anacarolines\\anaconda3\\lib\\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel) (0.2.5)\n"
     ]
    }
   ],
   "source": [
    "! pip install azure-cognitiveservices-language-luis\n",
    "! pip install ipykernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.cognitiveservices.language.luis.authoring import LUISAuthoringClient\n",
    "from msrest.authentication import CognitiveServicesCredentials\n",
    "import math\n",
    "import datetime, json, os, time\n",
    "import yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Load Luis Configs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_file = os.path.join(\"config\",\"config.yaml\")\n",
    "with open(config_file, 'r') as ymlfile:\n",
    "    config = yaml.load(ymlfile, Loader=yaml.FullLoader)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "authoring_key = config['luis_authoring']['auth_key']\n",
    "authoring_name = config['luis_authoring']['auth_name']\n",
    "authoring_endpoint = 'https://{}.cognitiveservices.azure.com/'.format(authoring_name)\n",
    "authoring_region = config['luis_authoring']['auth_region']\n",
    "app_name    = config['luis_app']['app_name']\n",
    "app_desc    = config['luis_app']['app_desc']\n",
    "app_version = str(config['luis_app']['app_version'])\n",
    "app_locale  = config['luis_app']['app_locale']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Luis App"
   ]
  },
  {
   "source": [
    "# Instantiate a LUIS client\n",
    "client = LUISAuthoringClient(authoring_endpoint, CognitiveServicesCredentials(authoring_key))\n"
   ],
   "cell_type": "code",
   "metadata": {},
   "execution_count": 118,
   "outputs": []
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_id = None\n",
    "\n",
    "def create_app(app_name,app_desc, app_version, app_locale):\n",
    "    # Create a new LUIS app\n",
    "\n",
    "\n",
    "    app_id = client.apps.add(dict(name=app_name,\n",
    "                                initial_version_id=app_version,\n",
    "                                description=app_desc,\n",
    "                                culture=app_locale))\n",
    "\n",
    "    print(\"Created LUIS app {}\\n    with ID {}\".format(app_name, app_id))\n",
    "    return app_id, app_version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Created LUIS app luis-seguradora-app\n    with ID 0469d26f-0302-4fb2-9b69-9828720ad816\n"
     ]
    }
   ],
   "source": [
    "app_id, app_version = create_app(app_name,app_desc, app_version, app_locale)"
   ]
  },
  {
   "source": [
    "> **Important**: Fill the *app_id* at *config/config.yml* file:\n",
    "```yaml\n",
    " luis_app:\n",
    "  app_id: <----\n",
    "  ```"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "0469d26f-0302-4fb2-9b69-9828720ad816\n"
     ]
    }
   ],
   "source": [
    "# Carregando o app id caso precise rodar o notebook a partir desse passo\n",
    "with open(config_file, 'r') as ymlfile:\n",
    "    config = yaml.load(ymlfile, Loader=yaml.FullLoader)\n",
    "app_id = config['luis_app']['app_id']\n",
    "print(app_id)"
   ]
  },
  {
   "source": [
    "### Check if the app was created at Luis Portal\n",
    "* Access [Luis Portal](https://www.luis.ai/)\n",
    "\n",
    "<img src=\"media/luis-app.png\" alt=\"Create Azure Account\"\n",
    "\ttitle=\"Create Azure Account\" width=\"50%\" height=\"50%\" />\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add Intents\n",
    "\n",
    "Example:\n",
    "* O número do protocolo é **456748674687**\n",
    "    * intent: *dizer_protocolo*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_intents(app_id, app_version, intents):\n",
    "    for intent in intents:\n",
    "        try:\n",
    "            intentId = client.model.add_intent(app_id, app_version, intent)\n",
    "            print(\"Intent {} {} added.\".format(intent, intentId))  \n",
    "            \n",
    "        except Exception as e:\n",
    "            if e.__class__.__name__ == 'ErrorResponseException':\n",
    "                print('Failed to insert intent %s. Reason: Intent already exists!' % (intent))  \n",
    "            else:\n",
    "                print(e.message)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "intents =  ['classe_roubo','classe_colisao','classe_furto','classe_quebra_retrovisores',\n",
    "            'dizer_protocolo']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Intent classe_roubo 3328a8cd-19bf-41ee-8653-37cf033f4426 added.\n",
      "Intent classe_colisao 4b553210-9687-4283-a3f4-0c003ee06625 added.\n",
      "Intent classe_furto 81d5e459-c625-4663-ac5e-e8f67b58ec53 added.\n",
      "Intent classe_quebra_retrovisores 911527fe-6298-4855-8223-10c4eccbd7f7 added.\n",
      "Intent dizer_protocolo 88186937-836d-46e4-9eb6-c42e983d4633 added.\n"
     ]
    }
   ],
   "source": [
    "add_intents(app_id, app_version, intents)"
   ]
  },
  {
   "source": [
    "### Check if the intents was created at Luis Portal\n",
    "* Access [Luis Portal](https://www.luis.ai/)\n",
    "* Click on your Luis app\n",
    "* Click on **Intents**\n",
    "\n",
    "<img src=\"media/luis-intents.png\" alt=\"Create Azure Account\"\n",
    "\ttitle=\"Create Azure Account\" width=\"50%\" height=\"50%\" />\n",
    "\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Add Entities\n",
    "\n",
    "Example:\n",
    "* O número do protocolo é **456748674687**\n",
    "    * intent: *dizer_protocolo*\n",
    "    * entity:  **456748674687** (*protocolo*)"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_entities(app_id, app_version, entity_name, entity_type, regex_pattern=None):\n",
    "    '''entity_type: 'ml', 'prebuilt','regex' '''\n",
    "    try:\n",
    "        if (entity_type == 'ml'):\n",
    "            client.model.add_entity(app_id, app_version, name=entity_name)\n",
    "        elif (entity_type == 'prebuilt'):\n",
    "            client.model.add_prebuilt(app_id, app_version  , prebuilt_extractor_names=[entity_name])\n",
    "        elif (entity_type == 'regex'):\n",
    "            client.model.create_regex_entity_model(app_id, app_version, regex_pattern=regex_pattern, name=entity_name)\n",
    "        else:\n",
    "            print(\"Choose one of those entity_types: 'ml', 'prebuilt','regex'\")\n",
    "    except Exception as e:\n",
    "        if e.__class__.__name__ == 'ErrorResponseException':\n",
    "            print('Failed to insert entity %s. Reason: Entity already exists!' % (entity_name))  \n",
    "        else:\n",
    "            print(e)\n",
    "    else:\n",
    "        print (\"Entity {} successfully added.\".format(entity_name))\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "#entity_name, entity_type, regex_pattern\n",
    "entities = [\n",
    "    [\"email\", \"prebuilt\"],\n",
    "    [\"placa\", \"regex\", \"[A-Z|a-z]{3}\\s[0-9]{4}\"],\n",
    "    [\"protocolo\", \"ml\"],\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Entity email successfully added.\n",
      "Entity placa successfully added.\n",
      "Entity protocolo successfully added.\n"
     ]
    }
   ],
   "source": [
    "for e in entities:\n",
    "    reg = ''\n",
    "    if e[1] == \"regex\": reg = e[2]\n",
    "    add_entities(app_id, app_version, e[0],e[1], reg)"
   ]
  },
  {
   "source": [
    "### Check if the intents was created at Luis Portal\n",
    "* Access [Luis Portal](https://www.luis.ai/)\n",
    "* Click on your Luis app\n",
    "* Click on **Entities**\n",
    "\n",
    "<img src=\"media/luis-entity.png\" alt=\"Create Azure Account\"\n",
    "\ttitle=\"Create Azure Account\" width=\"50%\" height=\"50%\" />\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add Utterances\n",
    "\n",
    "\n",
    "Example with entity:\n",
    "* O número do protocolo é **456748674687**\n",
    "    * intent: *dizer_protocolo*\n",
    "    * entity:  **456748674687** (*protocolo*)\n",
    "    * utterance: O número do protocolo é 456748674687\n",
    "\n",
    "    ```['dizer_protocolo','o seu protocolo é 456748674687',[('protocolo','456748674687']]```\n",
    "\n",
    "\n",
    "Example with no entity:\n",
    "* Roubaram meu carro\n",
    "    * intent: *classe_roubo*\n",
    "    * utterance: Roubaram meu carro\n",
    "\n",
    "    ```['classe_roubo','roubaram meu carro']```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "utterances = [\n",
    "    ['classe_roubo','roubaram meu carro'],\n",
    "    ['classe_roubo','meu carro foi roubado'],\n",
    "    ['classe_roubo','assaltaram meu carro'],\n",
    "    ['classe_roubo','me roubaram e levaram meu carro'],\n",
    "    ['classe_roubo','meu carro foi levado no assalto'],\n",
    "    ['classe_furto','meu carro foi furtado'],\n",
    "    ['classe_furto','furtaram meu carro'],\n",
    "    ['classe_furto','meu carro sumiu'],\n",
    "    ['classe_furto','meu carro desapareceu'],\n",
    "    ['classe_furto','meu carro não estava mais lá'],\n",
    "    ['classe_quebra_retrovisores','meu retrovisor quebrou'],\n",
    "    ['classe_quebra_retrovisores','quebraram o retrovisor do meu carro'],\n",
    "    ['classe_quebra_retrovisores','meu retrovisor estourou'],\n",
    "    ['classe_quebra_retrovisores','quebrou o espelho do meu retrovisor'],\n",
    "    ['classe_quebra_retrovisores','pifou meu retrovisor'],\n",
    "    ['classe_colisao','bateram no meu carro'],\n",
    "    ['classe_colisao','bater no meu carro'],\n",
    "    ['classe_colisao','meu carro foi batido'],\n",
    "    ['classe_colisao','colidiram em meu carro'],\n",
    "    ['classe_colisao','houve uma colisão'],\n",
    "    ['classe_colisao','houve uma pancada em meu carro'],\n",
    "    ['None','Boa tarde'],\n",
    "    ['None','como posso te ajudar'],\n",
    "    ['None','eu preciso de algumas informações para registar o sinistro'],\n",
    "    ['None',' tudo bem'],\n",
    "    ['None','Poderia me informar a placa do seu carro'],\n",
    "    ['None','Qual seria seu  CPF'],\n",
    "    ['None','qual seria seu telefone com ddd'],\n",
    "    ['None','E em qual rua aconteceu o roubo'],\n",
    "    ['None','De qual cidade'],\n",
    "    ['None','Muito obrigada pelas informações, você poderia anotar o número de protocolo?'],\n",
    "    ['None','Irei encaminhar seu caso para a área responsável, e eles irão entrar em contato com vocÊ, posso te ajudar em mais alguma coisa'],\n",
    "    ['None','Boa tarde'],\n",
    "    ['dizer_protocolo','o número do protocolo é 123456789456 irei agora encaminhar o seu caso para área responsável',[('protocolo','123456789456')]],\n",
    "    ['dizer_protocolo','o seu protocolo é 123456789456 irei agora encaminhar o seu caso para área responsável',[('protocolo','123456789456')]],\n",
    "    ['dizer_protocolo','o protocolo é 12345-6789456 irei agora encaminhar o seu caso para área responsável',[('protocolo','12345-6789456')]],\n",
    "    ['dizer_protocolo','o número de protocolo é 54 04 5795-9221 irei agora encaminhar o seu caso para área responsável',[('protocolo','54 04 5795-9221')]],\n",
    "    ['dizer_protocolo','o número de protocolo é 5404 5795221 irei agora encaminhar o seu caso para área responsável',[('protocolo','5404 5795221')]],\n",
    "    ['dizer_protocolo','o número de protocolo é 54 04 5795-9221',[('protocolo','54 04 5795-9221')]]\n",
    "\n",
    "\n",
    "    \n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_utterance(utterances):\n",
    "    utterances_luis = list()\n",
    "    for i in utterances:    \n",
    "        entities = []\n",
    "        if len(i) > 2:\n",
    "            for e in i[2]:\n",
    "                start = i[1].index(e[1])\n",
    "                entities.append(dict(entity_name=e[0], \n",
    "                                     start_char_index=start,\n",
    "                                     end_char_index=start + len(e[1])))\n",
    "        utterances_luis.append({'text': i[1].lower(), \n",
    "                                'intent_name' : i[0], \n",
    "                                'entity_labels': entities})\n",
    "    return utterances_luis\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[{'text': 'roubaram meu carro',\n",
       "  'intent_name': 'classe_roubo',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu carro foi roubado',\n",
       "  'intent_name': 'classe_roubo',\n",
       "  'entity_labels': []},\n",
       " {'text': 'assaltaram meu carro',\n",
       "  'intent_name': 'classe_roubo',\n",
       "  'entity_labels': []},\n",
       " {'text': 'me roubaram e levaram meu carro',\n",
       "  'intent_name': 'classe_roubo',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu carro foi levado no assalto',\n",
       "  'intent_name': 'classe_roubo',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu carro foi furtado',\n",
       "  'intent_name': 'classe_furto',\n",
       "  'entity_labels': []},\n",
       " {'text': 'furtaram meu carro',\n",
       "  'intent_name': 'classe_furto',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu carro sumiu',\n",
       "  'intent_name': 'classe_furto',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu carro desapareceu',\n",
       "  'intent_name': 'classe_furto',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu carro não estava mais lá',\n",
       "  'intent_name': 'classe_furto',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu retrovisor quebrou',\n",
       "  'intent_name': 'classe_quebra_retrovisores',\n",
       "  'entity_labels': []},\n",
       " {'text': 'quebraram o retrovisor do meu carro',\n",
       "  'intent_name': 'classe_quebra_retrovisores',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu retrovisor estourou',\n",
       "  'intent_name': 'classe_quebra_retrovisores',\n",
       "  'entity_labels': []},\n",
       " {'text': 'quebrou o espelho do meu retrovisor',\n",
       "  'intent_name': 'classe_quebra_retrovisores',\n",
       "  'entity_labels': []},\n",
       " {'text': 'pifou meu retrovisor',\n",
       "  'intent_name': 'classe_quebra_retrovisores',\n",
       "  'entity_labels': []},\n",
       " {'text': 'bateram no meu carro',\n",
       "  'intent_name': 'classe_colisao',\n",
       "  'entity_labels': []},\n",
       " {'text': 'bater no meu carro',\n",
       "  'intent_name': 'classe_colisao',\n",
       "  'entity_labels': []},\n",
       " {'text': 'meu carro foi batido',\n",
       "  'intent_name': 'classe_colisao',\n",
       "  'entity_labels': []},\n",
       " {'text': 'colidiram em meu carro',\n",
       "  'intent_name': 'classe_colisao',\n",
       "  'entity_labels': []},\n",
       " {'text': 'houve uma colisão',\n",
       "  'intent_name': 'classe_colisao',\n",
       "  'entity_labels': []},\n",
       " {'text': 'houve uma pancada em meu carro',\n",
       "  'intent_name': 'classe_colisao',\n",
       "  'entity_labels': []},\n",
       " {'text': 'boa tarde', 'intent_name': 'None', 'entity_labels': []},\n",
       " {'text': 'como posso te ajudar', 'intent_name': 'None', 'entity_labels': []},\n",
       " {'text': 'eu preciso de algumas informações para registar o sinistro',\n",
       "  'intent_name': 'None',\n",
       "  'entity_labels': []},\n",
       " {'text': ' tudo bem', 'intent_name': 'None', 'entity_labels': []},\n",
       " {'text': 'poderia me informar a placa do seu carro',\n",
       "  'intent_name': 'None',\n",
       "  'entity_labels': []},\n",
       " {'text': 'qual seria seu  cpf', 'intent_name': 'None', 'entity_labels': []},\n",
       " {'text': 'qual seria seu telefone com ddd',\n",
       "  'intent_name': 'None',\n",
       "  'entity_labels': []},\n",
       " {'text': 'e em qual rua aconteceu o roubo',\n",
       "  'intent_name': 'None',\n",
       "  'entity_labels': []},\n",
       " {'text': 'de qual cidade', 'intent_name': 'None', 'entity_labels': []},\n",
       " {'text': 'muito obrigada pelas informações, você poderia anotar o número de protocolo?',\n",
       "  'intent_name': 'None',\n",
       "  'entity_labels': []},\n",
       " {'text': 'irei encaminhar seu caso para a área responsável, e eles irão entrar em contato com você, posso te ajudar em mais alguma coisa',\n",
       "  'intent_name': 'None',\n",
       "  'entity_labels': []},\n",
       " {'text': 'boa tarde', 'intent_name': 'None', 'entity_labels': []},\n",
       " {'text': 'o número do protocolo é 123456789456 irei agora encaminhar o seu caso para área responsável',\n",
       "  'intent_name': 'dizer_protocolo',\n",
       "  'entity_labels': [{'entity_name': 'protocolo',\n",
       "    'start_char_index': 24,\n",
       "    'end_char_index': 36}]},\n",
       " {'text': 'o seu protocolo é 123456789456 irei agora encaminhar o seu caso para área responsável',\n",
       "  'intent_name': 'dizer_protocolo',\n",
       "  'entity_labels': [{'entity_name': 'protocolo',\n",
       "    'start_char_index': 18,\n",
       "    'end_char_index': 30}]},\n",
       " {'text': 'o protocolo é 12345-6789456 irei agora encaminhar o seu caso para área responsável',\n",
       "  'intent_name': 'dizer_protocolo',\n",
       "  'entity_labels': [{'entity_name': 'protocolo',\n",
       "    'start_char_index': 14,\n",
       "    'end_char_index': 27}]},\n",
       " {'text': 'o número de protocolo é 54 04 5795-9221 irei agora encaminhar o seu caso para área responsável',\n",
       "  'intent_name': 'dizer_protocolo',\n",
       "  'entity_labels': [{'entity_name': 'protocolo',\n",
       "    'start_char_index': 24,\n",
       "    'end_char_index': 39}]},\n",
       " {'text': 'o número de protocolo é 5404 5795221 irei agora encaminhar o seu caso para área responsável',\n",
       "  'intent_name': 'dizer_protocolo',\n",
       "  'entity_labels': [{'entity_name': 'protocolo',\n",
       "    'start_char_index': 24,\n",
       "    'end_char_index': 36}]},\n",
       " {'text': 'o número de protocolo é 54 04 5795-9221',\n",
       "  'intent_name': 'dizer_protocolo',\n",
       "  'entity_labels': [{'entity_name': 'protocolo',\n",
       "    'start_char_index': 24,\n",
       "    'end_char_index': 39}]}]"
      ]
     },
     "metadata": {},
     "execution_count": 130
    }
   ],
   "source": [
    "create_utterance(utterances)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "#break the list of utterances because there are a 100 utterances limit batch\n",
    "qnt_quebras = (math.ceil(len(utterances)/100))\n",
    "qnt_por_quebras = round(len(utterances)/qnt_quebras)\n",
    "\n",
    "inicio = 0\n",
    "proximo = 0\n",
    "list_quebras = list(range(1,qnt_quebras+1))\n",
    "broken_utterances = list()\n",
    "for i in list_quebras:\n",
    "    if (i == 1):\n",
    "        proximo = qnt_por_quebras\n",
    "        ut = utterances[0:proximo]\n",
    "        broken_utterances.append(ut)\n",
    "    #a partir do segunda quebra\n",
    "    else:\n",
    "        inicio = proximo\n",
    "        if (i < qnt_quebras):\n",
    "            proximo = inicio + qnt_por_quebras\n",
    "        #última quebra\n",
    "        else:\n",
    "            proximo = len(utterances)\n",
    "        ut = utterances[inicio:proximo]\n",
    "        broken_utterances.append(ut)\n",
    "    next = qnt_por_quebras + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "39 example utterance(s) added.\n"
     ]
    }
   ],
   "source": [
    "for u in broken_utterances:\n",
    "    client.examples.batch(app_id, app_version, create_utterance(u))\n",
    "    print(\"{} example utterance(s) added.\".format(len(u)))"
   ]
  },
  {
   "source": [
    "### Check if the utterances was created at Luis Portal\n",
    "* Access [Luis Portal](https://www.luis.ai/)\n",
    "* Click on your Luis app\n",
    "* Click on **Intents** and check the column **Examples**\n",
    "\n",
    "<img src=\"media/luis-utterances.png\" alt=\"Check utterances\"\n",
    "\ttitle=\"Check utterance\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "### Check an utterance with entities\n",
    "* Select some intent with entity\n",
    "\n",
    "<img src=\"media/luis-intent-utterance-entity.png\" alt=\"Check an utterance with entities\"\n",
    "\ttitle=\"Check an utterance with entities\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "### Check an utterance with no entities\n",
    "* Select some intent with no entity\n",
    "\n",
    "<img src=\"media/luis-intent-utterance.png\" alt=\"Check an utterance with no entities\"\n",
    "\ttitle=\"Check an utterance with no entities\" width=\"50%\" height=\"50%\" />\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the LUIS model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_app(app_id, app_version):\n",
    "    response = client.train.train_version(app_id, app_version)\n",
    "    waiting = True\n",
    "    while waiting:\n",
    "        info = client.train.get_status(app_id, app_version)\n",
    "\n",
    "        # get_status returns a list of training statuses, one for each model. Loop through them and make sure all are done.\n",
    "        waiting = any(map(lambda x: 'Queued' == x.details.status or 'InProgress' == x.details.status, info))\n",
    "        if waiting:\n",
    "            print (\"Waiting 10 seconds for training to complete...\")\n",
    "            time.sleep(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Waiting 10 seconds for training to complete...\n"
     ]
    }
   ],
   "source": [
    "train_app(app_id,app_version)"
   ]
  },
  {
   "source": [
    "### Test at Luis Portal \\[Alternative]\n",
    "You can also publish on [Luis Portal](https://www.luis.ai/):\n",
    "* Click on your Luis app\n",
    "* Click on **Test**\n",
    "\n",
    "<img src=\"media/luis-test.png\" alt=\"\"\n",
    "\ttitle=\"\" width=\"50%\" height=\"50%\" />"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publish the LUIS model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "def publish_app(app_id, app_version):\n",
    "    responseEndpointInfo = client.apps.publish(app_id, app_version, is_staging=False,\n",
    "                                                   speech_priming = True, sentiment_analysis = False)\n",
    "    print(\"Application published. Endpoint URL: \" + responseEndpointInfo.endpoint_url)\n",
    "    return responseEndpointInfo.endpoint_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Application published. Endpoint URL: https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/0469d26f-0302-4fb2-9b69-9828720ad816\n"
     ]
    }
   ],
   "source": [
    "endpoint = publish_app(app_id, app_version)"
   ]
  },
  {
   "source": [
    "### Publish at Luis Portal \\[Alternative]\n",
    "You can also publish on [Luis Portal](https://www.luis.ai/):\n",
    "* Click on your Luis app\n",
    "* Click on **Publish**\n",
    "\n",
    "<img src=\"media/luis-publish.png\" alt=\"\"\n",
    "\ttitle=\"\" width=\"50%\" height=\"50%\" />"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_intent(utterance):\n",
    "    import requests\n",
    "\n",
    "     # YOUR-APP-ID: The App ID GUID found on the www.luis.ai Application Settings page.\n",
    "    appId = app_id\n",
    "\n",
    "    # YOUR-PREDICTION-KEY: Your LUIS authoring key, 32 character value.\n",
    "    prediction_key = authoring_key\n",
    "\n",
    "    # YOUR-PREDICTION-ENDPOINT: Replace with your authoring key endpoint.\n",
    "    # For example, \"https://westus.api.cognitive.microsoft.com/\"\n",
    "    prediction_endpoint = \"https://{}.api.cognitive.microsoft.com/\".format(authoring_region)\n",
    "\n",
    "    # The utterance you want to use.\n",
    "    utterance = utterance\n",
    "    ##########\n",
    "\n",
    "    try:\n",
    "\n",
    "        # The headers to use in this REST call.\n",
    "        headers = {\n",
    "        }\n",
    "\n",
    "        # The URL parameters to use in this REST call.\n",
    "        params ={\n",
    "            'query': utterance,\n",
    "            'timezoneOffset': '0',\n",
    "            'verbose': 'true',\n",
    "            'show-all-intents': 'true',\n",
    "            'spellCheck': 'false',\n",
    "            'staging': 'false',\n",
    "            'subscription-key': authoring_key\n",
    "        }\n",
    "\n",
    "        # Make the REST call.\n",
    "        response = requests.get(f'{prediction_endpoint}luis/prediction/v3.0/apps/{appId}/slots/production/predict',   headers=headers, params=params)\n",
    "\n",
    "        # Display the results on the console.\n",
    "        print(response.json())\n",
    "\n",
    "\n",
    "    except Exception as e:\n",
    "        # Display the error string.\n",
    "        print(f'{e}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'query': 's plsvs GGE 4547', 'prediction': {'topIntent': 'None', 'intents': {'None': {'score': 0.7254737}, 'classe_colisao': {'score': 0.0137627907}, 'classe_furto': {'score': 0.0106957071}, 'classe_roubo': {'score': 0.009585956}, 'classe_quebra_retrovisores': {'score': 0.008993628}, 'dizer_protocolo': {'score': 0.00299739162}}, 'entities': {'placa': ['GGE 4547'], '$instance': {'placa': [{'type': 'placa', 'text': 'GGE 4547', 'startIndex': 8, 'length': 8, 'modelTypeId': 8, 'modelType': 'Regex Entity Extractor', 'recognitionSources': ['model']}]}}}}\n"
     ]
    }
   ],
   "source": [
    "get_intent(\"s plsvs GGE 4547\")"
   ]
  },
  {
   "source": [
    "### Test at Luis Portal \\[Alternative]\n",
    "You can also test on [Luis Portal](https://www.luis.ai/):\n",
    "* Click on your Luis app\n",
    "* Click on **Test**\n",
    "\n",
    "<img src=\"media/luis-test.png\" alt=\"Check an utterance with entities\"\n",
    "\ttitle=\"Check an utterance with entities\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "* Type the utterance to test\n",
    "\n",
    "<img src=\"media/luis-testing.png\" alt=\"Check an utterance with entities\"\n",
    "\ttitle=\"Check an utterance with entities\" width=\"20%\" />\n",
    "\n",
    "\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "Before testing with Speech Service for the next notebook go to [https://www.luis.ai/] and check the option below:\n",
    "\n",
    "\n",
    "<img src=\"media/luis-make-endpoints-public.png\" alt=\"Check an utterance with entities\"\n",
    "\ttitle=\"Check an utterance with entities\" width=\"50%\" />\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}