notebooks/04_Inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Location of Model Artifacts\n",
    "\n",
    "### Google Cloud Storage\n",
    "\n",
    "- **model for inference** (965 MB): `https://storage.googleapis.com/issue_label_bot/model/lang_model/models_22zkdqlr/trained_model_22zkdqlr.pkl`\n",
    "\n",
    "\n",
    "- **encoder (for fine-tuning w/a classifier)** (965 MB): \n",
    "`https://storage.googleapis.com/issue_label_bot/model/lang_model/models_22zkdqlr/trained_model_encoder_22zkdqlr.pth`\n",
    "\n",
    "\n",
    "- **fastai.databunch** (27.1 GB):\n",
    "`https://storage.googleapis.com/issue_label_bot/model/lang_model/data_save.pkl`\n",
    "\n",
    "\n",
    "- **checkpointed model** (2.29 GB): \n",
    "`https://storage.googleapis.com/issue_label_bot/model/lang_model/models_22zkdqlr/best_22zkdqlr.pth`\n",
    "\n",
    "### Weights & Biases Run\n",
    "\n",
    "`https://app.wandb.ai/github/issues_lang_model/runs/22zkdqlr/overview`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Load Full Model + DataBunch In Order To Save Model For Inference\n",
    "\n",
    "A fastai learner comes packaged with the training data and other data, however for inference we don't need this.  There is a way to export just the model weights without the data with `learn.export` or just the encoder base with `learn.save_encoder`.  Unfortunately, I forgot to do this during model training therefore we need to load the full checkpointed model and a databunch and save these artificacts for inference.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\"\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"1\"\n",
    "\n",
    "\n",
    "from pathlib import Path\n",
    "from fastai.basic_train import load_learner\n",
    "from fastai.text import load_data\n",
    "from fastai.text.learner import language_model_learner\n",
    "from fastai.text.models import AWD_LSTM, awd_lstm_lm_config\n",
    "\n",
    "emb_sz=800\n",
    "qrnn=False\n",
    "bidir=False\n",
    "n_layers=4\n",
    "n_hid=2400\n",
    "\n",
    "\n",
    "# https://app.wandb.ai/github/issues_lang_model/runs/22zkdqlr/overview\n",
    "data_path = Path('/ds/lang_model')\n",
    "model_path = data_path/'models_22zkdqlr'\n",
    "\n",
    "\n",
    "def pass_through(x):\n",
    "    return x\n",
    "\n",
    "\n",
    "awd_lstm_lm_config.update(dict(emb_sz=emb_sz, qrnn=qrnn, bidir=bidir, n_layers=n_layers, n_hid=n_hid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load learner object\n",
    "\n",
    "\n",
    "Note: you don't have to do this over and over again, you just have to call `learn.export()` to save the learner after you have loaded everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(data_path, bs=96)\n",
    "\n",
    "learn = language_model_learner(data=data_lm,\n",
    "                               arch=AWD_LSTM,\n",
    "                               model_dir=model_path,\n",
    "                               pretrained=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load weights of trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LanguageLearner(data=TextLMDataBunch;\n",
       "\n",
       "Train: LabelList (16762799 items)\n",
       "x: LMTextList\n",
       "xxbos xxxfldtitle xxmaj grab excerpt and image using xxmaj open xxmaj graph xxxfldbody xxmaj we can update the wordpress excerpt and featured image by using xxmaj open xxmaj graph to get that information . xxmaj idea comes from xxmaj dan xxmaj scott via e - mail .,xxbos xxxfldtitle xxmaj gracefully handling ctrl+c ignores xxmaj program xxmaj main exit code . xxxfldbody xxmaj repro : \n",
       "  xxxlistb dotnet new console xxmaj paste the repro code below into xxmaj program.cs dotnet run xxmaj hit xxmaj ctrl + c xxmaj look at the last exit code . xxmaj we expect in the below repo for the exit code to be 1337 xxmaj built a repro that 's inspired by xxup asp.net 's hosting bits here : https : xxxfilepath xxmaj repro : using xxmaj system ; using xxmaj system . xxmaj threading ; using xxmaj system . xxmaj threading . xxmaj tasks ; namespace xxunk xxxjson \n",
       "  / cc xxxatmention xxxatmention xxxliste,xxbos xxxfldtitle xxunk - xxmaj android not working within my app xxxfldbody xxmaj hi i 'm on xxmaj mac xxup os x 10.12.6 with xxmaj android xxmaj studio 2.3.3 and trying to write my own trace functionality so that i can see all of my app 's method calls with arguments on xxmaj android and stream them to my computer with xxxcdb adb logcat xxxcde \n",
       "  . xxmaj this trace will give me a broad view of the projects i work on to cut through callback hell in xxmaj android xxup api . i tried your example app at https : xxxfilepath xxxlnkhb github.com xxxlnkhe and it worked ( after many blind xxunk with xxunk , xxmaj frida , etc ) thank you for that ! i 've spent many days searching the web and as far as i can tell , your xxunk - xxmaj android library is the only one that provides a working example for catching all app method calls using aspectj on xxmaj android without having to annotate every method . i even managed to strip out the xxxcdb kotlin xxxcde \n",
       "  , xxxcdb groovy xxxcde \n",
       "  , xxxcdb proguard xxxcde \n",
       "  , xxxcdb dagger xxxcde \n",
       "  and xxxcdb xxunk xxxcde \n",
       "  stuff , so it simply has these aspectj hooks in xxxcdb xxmaj profiler xxxcde \n",
       "  and xxxcdb xxunk xxxcde \n",
       "  : before ( ) : xxunk ( ) xxxjson after ( ) : xxunk ( ) xxxjson i left xxxcdb xxunk xxxcde \n",
       "  in the project because i was n't sure if it was required for detecting anonymous classes ( which i use in place of anonymous methods for now ) . \n",
       "  xxxhm xxmaj however , a few things : xxmaj your xxunk never gets called if i create xxunk xxxfilepath with : \\ timport xxunk ; \\ timport xxunk \\ timport xxunk ; \\ timport xxunk \\ xxunk xxunk xxxjson \n",
       "  xxxhm xxmaj you can try it yourself ( i do n't know if it 's not detecting the xxxcdb xxunk ( ) xxxcde \n",
       "  etc ) . xxmaj which means i have no starting point for writing my own aspectj code . xxxhm xxmaj also , i 've spent 2 days so far trying to copy your xxxcdb build.gradle xxxcde \n",
       "  settings to my app and no matter how hard i try , it never calls my xxxcdb xxmaj profiler xxxcde \n",
       "  or xxxcdb xxunk xxxcde \n",
       "  . i 'm pretty sure i 've nested it properly in xxxcdb src xxxfilepath xxxcde \n",
       "  but i ca n't get it to work . xxmaj without a better understanding of how the xxunk call works , how aspectj is detecting the app 's methods , and how that gets called from gradle , there 's simply no further i can go with this . xxxhm i even tried building issue https : xxxfilepath xxxlnkhb github.com xxxlnkhe https : xxxfilepath xxxlnkhb github.com xxxlnkhe but no matter how hard i try , it no longer builds and runs as of xxmaj android xxmaj studio 2.3.3 . xxmaj if someone can fork that somewhere and get it working , that might give me a starting point . xxmaj so without a minimalist app containing an xxmaj empty xxmaj activity , xxmaj basic xxmaj activity ( or some other example from the xxmaj android xxmaj studio project stationary ) with the addition of your library and a single aspectj example that demonstrates detecting the app 's methods without annotations , developers are going to have a hard time integrating your work . xxmaj there are simply too many permutations to explore if it does n't work . i 'm unable to post my app to github , but do you have any suggestions i can try to show : \n",
       "  xxxlistb xxmaj that my xxmaj profiler aspectj code is compiling and running correctly ? a list of the app 's methods collected by aspectj , or proof that they are being detected ? xxmaj here is a minimal aspectj method autodetection example , forked from yours with the above steps applied https : xxxfilepath and even though it works , i ca n't get the same build.gradle settings to work within my app . xxmaj thanks in advance for any help you can provide ! \n",
       "  xxxliste,xxbos xxxfldtitle xxmaj graph : xxmaj components problem xxxfldbody xxmaj solution for xxmaj components problem in xxmaj java with sample test . xxmaj click for xxmaj problem xxmaj statement and test cases . xxxlnkhb github.com xxxlnkhe,xxbos xxxfldtitle xxmaj great xxmaj plains xxmaj seeder xxup xxunk xxxfldbody xxmaj mode 6 does not lower this implement on course xxunk can manually lower it and it drives just fine . xxmaj the turn maneuvers is raising and lowering xxunk not stop at field edge on turn finish ) . xxmaj could n't test the refill course xxmaj drive xxmaj now button does nothing . xxmaj no errors in log ) . xxmaj also if you leave the ridge markers on auto it spits out . \n",
       "  xxmaj error : xxmaj running xxup lua method ' update ' . datas xxxfilepath : attempt to index a nil value xxmaj and requires a game restart to clear . xxmaj this seeder has no ridge markers xxmaj debug channel 17 at pastebin xxxlnkhb pastebin.com xxxlnkhe xxmaj sorry for breaking xxup cp so quickly . i have this mod do just about everything for me in xxup fs . xxmaj keep up the good work xxup :d\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: /IssuesLanguageModel/model/lang_model;\n",
       "\n",
       "Valid: LabelList (1858033 items)\n",
       "x: LMTextList\n",
       "xxbos xxxfldtitle xxmaj google . xxmaj cloud . xxmaj translation . xxup v2 - xxmaj translate xxmaj html xxxfldbody xxxhs xxmaj environment details \n",
       "  xxxlistb xxup os : xxmaj windows xxup .net version : 4.5.2 xxmaj package name and version : xxmaj google . xxmaj cloud . xxmaj translation . xxup v2 xxrep 4 # xxmaj steps to reproduce xxmaj large xxmaj html snippet to translate xxmaj code xxunk gc = xxunk xxunk xxunk = xxunk . xxmaj xxunk , xxunk xxunk result = xxunk . xxunk , xxunk ( ) , xxunk xxmaj error [ xxunk : xxmaj invalid xxup uri : xxmaj the xxmaj uri string is too long . ] xxmaj system . xxmaj uri . xxunk uri , xxmaj boolean xxunk , xxunk xxunk ) + * xxup phone * xxmaj system . xxmaj uri .. xxunk xxunk ) xxunk xxunk baseuri , xxmaj string path ) xxunk xxunk xxunk xxunk +49 xxunk xxunk ) xxunk xxmaj google . xxmaj apis . xxmaj requests . xxxhtml xxunk ( ) xxunk xxunk xxunk xxunk xxunk xxunk xxunk , xxmaj string xxunk , xxmaj string xxunk , xxmaj nullable`1 model ) xxunk \n",
       "  xxmaj related reading would suggest the library code does not switch from xxup get to xxup post for large requests : https : xxxfilepath xxxliste,xxbos xxxfldtitle xxunk : xxunk : xxmaj this xxup api requires billing to be enabled on the project . xxmaj visit https : xxxfilepath xxxlnkhb xxunk xxxlnkhe to enable billing . xxxfldbody xxmaj view details in xxmaj rollbar : https : xxxfilepath xxxlnkhb rollbar.com xxxlnkhe \n",
       "  ` ` ` \n",
       "  xxunk : xxunk : xxmaj this xxup api requires billing to be enabled on the project . xxmaj visit https : xxxfilepath xxxlnkhb xxunk xxxlnkhe to enable billing . xxmaj file \" xxxfilepath \" , line 211 , in xxunk xxmaj file \" xxxfilepath \" , line 102 , in xxunk xxmaj file \" xxxfilepath \" , line 179 , in xxunk xxmaj file \" xxxfilepath \" , line 286 , in xxunk xxmaj file \" xxxfilepath \" , line 107 , in block ( 2 levels ) in execute xxmaj file \" xxxfilepath \" , line 54 , in block in retriable xxmaj file \" xxxfilepath \" , line 48 , in times xxmaj file \" xxxfilepath \" , line 48 , in retriable xxmaj file \" xxxfilepath \" , line 104 , in block in execute xxmaj file \" xxxfilepath \" , line 54 , in block in retriable xxmaj file \" xxxfilepath \" , line 48 , in times xxmaj file \" xxxfilepath \" , line 48 , in retriable xxmaj file \" xxxfilepath \" , line 96 , in execute xxmaj file \" xxxfilepath \" , line 351 , in xxunk xxmaj file \" xxxfilepath \" , line 194 , in xxunk xxmaj file \" xxxfilepath \" , line 29 , in search xxmaj file \" xxxfilepath \" , line 73 , in xxunk xxmaj file \" xxxfilepath \" , line 48 , in xxunk xxmaj file \" xxxfilepath \" , line 22 , in block in xxunk xxmaj file \" xxxfilepath \" , line 113 , in block in fetch xxmaj file \" xxxfilepath \" , line 371 , in block in instrument xxmaj file \" xxxfilepath \" , line 166 , in instrument xxmaj file \" xxxfilepath \" , line 371 , in instrument xxmaj file \" xxxfilepath \" , line 112 , in fetch xxmaj file \" xxxfilepath \" , line 21 , in xxunk xxmaj file \" xxxfilepath \" , line 11 , in search xxmaj file \" xxxfilepath \" , line 4 , in send_action xxmaj file \" xxxfilepath \" , line 188 , in process_action xxmaj file \" xxxfilepath \" , line 30 , in process_action xxmaj file \" xxxfilepath \" , line 20 , in block in process_action xxmaj file \" xxxfilepath \" , line 126 , in call xxmaj file \" xxxfilepath \" , line 506 , in block ( 2 levels ) in compile xxmaj file \" xxxfilepath \" , line 455 , in call xxmaj file \" xxxfilepath \" , line 101 , in run_callbacks xxmaj file \" xxxfilepath \" , line 750 , in _ run_process_action_callbacks xxmaj file \" xxxfilepath \" , line 90 , in run_callbacks xxmaj file \" xxxfilepath \" , line 19 , in process_action xxmaj file \" xxxfilepath \" , line 20 , in process_action xxmaj file \" xxxfilepath \" , line 32 , in block in process_action xxmaj file \" xxxfilepath \" , line 164 , in block in instrument xxmaj file \" xxxfilepath \" , line 21 , in instrument xxmaj file \" xxxfilepath \" , line 164 , in instrument xxmaj file \" xxxfilepath \" , line 30 , in process_action xxmaj file \" xxxfilepath \" , line 248 , in process_action xxmaj file \" xxxfilepath \" , line 18 , in process_action xxmaj file \" xxxfilepath \" , line 126 , in process xxmaj file \" xxxfilepath \" , line 30 , in process xxmaj file \" xxxfilepath \" , line 190 , in dispatch xxmaj file \" xxxfilepath \" , line 262 , in dispatch xxmaj file \" xxxfilepath \" , line 50 , in dispatch xxmaj file \" xxxfilepath \" , line 32 , in serve xxmaj file \" xxxfilepath \" , line 39 , in block in serve xxmaj file \" xxxfilepath \" , line 26 , in each xxmaj file \" xxxfilepath \" , line 26 , in serve xxmaj file \" xxxfilepath \" , line 725 , in call xxmaj file \" xxxfilepath,xxbos xxxfldtitle xxunk model should not include the task lists . xxxfldbody xxxcdb xxunk xxxcde \n",
       "  is the key entry point for syncing project in intellij ( xxup ij ) and xxmaj android xxmaj studio ( xxup as ) . xxmaj we need to query this model first before querying our more specialized models . xxmaj currently , it also contains the list of tasks . xxmaj this is used in xxup ij / xxup as to display the task in the xxmaj gradle pane on the right . xxmaj we would like to make this optional somehow . xxmaj maybe make the current model always return an empty list , and provide a different model that returns the task list if needed . xxmaj our reasoning : \n",
       "  xxxlistb xxmaj nobody uses this in xxmaj studio ( do n't know about xxup ij ) . xxmaj we do n't have actual metrics ( we will add some ) but users have told us they do n't use it . xxmaj this does not scale . a 100 module project will create 20,000 tasks . xxmaj we have heard from users with several 100s of modules that this is something that impact performance and are doing hacks to disable it on the xxup ide side . xxmaj this triggers task configuration during sync because xxunk includes the group and description that requires a task configuration . xxmaj side note , it would be nice to be able to set these , as well as dependencies on xxunk , but in our case it 's just simpler to not return the tasks at all . a different option might be to only provide the tasks that have a group associated with them . xxmaj right now it includes all the tasks , including the ones that do n't have groups . xxmaj none of these are meant to be called directly ( in the case of xxmaj android at least ) . xxmaj doing this would require us to be able to set group on xxunk to avoid triggering configuration . xxxliste,xxbos xxxfldtitle xxmaj grand xxmaj theft xxmaj auto v - blurry textures xxxfldbody xxmaj hi , i play xxup xxunk using xxmaj proton 3.7 - 3 and it runs great , xxunk textures look bad . i 'm not sure why , i have xxup af set to 16x and textures to high . xxximg xxunk_1 xxximgf jpg a few more screens including in - game settings : https : xxxfilepath xxxlnkhb imgur.com xxxlnkhe xxmaj see ? xxmaj the far you look the worse the textures are . xxmaj it looks like broken anisotropic filtering even though it 's set to 16x . xxmaj computer info : xxup os : xxmaj arch xxmaj linux x64 + xxmaj gnome xxmaj shell 3.28 xxup cpu : i5 xxup xxunk xxup gpu : xxup nvidia geforce xxup gtx 1050 xxmaj ti [ 396.54 driver ] xxup ram : 24 xxup gb xxup ddr4 xxunk xxmaj steam system info : https : xxxfilepath xxxlnkhb pastebin.com xxxlnkhe i tried disabling xxup af in the game , forcing xxup af to 16x in xxmaj nvidia settings , setting shader to high but it 's exactly the same . i use default xxmaj proton 3.7 - 3 with only the workaround suggested by xxxatmention here https : xxxfilepath xxxlnkhb github.com xxxlnkhe,xxbos xxxfldtitle xxmaj graphical glitch / bug in xxmaj double xxmaj dragon / w and / w - out sprite limit xxxfldbody xxximg double dragon u _ 000 xxximgf png xxmaj in lower right corner u can see a little glitch / bug in xxup xxunk\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: /IssuesLanguageModel/model/lang_model;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(60000, 800, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(60000, 800, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(800, 2400, batch_first=True)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(2400, 2400, batch_first=True)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(2400, 2400, batch_first=True)\n",
       "      )\n",
       "      (3): WeightDropout(\n",
       "        (module): LSTM(2400, 800, batch_first=True)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "      (3): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=800, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7f857225c378>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/ds/lang_model'), model_dir=PosixPath('/ds/lang_model/models_22zkdqlr'), callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[RNNTrainer\n",
       "learn: LanguageLearner(data=TextLMDataBunch;\n",
       "\n",
       "Train: LabelList (16762799 items)\n",
       "x: LMTextList\n",
       "xxbos xxxfldtitle xxmaj grab excerpt and image using xxmaj open xxmaj graph xxxfldbody xxmaj we can update the wordpress excerpt and featured image by using xxmaj open xxmaj graph to get that information . xxmaj idea comes from xxmaj dan xxmaj scott via e - mail .,xxbos xxxfldtitle xxmaj gracefully handling ctrl+c ignores xxmaj program xxmaj main exit code . xxxfldbody xxmaj repro : \n",
       "  xxxlistb dotnet new console xxmaj paste the repro code below into xxmaj program.cs dotnet run xxmaj hit xxmaj ctrl + c xxmaj look at the last exit code . xxmaj we expect in the below repo for the exit code to be 1337 xxmaj built a repro that 's inspired by xxup asp.net 's hosting bits here : https : xxxfilepath xxmaj repro : using xxmaj system ; using xxmaj system . xxmaj threading ; using xxmaj system . xxmaj threading . xxmaj tasks ; namespace xxunk xxxjson \n",
       "  / cc xxxatmention xxxatmention xxxliste,xxbos xxxfldtitle xxunk - xxmaj android not working within my app xxxfldbody xxmaj hi i 'm on xxmaj mac xxup os x 10.12.6 with xxmaj android xxmaj studio 2.3.3 and trying to write my own trace functionality so that i can see all of my app 's method calls with arguments on xxmaj android and stream them to my computer with xxxcdb adb logcat xxxcde \n",
       "  . xxmaj this trace will give me a broad view of the projects i work on to cut through callback hell in xxmaj android xxup api . i tried your example app at https : xxxfilepath xxxlnkhb github.com xxxlnkhe and it worked ( after many blind xxunk with xxunk , xxmaj frida , etc ) thank you for that ! i 've spent many days searching the web and as far as i can tell , your xxunk - xxmaj android library is the only one that provides a working example for catching all app method calls using aspectj on xxmaj android without having to annotate every method . i even managed to strip out the xxxcdb kotlin xxxcde \n",
       "  , xxxcdb groovy xxxcde \n",
       "  , xxxcdb proguard xxxcde \n",
       "  , xxxcdb dagger xxxcde \n",
       "  and xxxcdb xxunk xxxcde \n",
       "  stuff , so it simply has these aspectj hooks in xxxcdb xxmaj profiler xxxcde \n",
       "  and xxxcdb xxunk xxxcde \n",
       "  : before ( ) : xxunk ( ) xxxjson after ( ) : xxunk ( ) xxxjson i left xxxcdb xxunk xxxcde \n",
       "  in the project because i was n't sure if it was required for detecting anonymous classes ( which i use in place of anonymous methods for now ) . \n",
       "  xxxhm xxmaj however , a few things : xxmaj your xxunk never gets called if i create xxunk xxxfilepath with : \\ timport xxunk ; \\ timport xxunk \\ timport xxunk ; \\ timport xxunk \\ xxunk xxunk xxxjson \n",
       "  xxxhm xxmaj you can try it yourself ( i do n't know if it 's not detecting the xxxcdb xxunk ( ) xxxcde \n",
       "  etc ) . xxmaj which means i have no starting point for writing my own aspectj code . xxxhm xxmaj also , i 've spent 2 days so far trying to copy your xxxcdb build.gradle xxxcde \n",
       "  settings to my app and no matter how hard i try , it never calls my xxxcdb xxmaj profiler xxxcde \n",
       "  or xxxcdb xxunk xxxcde \n",
       "  . i 'm pretty sure i 've nested it properly in xxxcdb src xxxfilepath xxxcde \n",
       "  but i ca n't get it to work . xxmaj without a better understanding of how the xxunk call works , how aspectj is detecting the app 's methods , and how that gets called from gradle , there 's simply no further i can go with this . xxxhm i even tried building issue https : xxxfilepath xxxlnkhb github.com xxxlnkhe https : xxxfilepath xxxlnkhb github.com xxxlnkhe but no matter how hard i try , it no longer builds and runs as of xxmaj android xxmaj studio 2.3.3 . xxmaj if someone can fork that somewhere and get it working , that might give me a starting point . xxmaj so without a minimalist app containing an xxmaj empty xxmaj activity , xxmaj basic xxmaj activity ( or some other example from the xxmaj android xxmaj studio project stationary ) with the addition of your library and a single aspectj example that demonstrates detecting the app 's methods without annotations , developers are going to have a hard time integrating your work . xxmaj there are simply too many permutations to explore if it does n't work . i 'm unable to post my app to github , but do you have any suggestions i can try to show : \n",
       "  xxxlistb xxmaj that my xxmaj profiler aspectj code is compiling and running correctly ? a list of the app 's methods collected by aspectj , or proof that they are being detected ? xxmaj here is a minimal aspectj method autodetection example , forked from yours with the above steps applied https : xxxfilepath and even though it works , i ca n't get the same build.gradle settings to work within my app . xxmaj thanks in advance for any help you can provide ! \n",
       "  xxxliste,xxbos xxxfldtitle xxmaj graph : xxmaj components problem xxxfldbody xxmaj solution for xxmaj components problem in xxmaj java with sample test . xxmaj click for xxmaj problem xxmaj statement and test cases . xxxlnkhb github.com xxxlnkhe,xxbos xxxfldtitle xxmaj great xxmaj plains xxmaj seeder xxup xxunk xxxfldbody xxmaj mode 6 does not lower this implement on course xxunk can manually lower it and it drives just fine . xxmaj the turn maneuvers is raising and lowering xxunk not stop at field edge on turn finish ) . xxmaj could n't test the refill course xxmaj drive xxmaj now button does nothing . xxmaj no errors in log ) . xxmaj also if you leave the ridge markers on auto it spits out . \n",
       "  xxmaj error : xxmaj running xxup lua method ' update ' . datas xxxfilepath : attempt to index a nil value xxmaj and requires a game restart to clear . xxmaj this seeder has no ridge markers xxmaj debug channel 17 at pastebin xxxlnkhb pastebin.com xxxlnkhe xxmaj sorry for breaking xxup cp so quickly . i have this mod do just about everything for me in xxup fs . xxmaj keep up the good work xxup :d\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: /IssuesLanguageModel/model/lang_model;\n",
       "\n",
       "Valid: LabelList (1858033 items)\n",
       "x: LMTextList\n",
       "xxbos xxxfldtitle xxmaj google . xxmaj cloud . xxmaj translation . xxup v2 - xxmaj translate xxmaj html xxxfldbody xxxhs xxmaj environment details \n",
       "  xxxlistb xxup os : xxmaj windows xxup .net version : 4.5.2 xxmaj package name and version : xxmaj google . xxmaj cloud . xxmaj translation . xxup v2 xxrep 4 # xxmaj steps to reproduce xxmaj large xxmaj html snippet to translate xxmaj code xxunk gc = xxunk xxunk xxunk = xxunk . xxmaj xxunk , xxunk xxunk result = xxunk . xxunk , xxunk ( ) , xxunk xxmaj error [ xxunk : xxmaj invalid xxup uri : xxmaj the xxmaj uri string is too long . ] xxmaj system . xxmaj uri . xxunk uri , xxmaj boolean xxunk , xxunk xxunk ) + * xxup phone * xxmaj system . xxmaj uri .. xxunk xxunk ) xxunk xxunk baseuri , xxmaj string path ) xxunk xxunk xxunk xxunk +49 xxunk xxunk ) xxunk xxmaj google . xxmaj apis . xxmaj requests . xxxhtml xxunk ( ) xxunk xxunk xxunk xxunk xxunk xxunk xxunk , xxmaj string xxunk , xxmaj string xxunk , xxmaj nullable`1 model ) xxunk \n",
       "  xxmaj related reading would suggest the library code does not switch from xxup get to xxup post for large requests : https : xxxfilepath xxxliste,xxbos xxxfldtitle xxunk : xxunk : xxmaj this xxup api requires billing to be enabled on the project . xxmaj visit https : xxxfilepath xxxlnkhb xxunk xxxlnkhe to enable billing . xxxfldbody xxmaj view details in xxmaj rollbar : https : xxxfilepath xxxlnkhb rollbar.com xxxlnkhe \n",
       "  ` ` ` \n",
       "  xxunk : xxunk : xxmaj this xxup api requires billing to be enabled on the project . xxmaj visit https : xxxfilepath xxxlnkhb xxunk xxxlnkhe to enable billing . xxmaj file \" xxxfilepath \" , line 211 , in xxunk xxmaj file \" xxxfilepath \" , line 102 , in xxunk xxmaj file \" xxxfilepath \" , line 179 , in xxunk xxmaj file \" xxxfilepath \" , line 286 , in xxunk xxmaj file \" xxxfilepath \" , line 107 , in block ( 2 levels ) in execute xxmaj file \" xxxfilepath \" , line 54 , in block in retriable xxmaj file \" xxxfilepath \" , line 48 , in times xxmaj file \" xxxfilepath \" , line 48 , in retriable xxmaj file \" xxxfilepath \" , line 104 , in block in execute xxmaj file \" xxxfilepath \" , line 54 , in block in retriable xxmaj file \" xxxfilepath \" , line 48 , in times xxmaj file \" xxxfilepath \" , line 48 , in retriable xxmaj file \" xxxfilepath \" , line 96 , in execute xxmaj file \" xxxfilepath \" , line 351 , in xxunk xxmaj file \" xxxfilepath \" , line 194 , in xxunk xxmaj file \" xxxfilepath \" , line 29 , in search xxmaj file \" xxxfilepath \" , line 73 , in xxunk xxmaj file \" xxxfilepath \" , line 48 , in xxunk xxmaj file \" xxxfilepath \" , line 22 , in block in xxunk xxmaj file \" xxxfilepath \" , line 113 , in block in fetch xxmaj file \" xxxfilepath \" , line 371 , in block in instrument xxmaj file \" xxxfilepath \" , line 166 , in instrument xxmaj file \" xxxfilepath \" , line 371 , in instrument xxmaj file \" xxxfilepath \" , line 112 , in fetch xxmaj file \" xxxfilepath \" , line 21 , in xxunk xxmaj file \" xxxfilepath \" , line 11 , in search xxmaj file \" xxxfilepath \" , line 4 , in send_action xxmaj file \" xxxfilepath \" , line 188 , in process_action xxmaj file \" xxxfilepath \" , line 30 , in process_action xxmaj file \" xxxfilepath \" , line 20 , in block in process_action xxmaj file \" xxxfilepath \" , line 126 , in call xxmaj file \" xxxfilepath \" , line 506 , in block ( 2 levels ) in compile xxmaj file \" xxxfilepath \" , line 455 , in call xxmaj file \" xxxfilepath \" , line 101 , in run_callbacks xxmaj file \" xxxfilepath \" , line 750 , in _ run_process_action_callbacks xxmaj file \" xxxfilepath \" , line 90 , in run_callbacks xxmaj file \" xxxfilepath \" , line 19 , in process_action xxmaj file \" xxxfilepath \" , line 20 , in process_action xxmaj file \" xxxfilepath \" , line 32 , in block in process_action xxmaj file \" xxxfilepath \" , line 164 , in block in instrument xxmaj file \" xxxfilepath \" , line 21 , in instrument xxmaj file \" xxxfilepath \" , line 164 , in instrument xxmaj file \" xxxfilepath \" , line 30 , in process_action xxmaj file \" xxxfilepath \" , line 248 , in process_action xxmaj file \" xxxfilepath \" , line 18 , in process_action xxmaj file \" xxxfilepath \" , line 126 , in process xxmaj file \" xxxfilepath \" , line 30 , in process xxmaj file \" xxxfilepath \" , line 190 , in dispatch xxmaj file \" xxxfilepath \" , line 262 , in dispatch xxmaj file \" xxxfilepath \" , line 50 , in dispatch xxmaj file \" xxxfilepath \" , line 32 , in serve xxmaj file \" xxxfilepath \" , line 39 , in block in serve xxmaj file \" xxxfilepath \" , line 26 , in each xxmaj file \" xxxfilepath \" , line 26 , in serve xxmaj file \" xxxfilepath \" , line 725 , in call xxmaj file \" xxxfilepath,xxbos xxxfldtitle xxunk model should not include the task lists . xxxfldbody xxxcdb xxunk xxxcde \n",
       "  is the key entry point for syncing project in intellij ( xxup ij ) and xxmaj android xxmaj studio ( xxup as ) . xxmaj we need to query this model first before querying our more specialized models . xxmaj currently , it also contains the list of tasks . xxmaj this is used in xxup ij / xxup as to display the task in the xxmaj gradle pane on the right . xxmaj we would like to make this optional somehow . xxmaj maybe make the current model always return an empty list , and provide a different model that returns the task list if needed . xxmaj our reasoning : \n",
       "  xxxlistb xxmaj nobody uses this in xxmaj studio ( do n't know about xxup ij ) . xxmaj we do n't have actual metrics ( we will add some ) but users have told us they do n't use it . xxmaj this does not scale . a 100 module project will create 20,000 tasks . xxmaj we have heard from users with several 100s of modules that this is something that impact performance and are doing hacks to disable it on the xxup ide side . xxmaj this triggers task configuration during sync because xxunk includes the group and description that requires a task configuration . xxmaj side note , it would be nice to be able to set these , as well as dependencies on xxunk , but in our case it 's just simpler to not return the tasks at all . a different option might be to only provide the tasks that have a group associated with them . xxmaj right now it includes all the tasks , including the ones that do n't have groups . xxmaj none of these are meant to be called directly ( in the case of xxmaj android at least ) . xxmaj doing this would require us to be able to set group on xxunk to avoid triggering configuration . xxxliste,xxbos xxxfldtitle xxmaj grand xxmaj theft xxmaj auto v - blurry textures xxxfldbody xxmaj hi , i play xxup xxunk using xxmaj proton 3.7 - 3 and it runs great , xxunk textures look bad . i 'm not sure why , i have xxup af set to 16x and textures to high . xxximg xxunk_1 xxximgf jpg a few more screens including in - game settings : https : xxxfilepath xxxlnkhb imgur.com xxxlnkhe xxmaj see ? xxmaj the far you look the worse the textures are . xxmaj it looks like broken anisotropic filtering even though it 's set to 16x . xxmaj computer info : xxup os : xxmaj arch xxmaj linux x64 + xxmaj gnome xxmaj shell 3.28 xxup cpu : i5 xxup xxunk xxup gpu : xxup nvidia geforce xxup gtx 1050 xxmaj ti [ 396.54 driver ] xxup ram : 24 xxup gb xxup ddr4 xxunk xxmaj steam system info : https : xxxfilepath xxxlnkhb pastebin.com xxxlnkhe i tried disabling xxup af in the game , forcing xxup af to 16x in xxmaj nvidia settings , setting shader to high but it 's exactly the same . i use default xxmaj proton 3.7 - 3 with only the workaround suggested by xxxatmention here https : xxxfilepath xxxlnkhb github.com xxxlnkhe,xxbos xxxfldtitle xxmaj graphical glitch / bug in xxmaj double xxmaj dragon / w and / w - out sprite limit xxxfldbody xxximg double dragon u _ 000 xxximgf png xxmaj in lower right corner u can see a little glitch / bug in xxup xxunk\n",
       "y: LMLabelList\n",
       ",,,,\n",
       "Path: /IssuesLanguageModel/model/lang_model;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): AWD_LSTM(\n",
       "    (encoder): Embedding(60000, 800, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(60000, 800, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(800, 2400, batch_first=True)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(2400, 2400, batch_first=True)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(2400, 2400, batch_first=True)\n",
       "      )\n",
       "      (3): WeightDropout(\n",
       "        (module): LSTM(2400, 800, batch_first=True)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "      (3): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=800, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7f857225c378>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/ds/lang_model'), model_dir=PosixPath('/ds/lang_model/models_22zkdqlr'), callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(800, 2400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(2400, 2400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(2400, 2400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(2400, 800, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): Embedding(60000, 800, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(60000, 800, padding_idx=1)\n",
       "  )\n",
       "  (2): LinearDecoder(\n",
       "    (decoder): Linear(in_features=800, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")], add_time=True, silent=None)\n",
       "alpha: 2.0\n",
       "beta: 1.0], layer_groups=[Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(800, 2400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(2400, 2400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(2400, 2400, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(2400, 800, batch_first=True)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): Embedding(60000, 800, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(60000, 800, padding_idx=1)\n",
       "  )\n",
       "  (2): LinearDecoder(\n",
       "    (decoder): Linear(in_features=800, out_features=60000, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")], add_time=True, silent=None)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.load('best_22zkdqlr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export Minimal Model State For Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.export('trained_model_22zkdqlr.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('trained_model_encoder_22zkdqlr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is very large so if you are running this notebook best to release memory by deleting these objects and loading the more lightweight inference artifacts that we just saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "del learn\n",
    "del data_lm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part II:  Load Minimal Model For Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from inference import InferenceWrapper, pass_through"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create an `InferenceWrapper` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "wrapper = InferenceWrapper(model_path='/ds/lang_model/models_22zkdqlr/',\n",
    "                           )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "issue_string = '# hello abacadabra world \\nA second line **something bold**.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.0168,  0.0424,  0.0784,  ...,  0.0782, -0.0466,  0.0152]],\n",
      "       device='cuda:0', grad_fn=<CatBackward>)\n",
      "torch.Size([1, 2400])\n"
     ]
    }
   ],
   "source": [
    "pooledfeat = wrapper.get_pooled_features(issue_string)\n",
    "print(pooledfeat)\n",
    "print(pooledfeat.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[-0.0038, -0.0458,  0.0113,  ...,  0.0395,  0.0149,  0.0079],\n",
      "         [ 0.0141, -0.0070,  0.0614,  ...,  0.0283,  0.0589, -0.0534],\n",
      "         [-0.0280,  0.0892,  0.0803,  ..., -0.0164,  0.0138,  0.1296],\n",
      "         ...,\n",
      "         [ 0.0004,  0.1302,  0.1215,  ..., -0.0525,  0.1411, -0.0619],\n",
      "         [-0.0224,  0.1757,  0.1024,  ...,  0.0663,  0.0730,  0.1068],\n",
      "         [ 0.0479, -0.0142,  0.1249,  ...,  0.0782, -0.0466,  0.0152]]],\n",
      "       device='cuda:0', grad_fn=<CudnnRnnBackward>)\n",
      "torch.Size([1, 12, 800])\n"
     ]
    }
   ],
   "source": [
    "rawfeat = wrapper.get_raw_features(issue_string)\n",
    "print(rawfeat)\n",
    "print(rawfeat.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predict the next  5 words\n",
    "\n",
    "We don't actually use this functionality, but it is interesting to see for those who are curious what the output of a langauge model is.  Recall that we are using the encoder of the language model to extract features from GitHub issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I am having trouble opening a project with the ios binary'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wrapper.learn.predict('I am having trouble opening a', 5)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}