Deploy in background via child process? (Neovim feature)

[chexxor]

Currently all vim-force.com commands block the UI. I'd like to be able to continue looking at code while a deploy is executing, as deploys can take >1 min.

Neovim has a job-control thing which enables a plugin to spawn a background process. I recall seeing a plugin which wraps this in a backward compatible way.

Have you thought about running these vim-force.com commands asynchronously?

[neowit]

Yes, I did consider asynchronous mode in the past. The problem is that there is no standard way of doing this with vim.

Neovim, as you mentioned, has one sort of job control, Vim (as of recently) has job_start which is incompatible with the one used in Neovim.
There is also vim-dispatch, which came along before both Neovim and job_start in vim.

I'd love to enable asynchronous deployment, but presently there is no clear winner in async approaches for vim, so if we pick any of the above methods it would be available only to a small minority of vim users.

Ideal scenario would be if vim & neovim were to standardise their async API.

Do you have experience with any of those async methods?

[chexxor]

I'd love to enable asynchronous deployment, but presently there is no clear winner in async approaches for vim,

After reading the Channel & job design thread thread on their vim-dev group, I see what you mean. Vim is currently working on adding async tools, but Bram and Thiago, the Vim and Neovim project leaders, haven't agreed on the basic structure of async (e.g. channels & callbacks).

I don't have have much experience with async methods, no.

I searched for how some Neovim plugins which use Neovim's job_control are written. fasd.vim wraps job_start:

    if has('nvim')
      call jobstart(cmd)
    elseif has('job')
      call job_start(cmd)
    else
      call system(join(cmd, ' '))
    endif

And redismru.vim wraps is in a similar way

  let start = has('nvim') ? 'jobstart' : 'job_start'
  execute 'call '.start.'(a:cmd, opts)'

vim-force.com already has a relatively complicated command runner to add support for Windows via Python and vimproc, so it seems ok to add another small conditional to support Neovim. :)

[neowit]

thank you for the examples, I'll give it a closer look at some point.

[chexxor]

@neowit, if you don't need background deploy, then you shouldn't spend too much time on it.

I wanted your quick opinion on it because I want to test background processing in vim. I'm just figuring the price we'd have to pay to get it. I plan to move forward with adding support for Neovim's background jobs (but not sure how soon, as I'm pretty slow). I'll update this thread if I make any discoveries.

[ViViDboarder]

In my plugin I wrote an implementation that leverages Neomake, Dispatch or just blocking commands in Vim. An ideal solution would be to use jobs directly, but this provides a super quick way to drop in support without worrying about how each API works.

https://github.com/ViViDboarder/force-vim/blob/master/plugin/ForceCli.vim#L71

[neowit]

Hello @ViViDboarder,

Thank you for the suggestion.

Have tried force-vim plugin just now (with Dispatch installed) and let g:force_dispatch_background = 1, but have not been able to see much visual feedback when running commands like :ForceTest or :ForceDeploy.
All I could see is last executed external command, e.g. !force test "Test1" (iterm/56905)
It also spawns a separate terminal window, which may not be to everyone's taste.

I am probably doing something wrong - do you have an example/video of running background job ?
When background mode enabled - is it possible to see a visual progress/feedback ?

Also, am I right assuming that neither Neomake nor Dispatch can be used in GUI mode, e.g. MacVim or gVim?

[ViViDboarder]

That behavior is part of Dispatch. Dispatch offers a few ways to spawn the background command. I typically use tmux to do it. You actually want foreground mode and tmux if you want to see it. Then it will auto create a split with it running.

https://github.com/tpope/vim-dispatch#foreground-builds

[neowit]

Thank you for the clarification.

In case of vim-force.com plugin - Dispatch or Neomake (or any other async utility) can not be easily employed to enable async calls.
Main reason is that existing code base does a lot of stuff after call to sfdc has finished.
Before any async method is implemented all those logical parts (following SFDC call completion) have to be moved to callback methods.

Additionally (assuming existing plugin codebase has been updated with callbacks) there is a question of how to present feedback (messages) shown upon command completion.
For example :ApexPrintConflicts may result in many messages with details. With current (synchronous method) vim does it for you automatically.
With any async method :messages window has to be opened somehow, ideally from vim script. Unfortunately few tests I have done so far have shown that with current vim build calling something like :10message from a callback method crashes vim most of the time. Hopefully this is just a temporary issue (since job_start/callbacks are a new thing), but it is a problem right now nevertheless.

[neowit]

Hi guys,

I have had a go on implementing new vim async jobs support in vim-force.com, but not quite ready to release it yet.
If you wan to try it out and give me your feedback please checkout branch vim-async and use tooling-force.com-0.3.6.2.jar.

vim-async version requires server mode (see :help 'g:apex_server') and still has few (mostly visual) problems (e.g. when you run first command, and server starts, the way it is shown looks ugly).
Not all commands have been implemented in async mode, but the important ones like :ApexSave, :ApexDeploy, :ApexTest are using async jobs.

There is no documentation yet, but the only new command you need to know is :ApexMessages - it opens special buffer with log of all operations done in current session. In case of problems/errors it should open automatically, but if it does not, or you are not sure what is happening, make sure to call :ApexMessages.

IMPORTANT: you must be on the latest vim version. Especially if you use MacVim. Up until very recently there were some problems with jobs implementation.

Here is the version I know works: VIM - Vi IMproved 7.4, Included patches: 1-1831

[chexxor]

I only tried :ApexDeployOne command for now, and it was successful. @neowit, this is a really awesome user experience. I'll keep using it to see if I encounter any problems.

I use Nix to install tools like Vim. I had to manually bump the vim version from 7.4.1585 to 7.4.1941 to get the commands in the vim-async branch to work. Here's the ~/.nixpkgs/config.nix file I used to do that, if others need to do same.

let
  vim-version = "7.4.1941";
in
{
  packageOverrides = pkgs: rec {
    vim = pkgs.vim.overrideDerivation (oldAttrs: {
      name = "vim-${vim-version}";
      src = pkgs.fetchFromGitHub {
        owner = "vim";
        repo = "vim";
        rev = "v${vim-version}";
        sha256 = "0apq7sv1fbyfzqh4q0r2z19bn4gbjlb97wyl80kj7qsqmsy7m1lm";                                                                                                                                         
      };
    });
  };
}

[chexxor]

@neowit, I wonder if you've also noticed this:

When I start an async command, like ApexDeployOne, I hit a key to return to the buffer I was using and continue to watch the deploy updates in the status bar. If I change focus to a non-Apex syntax buffer, like the command history, then I see error messages appearing in the status bar. They don't affect the status of deployment, however, so I don't mind too much.

Any idea where to look to debug this?

[neowit]

Hello @chexxor,

No, this is not something I have observed, but that is probably just because of insufficient amount of testing.

Can you give a step-by-step scenario how to reproduce this ?

Also, which vim do you use? Console or one of GUI variants? What version?

If you could record a video/gif of the issue that would be even better.
Thanks.

[chexxor]

I'm using Vim 7.4.1941 in Terminal on Mac Yosemite 10.10.5.
When I have an async command running, then do q: to get command history, I get the following error appearing the in command history window when the next async job status update appears. I'll look at getting a video soon.

Failed to identify resource name for file 
Error detected while processing function 112[8]..apexMessages#log[1]..<SNR>86_logHeader[3]..<SNR>86_cacheLines[6]..<SNR>86_isVisible[1]..<SNR>86_getBufNumber[1]..<SNR>86_getBufferName[1]..apex#getApexProjectSrcPath[5]..apex#getSFDCPro
jectPathAndName:
line   24:
E605: Exception not caught: src folder not found
Error detected while processing function 112[8]..apexMessages#log[1]..<SNR>86_logHeader[3]..<SNR>86_cacheLines:
-- More --

[neowit]

thank you for the details.
No need for video actually, I get the idea. Will try to reproduce it...

[neowit]

Hello @chexxor,

The error you were experiencing was caused by a redundant piece of code which is now removed.
Please try again with latest version of vim-async branch and let me know if the issue is still there.

[chexxor]

@neowit, I did some quick testing - the issue I described above with q: command seems to be resolved. Works great!