VIRA-253: slow performance

[chinwobble]

As a user, I don't want an open VIRA report to be slow and I don't want each report to take a long time to load.

I suspect something is wrong with the async code.

I'm going to put a few examples of how to profile the python and vim code.

• Override timeout for async updates
• Error after message and issue create (VIRA-247) [d846711]
  • c03487c was the last good one?!?
    • leads to menu layout again...
  • OoO demands tabs win...
  • I got "lucky" and had seen a flash that a second .vira_prompt is open on the fly and we need strings and numbers to work properly
  • One more test
  • I will need to go through the list again. This appears to fix the menus but I lost the fix for it.
• Fold lost (VIRA-221) [31c6fec]
• Refresh report goes blank (VIRA-213) [627b6c4]

profile python code

function! vira#_print_report(list) " {{{2
  " Write report output into buffer
  python3 << EOF
import vim
import cProfile
with cProfile.Profile() as pr:
  report = Vira.api.get_report().splitlines()
  vim.current.buffer.append(report)
pr.dump_stats('/tmp/vira_print_report.dmp')

EOF
endfunction

Then i use snakeviz to check the results.

[n0v1c3]

Please note pycontribs/jira#622 from our README as well. Not directly related on the connection but worth a look.

[n0v1c3]

I do no not have any major delays though.

[n0v1c3]

I know I broke quite a bit getting it working backwards but it was still very worth it.

[n0v1c3]

Regarding the snakeviz package. Using it to test is fine. We would need to make a good plane to make something like that long term.

We even needed to fold with yaml and that is just because @mikeboiko and @n0v1c3 like it.

[n0v1c3]

Older Comments was also broken between the following comets.

| * | 31c6fec VIRA-221: Fixed bug where opening menu/report cleared clipboard
| * | d3bbbbc Fixed dictionary merging bug using deepcopy()

[chinwobble]

I commented out this code and its fast again.

function! vira#_async() abort "{{{2
  " try
  "   python3 Vira.api._async(Vira.api._async_vim)
  " endtry
  " if s:vira_async_debug | echo s:versions | endif
  call timer_start(s:vira_async_timer, { -> execute('call vira#_async()', '') })
endfunction

I think you can test on your own server by creating hundreds of new versions.

[chinwobble]

This is the vim profile I get when I just have the vira_report open. This does not include connecting or print_report.

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
    8             13.436866  vira#_async()
  221   0.081763   0.054034  fugitive#Find()
  442   0.050392   0.029758  airline#extensions#ale#get()
  221   0.040925   0.027166  airline#check_mode()
 1547              0.017236  airline#util#append()
 2210   0.015839   0.013823  airline#util#wrap()
  221   0.014205   0.012562  <SNR>92_update_hg_branch()
  442   0.014639   0.012370  FugitiveGitDir()
  667   0.018845   0.012113  airline#util#shorten()
  221   0.140207   0.011385  <SNR>92_update_branch()
  221   0.093052   0.011289  fugitive#Head()
  442              0.011046  airline#extensions#coc#get()
  112   0.011688   0.010708  <SNR>74_Highlight_Matching_Pair()
  128   0.010266   0.009734  <SNR>56_notify()
  221   0.010510   0.009663  airline#extensions#fugitiveline#bufname()
  896              0.008813  airline#util#winwidth()
  221              0.007909  airline#extensions#keymap#status()
  221   0.163484   0.007854  airline#extensions#branch#get_head()
    1   0.007846   0.007837  ctrlp#utils#writecache()
  121   0.010140   0.007310  ale#cursor#EchoCursorWarningWithDelay()

[n0v1c3]

If you are shutting it right off I would try setting init and timer quite high in ms.

@mikeboiko and I are looking for a way around their extra lag bugs as well.

[n0v1c3]

Also have you tried to comment out the one mentioned above?

[chinwobble]

I think to implement true async, we need to use the job and channels api in vim.
Here is an example I found.
https://github.com/OmniSharp/omnisharp-vim/blob/03f8fd4c88d1d5447da55ec499f2ca563e7e54d0/autoload/OmniSharp/proc.vim#L4
https://vimhelp.org/channel.txt.html

[n0v1c3]

Probably smoother and cleaner.

[n0v1c3]

@chinwobble you should also be able to try:

let g:vira_async_timer = 30000
let g:vira_async_timer_init = 30000

That should be 30s apart. Worth a test with tack away the python time.

[chinwobble]

I just did a quick test, this doesn't block or lag the ui.

let s:save_cpo = &cpoptions
set cpoptions&vim

let s:jobs = get(s:, 'jobs', [])
let s:channels = get(s:, 'channels', [])

function! proc#EchoErr(channel, msg)
      let v:errmsg = a:msg
      echohl ErrorMsg | echomsg a:msg | echohl None
      echom a:channel
endfunction

function! proc#run_job(msg)
      let job = job_start(
                        \ ['python', '-c', 'import time; time.sleep(10); print("hello")'],
                        \ {'callback': 'proc#EchoErr'})
      call add(s:jobs, job)
      " let channel = job_getchannel(job)

endfunction

[n0v1c3]

Tested in vim and nvim? (I happen to be off the computer now)

[chinwobble]

I tested in vim, but not nvim. I think the API in nvim is slightly different.

If you look in the channels help page, there is a reference to an example server.
https://github.com/vim/vim/blob/master/runtime/tools/demoserver.py

This has a few advantages over the current approach:

• creating a separate demo server can strictly separate the UI and backend logic.
• The server can strictly handle caching versions, project names, etc
• you can send multiple commands to the server in parrallel
• The commands can be small and asynchronously send commands to the server to fetch data

Sometimes we want to block the UI to let users know something is happening.

[n0v1c3]

vim was fantastic. As always I will need to figure out how to use that for nvim

Might take a bit of work for a clean operation in vim as well.

[n0v1c3]

I have got a small vim to work however, I may spend my time on another workaround to make sure it works for nvim as well.

This does not directly give me multitasking as the real problem is still JQL and the core solution will still be to create two separate solutions (vim and vira) the main goal is to infect vim with the vira and not be separated from it. I will also be supporting nvim all the way as vim.

[n0v1c3]

BTW: you are able to call directly to vim from that command without an impact witch is pretty good but I want it to be completely unknown.

I will end up back at this issue once nvim catches up.

[n0v1c3]

I may need to try let s:is_neovim = has( 'nvim' )

[chinwobble]

No I think the server shouldn't know anything about vim.
The server should only handle caching, making api calls to atlassian and reshaping the json to make it easier to work with.

The syntax highlighting, formatting it fields into a table, autocmds, etc should all be handled inside vim outside of hte server.

This way, you can easily test your server logic

[n0v1c3]

We do have a goal for sure to have it out, I guess as long as I think of it as the "n0v1c3 vira" in a single package.

[n0v1c3]

VIRA-83: GitHub and Jira Usage

As a user I would like the ability to use this Vim plugin with either Jira or from GitHub to help make it unrequired to even change the software I am using to track my work on different servers. All function Vim level should look the same (unless simply something is only available from one of them) then the functions can do the server type confirmation and changes as required.

[n0v1c3]

VIRA-83 will most likely agree with that.

I basically want to be able to do this from only the single vira window.

[n0v1c3]

@chinwobble I got all the "new" major bugs that I created out of the way. I am sure that the delay is still an issue for you the bugs just happened to take priority as I tried to start looking at it.

[chinwobble]

The current dev branch is unusable for me since it keeps blocking the UI.

I'm keen to hear your thoughts on whether using VIM jobs api is the correct the approach.

[n0v1c3]

I have had a chat with @mikeboiko in the real world and we do want this to run as a separate process for sure especially as we keep growing. VIM will purely be the front end in the end.

[n0v1c3]

I do have this as the highest priority still as any "larger" users will want a smooth connection inside vim still. Fun part is just to rewrite all my code 👍😋

[n0v1c3]

I put a temp bypass on the version filter menu I am sure I created some bugs as I step away from my computer for a couple days. This should be the key of the lag you are seeing.

It should still keep a list of the versions you have opened the reports of related issues to.

[n0v1c3]

@chinwobble last push may still be a bit slow for you I did not comment out the full chance that opening a report will add that version to your versions list.

I am sure your versions have thousands of issues in them as this seem to be the core of the problem when I try to recreate it.

I can roll my versions menu way back and it will just be a tag for the version and not longer have the counts for number of issues and number completed. I believe I will also loose my percent complete in the reports.

[n0v1c3]

total = self.jira.version_count_related_issues(idx)['issuesFixedCount']
pending = self.jira.version_count_unresolved_issues(idx)

These lines of code will be the core of my problem. I will work on it "tomorrow" but the issue is that the plugin I am using goes through all issues hear instead of letting me go through each one individually. I have tested some to make my own copy and that will be the solution. There was a big difference between versions with 1000 issues and 50 issues, even though my base query is for 1 issue.

Update one issue at a time from one version at a time from one project at a time. And, always break away from the ones you were one "last" time. This will make it so we are only truly pulling one issue at a time and slowly updating the percent complete numbers.

[n0v1c3]

I am beginning to play with a db that is updated slowly and held locally for quicker readings. I will also do another temporary push to the master branch shortly to stop the version updates from lagging your system completely until this is working.

[n0v1c3]

I think it is pretty easy to say now that the db will be the answer for sure. We can make it handle quite a bit of work however, I will want to make sure it is good and safe and can handle a smooth updating process prior to being released based on the db. This will now no longer be a complete "re-wright" of the code but quite a large change in operation that was previously planed and additionally enforced.

Thanks @chinwobble! I will keep this one going for sure.

[n0v1c3]

Trying to make a smarter table design with what time I do have to spend. The branch I am on will be dangerous for a bit but eventually the db will have a version table that will manage the updates for the design changes. I also need to make sure that it is not just hitting hard on both "developers" side and the "owner's" side so I will make sure that there are some timers, limits and priorities with newer issues to not simply ping over the network all day and all night long as hard as we POSSIBLY can.

I will make a comment below this one to start a list of tables as they start coming to my mind and we may need to discuss some of them as some thoughts are quite large and may need some discussion or at least so on/off flags in my code.

[n0v1c3]

• issues

  • time tracking VIRA-10

• projects

• servers

• statuses

• users

• versions

• comments??

[n0v1c3]

If we take enough / all of the data it will be possible to have an offline mode where the firs offline step would be that you could at least read all the data from vira while you are disconnected. Going further we would want to hold comments for when you are back online, much much more fun.

[n0v1c3]

I hope that patch VIRA 0.4.8 will help out well I work on a real longer term solution. This should also help with the speed of the report loading time if that was a concern at all.

• Percents Removed

  • Could just turn this one back on. It will slow down each individual report load again but, you will get each version in the list as you open the issues.

• Version List Broken

[n0v1c3]

The base of a db is coming to life on my branch VIRA-253 it still doesn't really do anything useful but all the features are their for some real functionality to start happening. The first round of the db wont be to bad as I will not worry if it will need to loose its data for next updates however version handling and proper patching will be a high priority as well inside this issue as it should be with db related ones.

I am going to start with a simple state tracker for all issues on the servers you are connected to. It will most likely start with a very simple straight line through the issue numbers for updates but I already have some notes in my real issue for some better logic there as well.

• Update Times
• Open Issues Update Now
• Jump between projects/versions
• Priorities currentUser
• ...I believe there were some more.

My current goal is to keep reducing the async time and simply using logic as we should. Vim with async, db and a network will now basically be able to do anything so also please keep telling me how to improve with ideas like this one. :) - I will block that bad issue mapping one of these days! (VIRA-252)

[chinwobble]

I can you see you put a lot of working into working on the db.

Do you think it will be better to split all the db related code into a different python class.
This vira_api.py file is getting pretty large now.

[n0v1c3]

Good day @chinwobble, I have also told myself that as the db code keeps growing, I just had it in there to get the core of the db running and I am basically there now.

You should be able to have a test run if you are interested, the only thing is no db version control exist yet as it is next on the list. It should just update/delete your first db/tables once that is in place to control my table changes for users.

I am also becoming more and more interested in store basically all the data we display in properly designed tables. This will take up some local storage space however, our original async will now wast almost 0 network and become very very fast as it will be looking at the local db on searches. A query to the JQL server can be closer together only for recent updates if there was on check again shortly if not take a longer break and check for updates issues. This will become an important step to make us wast very little "data".

[n0v1c3]

If you are willing to have a test you will see that you have, well do not have data in your database when you first start. The following commands have not been added into the code mappings/logic but you can add them into .vimrc.

" One second updates ( "vira test slow" )
nnoremap <silent> <leader>vts :let g:vira_async_timer=1000<cr>

" Basically as fast as the network will let you updates ( "vira test fast" )
nnoremap <silent> <leader>vtf :let g:vira_async_timer=25<cr>

A file will also be created in ~/.config/vira called vira.db I will need mapping for actual locations there as well but, I will be interested in the size of your file at this current point and how many issues it has found (this is the core of the bugs I am looking at and it is mostly my sloppy query style I hope)

[n0v1c3]

https://stackoverflow.com/questions/33809881/whats-a-good-way-or-tool-to-version-control-a-sqlite-database-schema-only/38271631

[n0v1c3]

Currently only works with one server, looking into that next. I am also going to slow it down with an option for the fast update.

I have also caught on the side that issues can be missed randomly and a list will need to be created of "possible" missing issues. I know they can be transferred to projects and still apparently show up in the original project. Work around it will need to be made for all cases and make sure it is no longer tracked as missing once found, even if the label dosn't make sense.

[chinwobble]

@n0v1c3 if you want to test with a large jira server, maybe you can sign up to apache's jira and query their api.
https://issues.apache.org/jira

[n0v1c3]

@chinwobble Just added them to the list. I will add them to the tests for the db.

May "guess" is it can handle it. :) with latest pushes.

[n0v1c3]

@chinwobble welcome to the GitHub team! I am making the plan on Jira along with the clean merge with GitHub issues back and forth.

[n0v1c3]

I have at least created an epic for now VIRA-279. You can use this for branches and labels until it gets even better.

I try to use the right commands to label my pushes with the issue as a prefix ie "VIRA-279: ..." this makes GitHub look nice.

As for the dev once you are happy with something you are working on simply merge it into there (also with the issue label "VIRA-279: ...").

My version 0.5.0 goal was the epic issues and some other key features. I put it on hold as the db is changing everything.

[n0v1c3]

That was the correct website to do the testing on BTW. I got to GERONIMO-1632 and then I could not make it any further.

This happened to be the worst issue to crash on as it was called "Test" and I happened to crash there but it also happens to be the last of the more than one thousand issue that were closed on 2006/08/07 (August). The first official day of more than 1000 Jira updates on https://issues.apache.org/jira

This is a bug with my updating by timeline and no handle for more than 1000. There are a few ways I should be able to make this work with the best looking one for now down bellow. I will have a look at this next round and should be able to get past it.

https://confluence.atlassian.com/jirakb/changing-maxresults-parameter-for-jira-cloud-rest-api-779160706.html

[n0v1c3]

I will have more to clean it up but it looks like that should have got the query.

[n0v1c3]

And, now a cleaner, even faster query. My last issue there is simply the query time when pulling from https://issues.apache.org/jira is simply slower than my server in my basement. It will currently take me about a week to update my database with all of their information using JQL especially inside vim.

I am going to finally focus on make the things I have broken work, mostly menus now. Then I will most likely replace all features of all menus along with the reports from the db to call this done.

Outside of vim will be an issue all of it's own after this one and Epics are cleaned up for VIRA 0.5.0 release requirements. I was in the middle of Epic cleanup until db got on my mind.

[n0v1c3]

db now has it's own class ViraDB and it will be cleaned up more and more. I did not worry about braking the front end as the DB is working.

[n0v1c3]

I found some interesting ones, I know that you may be interested in @chinwobble , I had an issue with nvim vs vim while exiting the program and it turns out that there is a knowing issue for the sqlite and print functions and how they are handled in nvim.

I have also cleaned up the sqlite connections only for ONE vim session with vira running. This otherwise, will cause lag as each vim session is waiting for it's turn on sql.

[n0v1c3]

It took this long but I have actually select ONE single version from the menu.

There were of cores still core bugs left into what I have checked in however, this is the key of the original issue.

• Need to protect my SQL further then we will not miss any issues
• Change loading order to newest to oldest
• Clean version menu selection (FINALLY)
• Replace all front end with loading from the database.
• Handle multiple vira connections to a single db
• RELEASE