API Issues with initial sync (RateLimitExceeded) #222
Comments
|
This limit is from server side. I am afraid we have to wait until the time is out. |
|
Alright, but I have more than 20k notes to download locally in order to finish the sync I have read that with a fresh token the API limits should be initially lifted, then why am I facing all this issues? |
Do you mean this link? https://dev.evernote.com/doc/articles/rate_limits.php If you still have Evernote installed, and your Evernote has most of your notes synced, you can transfer locally. If your notes consist of text mainly, and not too many multimedia files, you could consider to sync with Evernote and then transfer locally. I remember that Evernote client has a limit for data amount every month for free user, I am not sure if it is enough for your notes. I don't have so many notes, and I don't know whether the option of 'Restart sync on API limit (experimental)' works or not in reality. But if nixnote doesn't restart the sync automatically, what about restarting nixnote every one hour with an outside shell commands or something like that, with the option of auto 'sync on start up' enabled? This method may take a long time, but it is available. |
Thank you very much for your reply. The idea of a script killing and restarting nixnote every 20 minutes is also a good idea and I could try that. |
You were using evernote legacy through wine, so the latest evernote client can't work with wine, right?
If there are only a few unsynced notes, how about using the web evernote and backing them up manually?
You have backup files, so you don't need to sync all of them, backing up the unsynced notes is enough. Once you get all the necessary notes you want, you could not only import them to nixnote but also other apps you like.
If you decide to use nixnote or other apps which are integrated with Evernote cloud, then this is a safe way, it just takes a long time.
Nixnote is functional in most of the time. Some bugs may appear, but not often. The ui may be a little legacy. and don't offer some modern, fancy functions just as other newer note apps such as obsidian. |
|
Thank you very much for your reply.
Yes, the new evernote version doesn't run under wine.
The backup is from about two weeks ago, and in the meanwhile I might have probably edited many old notes other than adding new ones.
So my last backup is not very fresh, but it looks like that tools like this are still working. So maybe I could pull down a fresh enex backup with that tool and then import the notes in nixnotes (but, by the way, why this tool is able to download and export the notes and nixnotes can't do the same?)
Yes, I agree, I would much prefer to do a clean sync from the client itself. Scripting a recurrent shutdown/restart of nixnote like you were suggesting could be a good option.
I'm not concerned about the look of the UI (and as I said, I was using the legacy version of evernote which was really looking oldshool, but it was quite snappy and functional). What I'm afraid of is inconsistencies between notes written in nixnote and in the new evernote official client. Recently I was using a mix a evernote legacy on one machine and evernote 10 on another machines and very often I was stepping into notes that were missing information in one client or in the other (it was happening especially with image attachments). I guess that overtime they changed things in their architecture and this is one of the reasons why they also dismissed the possibility to use the old clients. |
If this tool works and has no rate limit, it will be easy to solve the problem.
I have not studied the code, I just read the README page, and I guess that it syncs the notes by calling the offical Evernote client APIs. The offical Evernote client may call another group of APIs, which are different from the third-party ones called by nixnote, and have no rate limit, I guess.
Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years. |
I pulled a backup with that tool yesterday night and it correctly updated the backup I previously did last month (he even just downloaded the new and changed notes). Just great
I guess that integrating their syncing method or the tricks they are using would benefit this project a lot.
That makes me think that issues and inconsistencies might appear if someone is using nixnote and at the same time evernote 10 on other machines or on mobile clients |
Any source? I haven't heard of them changing anything. Considering that they still have lots of business API integrations, it seems unlikely such moves would be done, especially silently. What they could have actually changed is many other things which are present in note metadata but not in the note content (i.e. ENEX) itself - like all these reminder things and such. |
I have no source but I have written what is my direct experience. After many years of exclusively using the legacy version of evernote (under wine on linux), in the last 6 months I started using it alongside evernote 10 (on other machines), evernote android client, and evernote web. This didn't happen when I was using legacy evernote exclusively. |
It is nice.
I see that the API rate limit was often talked about on github before, if there are not too many notes, just waiting and restarting the sync can solve it. In recent years, I feel that the users are not as many as before, so I think the improvement on syncing will not be needed by too many people.
I am afraid so. |
What you said is reasonable, Evernote should be very serious about the format. But there are some posts about the note format on the evernote forum. For example, this link: https://discussion.evernote.com/forums/topic/148763-old-notes-bulleted-and-numbered-lists-messed-up/. To this kind of issues, I just guessed it was because of the format changes, because except the note format changes, I cannot think out other reasons. To find the source Evernote released, I searched just now, and found this link: https://evernote.com/blog/legacy-decommissioning#:~:text=Legacy%20versions%20of%20Evernote%20use,and%20update%20at%20lightning%20speed. Maybe the word I used and my understanding are not precise enough, but it seems that there is incompatibility between the legacy client and the latest one. |
the situation with the API syncing is making it very very difficult to switch to nixnote for any user that has more than 2k notes and above stored on evernote servers. |
Well, I don't know for certain either but I believe what has actually changed is not the ENEX format spec itself but the note editor implementation between legacy Evernote client and v10 and in how each of them interprets the format. ENEX in its core is just a stripped down XHTML which in turn is just a little more strict and XML-like HTML. So it all kinda boils down to how a web browser implements a rich text editor. V10 might use different HTML tags for lists than the legacy client used. But I don't think a change in ENEX spec was necessary for that to happen. |
Yes, it makes sense |
Yes, it is one of the reasons. On many sides, nixnote is not close enough to perfect. On the other hand, it is not an easy thing to make it a perfect software through working on it just in spare time. |
Of course, I was making a general point because I think it's a shame that a ready-made, working project is not easily accessible to a potentially very large group of users who are in need of an evernote client on linux. In all cases investing in a free project available to all is always a great act of generosity that should be appreciated |
I studied evernote-backup code, and I think we could have an easy solution for this problem. I'll try it, but I can't promise. |
|
Hi, wokawoka. I hope this reply is not too late. I made a PR for this issue and it may probably solve it, since I compared the logic of evernote-backup and nixnote, and removed tedious part of nixnote. For the details you can visit #223. It works fine for me, but I am not sure whether the rate limit exception will appear when there are thousands of notes with resource files. Could you please help to test it? |
|
If you'd like to, I can offer you a binary under Windows here: https://github.com/boo-yee/nixnote2/releases/download/v2.1.10/nixnote.zip. Update: And also for Linux there is appimage now. https://github.com/robert7/nixnote2/releases/download/continuous-develop/NixNote2-x86_64.AppImage |
|
this is very good news |
|
I feel very glad to receive your reply and appreciate your help. I think, if syncing all the notes takes too long time, syncing only part of your notes is enough, as long as there are no exception reported, for a longer period of time than previously. Syncing 2-3 thousand notes is enough I think. I created one thousand notes and synced them without exception. The only thing I am not sure is if there is any side effect after my changes. And a reminder for you for the sake of data safety: please do not let the new nixnote appimage mess up your old local database. |
|
yesterday night I tried to pull a full sync through the appimage you posted above but unfortunately I failed one again. |
It should be the code, not your mkstake. I think I have to ask you several questions. You said it stopped, did it report the rate limit exceeded exception? Or no exception reported, just the note list not updated? And do you know how long time around does it take to sync the first 800 notes? (I think the network speed may also determine the api calling frequency.) Do you think the new appimage peformed better than previous version? |
|
I never stepped into a "rate limit exceeded" error, the syncing each time just stopped pulling the notes (even if the syncing icon was still showing as spinning). |
|
If the rate limit exceeded exception does not appear, then the issue about the api should be fixed. As I was not sure if my changes would bring any side effects, I just focused on the rate limit issue, and left the other things alone, hoping to fix all the issues one by one.
For this issue, I think I have to continue to inspect the code to find the reason. (This issue is probably nothing about the rate limit exception. I think it is something about optimization. And the thread number is also another reason that nixnote is not as fast as evernote-backup, which uses multi-thread. So nixnote can be still improved.)
As the rate limit exception is fixed, we do not need to focus on the api and the network, I think. When I asked that question about the network speed, I thought the rate limit exception appeared again. However it did not in fact. |
|
The resources such as the images in the newly synced notes is correctly synced together, right? |
I actually didn´t inspect any note because I didn't want to interfere in any way with the syncing. |
Yes, it is just a download. Only syncing the notes from a fresh nixnote client will not affect the notes on the cloud server. As you synced the notes with evernote-backup earlier, just importing and using them is OK. Nixnote syncing is still slower, full sync will take too much time. For the very important notes, I think it's best to backup them well and carefully, as Nixnote is not reliable sometimes, one of my note's content once went out of order when I was using it. It is about charater encoding. It happend only once, so I didn't put it at the first priority. I also saw similar issues from another user: baumgarr#446 (comment) (I guess that it was because of the thread mutex not set correctly and not write back to the disk when exiting abnormally. And I'll work on this later.) |
|
'Restart sync on API limit (experimental)' works. |
|
If such error is normal - better to improve the message? |
I have no idea how many notes you have. Unless you have large amount of notes, the initial boost set by Evernote server is enough. It does not matter that how long nixnote has not been started, but the time since the new token is granted do. I read your log earlier, and I found that nixnote has synced 380 notes in only several minutes, when it received rate limit exceptions. The speed is OK without initial boost. These exceptions are from the server side, and all third-party applications have to follow the restrictions of the server. If you cannot finish the sync within initial boost, you have to stand with this exception, I am afraid.
I do not understand, but how to improve do you think? |
"API rate limit - its ok, NixNote will continue the sync in 62 minutes" |
Wow now I see all my notes, but NixNote crashed after a few clicks :( |
I disagree that the log message should be user friendly. Note that you get a message in nixnote2 GUI statusbar. If this one is not user friendly then that would be a bug. |
I guess a few resource files are missing (I mean things like pictures embedded into a note). You could check if only opening specific notes trigger the crash or if all the notes do so. The other bug is that resources can end up missing. |
These are permission errors from Evernote servers. Looks like the authentication token is not valid anymore and you need to reauthenticate. |
After a week you can have a lot of new notes to sync. The longer you wait the more likely you will hit the API limit. If you do not deauthenticate and reauthenticate you don't have the initial boost to sync a lot of notes this you will reach the Evernote server limit fast. Maybe nixnote2 could check how many notes there are to sync and ask if the user wants to reauthenticate to trigger the initial sync boost if there are too many notes to sync . This is obviously a feature request. |
Switch to a new language style? I think it is off the topic.
Without --logLevel=0? And what did you click apart from the sync button? |
During clicking on note titles - to see them. I did reauth - but still sync error. |
|
@vitaly-zdanevich Could you paste the whole log? (Sensitive parts can be masked.) There is no exception included in it. The second to last log reports EDAMUserException[PERMISSION_DENIED] for shared notes, so I guess that your access to these notes has some problems. |
|
@vitaly-zdanevich |
|
guid is 4cff8cea-f31a-44aa-7c47-a0d90bc89d6b? Or how to find it? With that guid I can view it in Evernote Web, this is not shared. |
Please run and you'll know the title. And then disable the sync for temporary, and click at that note, to see if Nixnote still crashes. |
|
Yes it crashes when I click to that note. |
If so, you could try to re-sync it. That is, backup it first, and then delete it on both Evernote web and nixnote, sync on nixnote, then restore it on the Evernote web, and finally sync on nixnote. |
|
Thanks, this note is not important - already in trash. But still on every sync I see this with error popup :( I tried to remove the token (from GUI) and login again, multiple times. |
|
From the log, I see that when downloading a note with the title of 'evernote', the exception is thrown. Is this note shared? |
Yes, I have a note with such title - shared to me - from another account. |
It is accessible via Evernote web, right? What is the permission for this note in this account? Read only or read and write? If it is read only, I think that you could try to change it to read and write, and try again, to see if this exception is thrown. |
|
Already I can edit it |
Are these shared notes put in one non-shared notebook or shared notebook? I have tested and found that if the shared notes are put in non-shared notebook, nixnote will not sync, and so no exception is thrown. If they are put in shared notebook, the sync will work well. So I think you could try to move the shared notes to a standalone or existing notebook if there are not too many, and share that notebook with your secondary account, and try again. |
|
Hi, are the shared notes synced now? |
|
I deleted |
How is the resync going? |
|
|
The server closed the connection, just syncing again is OK. |
|
Again
looks like finally it fetched all notes. |
Including the shared notes? About the exception, did you try to share the notebook in which the shared notes are located with your secondary account? |
Expected vs. actual behavior
After installation and having granted a fresh token I expect to be able able to download and sync my evernote database (around 17k notes) locally on nixnote2.
Unfortunately the sync get interrupted for an API error.
I installed nixnote2 on this machine just yesterday, the token is from yesterdays, to my knowledge the API limits should be initially lifted, instead this is what I find in the log:
Restart sync on API limit (experimental) is flagged in my preferences.
Do you have any suggestion?
Thank you so much!
Specifications
2.1.10ubuntu22.04.1The text was updated successfully, but these errors were encountered: