Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ijod #914

Open
wants to merge 57 commits into
base: master
Choose a base branch
from
Open

Ijod #914

wants to merge 57 commits into from

Conversation

temas
Copy link
Member

@temas temas commented Mar 10, 2012

This implemens ijod for core synclet results.

A few of the highlights now are:

  • JSON data is tracked with a murmurhash3 to ensure we don't reprocess as much
  • Writes from the synclets are batch processed leading to a large performance gain.

There are currently two primary TODOs before looking to merge.

  • Look into data access patterns such as /getCurrent
  • More testing!

Please test this branch and see how it feels.

@temas
Copy link
Member Author

temas commented Mar 12, 2012

getCurrent is unlost and functioning.

@temas
Copy link
Member Author

temas commented Mar 15, 2012

This is now ready to merge and start testing on our sandbox. I'd appreciate any final reviews before I pull the trigger.

There is a verification tool in the Ops directory called checkCurrent.js. It is ran such as: node checkCurrent.js localhost and to test against ijod use: node checkCurrent.js localhost ijod. These are using the same calls, just different file naming. The list of synclets can be adjusted at the top of the file for testing based on available data sets. The results are put in a verification directory from where it's ran. There seems to be a small issue with trailing \n missing sometimes, but all of the data is the same, verified with sha1 sums and diff.

Thomas Muldowney and others added 18 commits March 15, 2012 22:26
This changes transactions so it has actual transaction rollback.  The
batch add defaults to using this with 5 attempts to land a commit.  I'm
going fail hard if those 5 attempts fail until I understand why it's
failing.
This stuff is much more transparent if we push it into the source tree,
rather than putting it in the jenkins config.  To the greatest extent
possible, Jenkins should just run "make jenkins" and let us take care of
the rest here.
@@ -27,36 +26,24 @@ module.exports = function(locker) {
});
});

// copy pasta from the synclet code, these should be utilizing some generic stuff instead
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmmm, pasta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants