-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The 25 items issue #4
Comments
Hello @dm-grinko, allow me to ask you a few questions to help me understand what's happening:
Thanks |
|
Thanks @dm-grinko. It seems that there's a limit on the number of items that you can send in one request (25 top: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) You should split your source file into multiple files with 25 objects (max) each. It might be nice to have this feature supported in the tool, would you have some time to contribute with a PR? |
I have found another solution. I have created nodejs app where I am using
|
Thanks @dm-grinko, |
Could you please add an option to this library to break down the output files to some set of files that contains each 25 items? |
I improved @dm-grinko solution as follow:
|
I did some modifications to your code @mjza Copy the code below and save it as dynamo-chunk.js Now you just need to pass the file converted by this library (json-dynamo-putrequest) and pass the table name. It'll create the chunk files and automatically upload to dynamodb into your aws account. const fs = require("fs");
const shell = require("shelljs");
function processFile(filename, table) {
fs.readFile(filename, "utf8", (err, data) => {
if (err) {
console.log("Cannot parse", err);
process.exit(1);
}
const items = JSON.parse(data);
let counter = 0;
let req = {};
let str = "";
req[table] = [];
items[table].forEach((item) => {
counter++;
req[table].push(item);
if (counter % 25 === 0) {
str = JSON.stringify(req);
fs.writeFileSync("./tmp.json", str);
console.log(`Write ${counter / 25}th 25 items.`);
const exec =
"aws dynamodb batch-write-item --request-items file://tmp.json";
shell.exec(exec);
req[table] = [];
}
});
if (counter % 25 > 0) {
str = JSON.stringify(req);
fs.writeFileSync("./tmp.json", str);
console.log(`Write ${counter % 25} rest items.`);
const exec =
"aws dynamodb batch-write-item --request-items file://tmp.json";
shell.exec(exec);
req[table] = [];
}
});
}
if (!process.argv[2] || !process.argv[3]) {
console.log("Missing arguments");
console.log("Usage: node dynamo-chunk.js file tableName");
process.exit(1);
}
processFile(process.argv[2], process.argv[3]); |
I'm getting this error.
Member must have length less than or equal to 25, Member must have length greater than or equal to 1.
Any idea how to handle it? =)
The text was updated successfully, but these errors were encountered: