From 795edd7141776e76e1b19ab25ff3f8b70fa9c8e7 Mon Sep 17 00:00:00 2001 From: starlight Date: Sun, 19 Jan 2025 08:20:24 +1300 Subject: [PATCH] add more to the todo list --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 8a8324b..79a7471 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,10 @@ You can build the SQL script before making the comm/user though. - - since right now it just changes the upvotes to be negative or whatever the score is +- Remove the json fields from comments and posts that don't get used for importing, so that the final array of trees of posts/nested comments takes up less memory/space. + +- Save the final json to a file and read it through StreamArray, so that the memory is freed once the archive has finished processing + ## references https://github.com/mesmere/RedditLemmyImporter (basically stole the sql stuff from there) @@ -42,4 +46,4 @@ https://tanishiking.github.io/posts/count-unicode-codepoint/ https://exploringjs.com/js/book/ch_unicode.html -https://www.reddit.com/dev/api/ \ No newline at end of file +https://www.reddit.com/dev/api/