diff --git a/README.md b/README.md index 8a8324b..79a7471 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,10 @@ You can build the SQL script before making the comm/user though. - - since right now it just changes the upvotes to be negative or whatever the score is +- Remove the json fields from comments and posts that don't get used for importing, so that the final array of trees of posts/nested comments takes up less memory/space. + +- Save the final json to a file and read it through StreamArray, so that the memory is freed once the archive has finished processing + ## references https://github.com/mesmere/RedditLemmyImporter (basically stole the sql stuff from there) @@ -42,4 +46,4 @@ https://tanishiking.github.io/posts/count-unicode-codepoint/ https://exploringjs.com/js/book/ch_unicode.html -https://www.reddit.com/dev/api/ \ No newline at end of file +https://www.reddit.com/dev/api/