# processing of large input files - CSV or JSON ? **URL:** \ **Category:** Forms & APIs\ **Created:** [March 21, 2023, 7:45am UTC](https://forum.djangoproject.com/t/processing-of-large-input-files-csv-or-json/19688 "2023-03-21T07:45:21Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![anjanesh](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/anjanesh/32/4583_2.png) [@anjanesh](https://forum.djangoproject.com/u/anjanesh)\ **Post date:** [March 21, 2023, 7:45am UTC](https://forum.djangoproject.com/t/processing-of-large-input-files-csv-or-json/19688/1 "2023-03-21T07:45:21Z")
I have a [3GB CSV file](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.csv) and a [4GB equivalent JSON file](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.json). Which is best for loading & processing this data to a MySQL table ? I can load this data using MySQL command : ```auto LOAD DATA LOCAL INFILE index.csv REPLACE INTO TABLE `aws-pricing` FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\\\' LINES TERMINATED BY '\\n' IGNORE 6 LINES; ``` I need to filter them in code - I don’t require all 4 million rows. And definitely not all 91 fields. ---
**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\ **Post date:** [March 21, 2023, 11:44am UTC](https://forum.djangoproject.com/t/processing-of-large-input-files-csv-or-json/19688/2 "2023-03-21T11:44:04Z")
The CSV file is going to create a lesser load on the system, the JSON file would generally be processed faster - take your pick. You can process the CSV file one line at a time. You only need as much memory as what’s required for that “current” row. A JSON file must be loaded in its entirety. You’re going to create a memory object roughly the size of the complete file. ---
**Author:** ![anjanesh](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/anjanesh/32/4583_2.png) [@anjanesh](https://forum.djangoproject.com/u/anjanesh)\ **Post date:** [March 21, 2023, 11:59am UTC](https://forum.djangoproject.com/t/processing-of-large-input-files-csv-or-json/19688/3 "2023-03-21T11:59:11Z")
I guess I’ll pick CSV then - as the Azure VM is on a 1/2 CPU and 4/8 GB RAM. I need to dump this newly generated (final) MySQL table (~1M rows consuming ~200MB) in a Redis cache and hence thought of using JSON for `json.dumps`. ---
**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\ **Post date:** [March 21, 2023, 12:05pm UTC](https://forum.djangoproject.com/t/processing-of-large-input-files-csv-or-json/19688/4 "2023-03-21T12:05:01Z")
Are you talking about a separate serialized JSON object for each row? Or are you talking about a single JSON object for the entire table? If the former, then that would be a file you could process one line at a time. It’s only the latter case where the entire table would become memory-resident. ---
**Author:** ![anjanesh](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/anjanesh/32/4583_2.png) [@anjanesh](https://forum.djangoproject.com/u/anjanesh)\ **Post date:** [March 21, 2023, 12:15pm UTC](https://forum.djangoproject.com/t/processing-of-large-input-files-csv-or-json/19688/5 "2023-03-21T12:15:36Z")
> [@KenWhitesell](#): > > separate serialized JSON object for each row 1. But what would be the `key` for each row ? `"aws-" + SKU` ? 2. If its a single JSON object wouldn’t it be just `"aws":[{ row1 }, { row2 }, { row3 },......{ row 1M }]` Which one would be lower in size for each of the above ? ```auto redis-cli info memory | grep 'used_memory.*human'; ``` ---
**Author:** ![KenWhitesell](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/kenwhitesell/32/280_2.png) [@KenWhitesell](https://forum.djangoproject.com/u/KenWhitesell)\ **Post date:** [March 21, 2023, 12:19pm UTC](https://forum.djangoproject.com/t/processing-of-large-input-files-csv-or-json/19688/6 "2023-03-21T12:19:26Z")
> [@anjanesh](#): > > But what would be the `key` for each row ? `"aws-" + SKU` ? I wouldn’t know. I don’t know your data or how you’re planning to use it in redis. > [@anjanesh](#): > > If its a single JSON object wouldn’t it be just `"aws":[{ row1 }, { row2 }, { row3 },......{ row 1M }]` Yes - so in order to load and parse it, it needs to load the entire `aws` list. > [@anjanesh](#): > > Which one would be lower in size for each of the above ? Within redis? You’d need to check that yourself. I have no idea what sort of compression redis may use internally for storing data and indexes. I suggest you try it both ways and see. ---
**Author:** ![anjanesh](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.djangoproject.com/anjanesh/32/4583_2.png) [@anjanesh](https://forum.djangoproject.com/u/anjanesh)\ **Post date:** [March 21, 2023, 12:59pm UTC](https://forum.djangoproject.com/t/processing-of-large-input-files-csv-or-json/19688/7 "2023-03-21T12:59:53Z")
I just realized the idea of a sing JSON object is a bad way to go because the entire value of the `aws` key, would be one large string containing a JSONified object which I would have to load using `json.load` which would need to parse the entire 1M rows from which I have to filter ! My bad ! One row per JSON object would be using redis to `get` a specific key.