To database: pd.Dataframes + additional data

Using Django Using the ORM
1

Hello everyone,

I’ve got a fileupload that accepts up to 8 csv files from a datalogger (same format, data varies). Using pandas each file is split into 3 different DataFrames belonging to a respective table in the postgresql db.

2 of these DataFrames are complete as in all data necessary is read from the csv files.
1 DataFrame is incomplete and I want to render a form to allow adding the missing data.

Now I’m wondering how this can be handled best in terms of pushing all the data to my database.

Do I just use the DataFrame.to_sql() function on all 3 DataFrames and create a form+model to add columns and their respective data to the table that holds the data of the incomplete DataFrame?

Or is it actually more practical to use the form to complete the DataFrame and use DataFrame.to_sql() once it’s completed?

Or would you advise taking a different route altogether.

I hope I articulated my issue clearly, I’m obviously no professional (:

2

yeah second option makes more sense, get your dataframes ready then input everything in one go.

This ensures that when creating your schema it will be designed for complete data, you don’t have to allow for nullable columns and empty fields or relationships just cause they are yet to be inputted, and have to go back and undo them later.