If two Django applications are running on different servers, how do I synchronize the data from application A to service B. There are primary key conflicts and foreign keys.
Application A needs to synchronize data to application B from time to time. The two applications can communicate only for an indefinite period of time. Could you give me some advice on what to do, please
The term “synchronize” actually has a couple of different meanings depending upon the degree of synchronization required.
At one end of the extremes, you could be talking about a database-level synchronization where every table is replicated between the two systems such that one could be swapped out for the other and neither would be able to tell that anything had changed.
At the other end, “synchronization” is simply referring to some tables having some data that is maintained the same on both systems.
Additionally, you need to identify which system is going to be the “source of record” in case of conflict. If both systems are updated, which changes apply?
So the first thing to do is to identify the degree to which the two need to be synchronized.
Then you imply that the two systems are not in constant communication. Obviously, this creates quite a bit of friction when trying to do this. There are architectural decisions that may change depending upon how frequently the two systems can exchange data, and whether or not you can schedule activities in advance of such connections.
So the second step is to ensure that you have an accurate understanding of the constraints involved in this process.
How frequently are the two systems connected? How long are those two systems connected when they are connected? What’s the bandwidth between systems when they’re connected? What’s the amount of data to be transferred during each window of connectivity?
Are you specifically referring to exactly two systems that need to do this? Or is this possibly the case of having multiple systems coordinating with a “common core system”?
Do you have sufficient access and control over both applications to be able to implement any required processes to perform this? (This becomes a significantly more difficult issue if you can’t coordinate efforts on both ends.)
This is one of those “hard problems” for which there is no quick and easy solution, and certainly no “one-size fits all” answer to be provided.
First of all, thank you very much for your reply.
My application is an internal service, and one of the services needs to be deployed in a virtual machine on the PC side to provide services at the construction site. After the task is completed at the construction site, the generated data needs to be imported into the company’s internal service host. Only when both servers are on the same intranet can they communicate with each other. The current problem is that both services will generate data, and there will be primary key conflicts and foreign key problems when importing them. I have not come up with a good solution so far.
Then my recommendation is that you hire a consultant to properly guide you with this.
As I mentioned earlier, this is not a trivial topic and not something you’re going to be able to just put together without a lot of thought and planning to handle all the edge cases that will arise with something like this. A full solution is going to require more than just “importing data” from one source to the other.
Thank you for your advice.
To resolve the problem of primary key conflict you can use UUIDs rather than integers. Typically a replication setup between several hosts running your database (eg MySQL) will resolve the issue of primary key (as integers) conflict by assigning even values to one host and odd to another (or some other scheme where there are more hosts).
It is not rocket science, but can become quite complex.