I am sure many people reading this article will have worked on one, or probably many, projects which involve migration of data from one IT system to another. One of the things that the data migration elements of these projects tend to have in common is that they are painful. It is not that migration of data itself is always tricky, it is more that the quality of data on the source system tends to be an issue. Data cleansing and data mapping can be tricky, particularly when there has been poor data governance and stewardship. In a worst case scenario, we might even find fields being used for different purposes by different teams, or there might be ‘debates’ over what specific data items actually mean (“Order date is the date the order is received”. “No! Order date is the date that the order is accepted, which means payment has also been made”).
There are so many issues that we could discuss, I’d like to zoom in on one: the granularity of data.
What Does ‘Granularity of Data’ Mean for Migration?
This is probably best illuminated with an example. My home insurance was recently up for renewal, which I purchased via a large online insurer. Technically, they are a broker rather than an insurer—but you wouldn’t know that to look at them (as the policies are branded with their logo, and you have to look pretty hard to find out which insurer actually underwrites them).
Close to the renewal date I shopped around and found some slightly better prices . I rang my insurer to see if they could price-match, and I was surprised as they asked me for a whole bunch of information that they already had on file. Concerned, I asked why, and the agent explained to me that they had migrated from one IT system to another, and he no longer had access to the ‘legacy’ data.