The purpose of SSIS Integration Services is to solve complex business problems by copying or downloading files, sending e-mail messages in response to events. In the process you update data warehouses and perform data cleaning and data mining.
We have tremendous experience to handle large scale of data. Import large amount of data to MS SQL server while using SSIS SQL server. We are handling hundreds of gigabyte of data this way. We are dealing with terabytes of data on a daily basis. The import is the end result of our efforts to implement data cleansing and De-Duping - Remove duplicate records. The challenge is to make sense of your data and also making the data meaningful and readable while not including any junk data.
The ability to extract corporate data from various sources and transforming it into meaningful information is essential to ensure competitiveness and helping IT departments of enterprises to analyze the past and lay the foundation for the future. Our highly experienced Business Intelligence Development team has been involved in many packages development assignments.
We have developed different techniques that find out the best record of any duplicate record while combing certain data from different rows into the one clean record. Say you want to De-Dupe (remove duplicate record) records based on First, Last, Email, City and State, and our program found 3 records with the exact data. Now the first record, the Phone and the DOB fields are empty, the second record has a Phone number filled and the 3rd record has the DOB data filled. Our process will remove the 2 other records while integrating the Phone and the DOB into one complete good record. This way you are improving your data quality by far and do not lose any valuable data.
We developed SQL routines combined with SSIS SQL server to update hundreds of GB of data in a minimum time.
The update is done very carefully. Say you have 30 million new records that you want to add or Update to current table. Based on certain rules will update the table and if it is a new record simply will add it.
If the record is found based on certain fields such as: First Name, Last Name, Email, City and State, then the Phone or DOB will be copied if are found empty. Obviously the source for these fields is coming from a duplicated record but from a different data source.
This way you maximize the quality of your data with a cutting edge technology. There are many other scenarios that we dealt with and provided a full automated solution using SSIS SQL server. We explained above the data cleansing process that could be implemented for any future projects that you may have.