Analysis: Data Integrity Monitoring Techniques in Cloud Storage

Cloud storage is a very attractive service for outsourcing day-to-day data management, but once the data is lost, all the consequences will be borne by the company that owns the data, not the hosting provider. With this in mind, it is important to understand the reasons for data loss, how much responsibilities a cloud service provider assumes, the basic methods for securely utilizing cloud storage, and integrity monitoring methods and standards, whether the data is stored locally or in the cloud.

Integrity monitoring is essential in cloud storage services. Similarly, data integrity is the core task of all data centers. Data corruption can occur at any level of storage and any type of media. Bit attenuation (decreased or lost data on the storage medium), controller failure, deduplication metadata corruption, and tape failure are the main factors that cause corruption of different types of media data. Metadata corruption is a direct result of such failures, such as bit decay, and is also extremely susceptible to software failures other than hardware error rates. Unfortunately, one side effect of deduplication is that corrupted files, blocks, or bytes affect each piece of metadata associated with it. In fact, data corruption may occur at any point in the storage. Migrating data to different platforms can easily be compromised by migrating data to the cloud. Cloud storage systems are also data centers made up of hardware and software that are also vulnerable to data corruption. For example, the recently known Amazon cloud machine downtime incident. Many companies are not only affected by long periods of downtime. In fact, 0.07% of their customer data has been lost. According to reports, the cause of data loss is "Amazon ESB volumes ... inconsistent data snapshot recovery." This means that the data in the Amazon system has been damaged, so the customer data has been lost. Whenever data is lost, especially important data is lost, people tend to blame each other to shirk responsibility. In the IT industry, this usually leads to the dismissal of workers, the company’s huge economic losses, and even the worst case of bankruptcy. Therefore, the key is to understand the legal responsibilities that cloud service providers must assume, and that every service level agreement (SLA) has taken all possible measures to ensure data security and prevent data loss. As far as many legal documents are concerned, most of the SLAs are biased towards the interests of the providers, not the interests of the customers. Many cloud service providers offer different levels of data protection, but all storage vendors are not responsible for data integrity.

The Cloud SLA agreement, including the protection of cloud providers, clearly shows that data loss or damage is the most common situation. Amazon’s customer web service agreement, for example, states that “we...does not provide any form of representation or warranty that the service or third-party content provided is uninterrupted, error-free, fault-free, or anything... It will be safe, not lost or undamaged." This agreement even suggests that customers "frequently archive" their data. As mentioned earlier, the integrity of data management, whether it is in the data center, private cloud, hybrid cloud, or public cloud, is always the responsibility of the actual owner of the data.

Some common best practices will allow companies to take advantage of the flexibility and accessibility of the cloud without compromising their data security. Disperse risks on the premise of data protection to minimize the possibility of data loss. Even if you are storing data in the cloud, it makes sense to keep a backup copy of your master copy and live data. In this way, accessing your data does not depend on network performance or connectivity. Stick to these basic best practices, understand the details of cloud service provider SLAs, and build modules to proactively monitor data integrity, whether it is stored in the cloud or locally.

One of the ways to verify the integrity of a set of data is based on hash values. A hash value is a unique value obtained by compressing a set of data in a predefined way. Since the hash value is obtained from the original data itself, if the two hash values ​​are not exactly the same, it means that at least one of the two copies has been changed or damaged.

Ensuring that cloud providers store copies whenever and wherever they want, provides a hash check of the data and compares it with the hash of the second data copy. Performing this level of data monitoring manually will be very tedious. Fortunately there are other methods available, including headline checking. SpectraLogic and other ActiveArchiveAlliance members provide automated monitoring of data integrity within the system.

Although dynamic archiving is one of the methods to monitor the integrity of data, it still requires the currently widely used cloud standard protocol to support its integrity monitoring and interoperability. Because not all data center or cloud hosting infrastructures use the same standard equipment, interoperability between different storage devices is critical. The Cloud Storage Management Interface (CDMI) standard was proposed by the Global Network Storage Industry Association (SNIA) in 2010. A CDMI-compliant system may query the hash value of another CDMI compatible system's object to verify that the two data copies are the same. By monitoring the integrity of the master data copy and the backup copy, the enterprise can confirm whether the data copy stored in the cloud is damaged. These data sets can be monitored frequently by data values. Industry standards such as CDMI not only ensure interoperability among heterogeneously compatible systems, but also provide a convenient mechanism for data integrity monitoring.

It's hard to see the cloud industry appearing in the media recently, especially after Iron Mountain has stopped its most basic cloud storage service and pre-discussed Amazon downtime. However, the purpose of this article is not to discuss whether the cloud storage platform is wise, but rather to consider more factors when researching and implementing a cloud strategy, rather than simply considering the storage cost per GB. If cloud storage is implemented correctly, it will provide many benefits to all enterprises. Eliminating cloud disadvantages requires intelligent data management strategies. Regardless of where or how data is stored, it is absolutely crucial that it be accessible and recoverable when needed. This commitment is the core task of all data integrity monitoring and verification.