Business Intelligence Lowdown

Data Reduction Techniques

Organizations are struggling to maintain quality information, what with the vast amount of data floating around in all their departments. Effective data storage and data management techniques involve reduction of the data by minimizing the storage capacity or the bandwidth used. A few data reduction techniques are:

• Single-Instance Storage: which are used by e-mail applications to store only one instance of multiple copies of the same e-mail or attachment.
• Content Addressed Storage (CAS): which are systems used in archival of data to ferret out identical files or chunks of files and then minimize the number of copies that need to be stored.
• Wide Area File Services (WAFS): which try to minimize bandwidth used by transmitting only the bytes that have changes between multiple site servers.
• Backup Agents: which are used to reduce the amount of data sent by backup clients to the backup server by sending only parts of files that have changed since the last backup.
• Disk-based Backup Systems: which are used to maintain backup data over long periods of time with only the minimum amount of disk capacity.

Xerox Unveils Two Applications

DocuShare CPX and DocuShare 5.0 are the latest enterprise content management software solutions from the Xerox Corporation, which has used a single technology platform as the foundation for both applications. While DocuShare CPX will help users share data, and work together on important processes, DocuShare 5.0 features core enterprise content management capabilities. Both applications will help organizations manage their documents and other content in accordance with the regulations set by various compliance agencies. Database Trends and Applications reports:

Unlike competitive ECM products, DocuShare software is entirely Web-based; works across multiple operating systems and can integrate with enterprise infrastructure databases from Oracle, IBM and Microsoft.

What Compliance Tools Comply

With all the hue and cry over compliance with mandatory regulations, how many organizations have bothered to update and upgrade the tools used for the process? Not many, if you look at a survey conducted by Unisphere Media. The report, Beyond Compliance: 2006 OAUG Compliance Issues Survey, sponsored by Teradata, reveals that 72 percent of companies still use spreadsheets for their compliance management requirements, 66 percent use the reporting tools that come bundled with their ERP applications, while a surprising 43 percent still use paper-based aspects to meet compliance requirements. Only 33 percent use advanced technology like data warehouses as part of their compliance process, with an even smaller 25 percent deploying business intelligence tools for the task.

Temtec Strikes Deal With Microsoft

Microsoft and Temtec, provider of intuitive self-service analysis and reporting business intelligence software, have signed an agreement under which Microsoft will market a private label version of Temtec's Executive Viewer along with its analytical platform. The software will facilitate easy access and analysis methods for users who access data stored in Microsoft Analysis Services. Data Warehouse News reports:

Temtec, a Microsoft Gold Certified partner, said the partnership will allow it to extend its presence into the mid market substantially and add value to the analytical capabilities of Microsoft's solution. Microsoft says the agreement with Temtec will provide its customers with a cost effective analytics solution that will allow them to amplify the performance of people in an organization, at all levels.

USB Ports Pose Threats

Gone are the days when organizations had to worry about data being compromised through the use of floppy disks and CDs. A new threat looms over the horizon in the form of plug and play devices that can hold a large volume of data, are portable, and can be connected in a jiffy to USB ports. Flash drives and iPods are increasingly being used as intermediate storage devices to cart around data and information. The drives pose theft risks where data can be stolen easily and malware menaces by acting as conduits to transfer viruses, Trojans, and the like.

This alarming trend is driving many companies to enforce encryption techniques and password protection methods to protect their data. Another method being used to prevent misuse of data is to monitor the usage of the USB ports. Reporting software is then used to list out what is copied onto or off a device. With hardly 10 percent of companies enforcing strict policies regarding USB port devices, there is a high probability of this avenue being used to launch security attacks and steal sensitive data.

Data Protection Pays

Continuing on the lines of my previous post where we saw how important data is to an organization, let's look at ways we can protect onsite data.

• The first step to clean data is good organization. Don't be an electronic packrat. Clean out that clutter, get rid of files and emails that you don't need.
• Constantly update your anti-virus software, spyware, and firewall on your system.
• Put in place effective procedures to encrypt and back up your data to a remote, secure location.
• Take a regular review of who has access to what data, and incorporate the authorization changes as and when they happen to prevent intentional misuse of data.
• Change your passwords randomly.
• Keep your recovery and original software disks in a safe and easily accessible place to ensure quick recovery from system crashes.

Backing Up Your Data

No matter how small or big your company is, your data is one of the most valuable assets it owns. So it makes sense that you provide it adequate protection. A good data backup solution is a sound investment in most situations. But before you spend those hard-earned dollars, make sure of what you are actually getting.

• If you are backing up your data using a service organization, find out if the process is done automatically or if you have to check if the copies have been made.
• Ensure security of your data during transfer by suitably encrypting it.
• Make sure that your backed up data is safe from threats like theft, fire, and natural disasters.
• Find out the total cost involved in the backup process.
• Know how to recover and access the data that you backed up, and just in case, keep a list of people you can call if you encounter unforeseen problems.

Information Classification And Management

Any organization looks to proper information lifecycle management (ILM) as a way to reduce costs and improve efficiency. The first and most important step in the ILM process involves the identification and classification of unstructured data. Information Classification and Management (ICM) seeks to structure and organize data based on the following criteria:

• Identifying files and information that need to be stored indefinitely, and associating the appropriate back up storage devices.
• Ensuring that confidential and private information like identification social security data are adequately protected from unauthorized users.
• Creating search and retrieve methods that ensure quick access to relevant data and information as and when the need arises.
• Identifying and eliminating redundant and corrupt files so as to correctly assess the capacity of archival and back up storage that will be needed.

Business Objects Buys Data Quality Provider

Business Objects SA has moved closer to providing its customers with data quality management solutions along with its data integration software. The business intelligence software provider has signed a deal to acquire data quality software vendor Firtstlogic Inc. for US$69 million. IT World reports:

Business Objects said that it expects the acquisition to make it more competitive because customers are looking to standardize on a single platform that can deliver a complete information management tool.

Master Data Management

Organizations are being compelled by regulatory demands to define and implement data governance processes that ensure that the definition, quality, security, usage, and management of data are consistent across all business units and IT departments. But while most companies have established data governance programs, there are very few who have taken steps to include master data management (MDM) in these processes. A survey of 220 companies conducted by Ventana Research has found that while 59 percent of them have implemented data governance programs, only 26 percent have proper MDM procedures in place. As a result, very few have total control over their data.

In the data governance process, a team is created to document the lifecycle of information, from the creation of data, to its consumption. The team is also responsible for investigating how data is being reviewed, copied, manipulated and shared, and for identifying the information technologies used to carry out these manipulations on data.

MDM involves integrating data by creating and maintaining common definitions and contexts of data, including data usage, security, and quality. In order to take control of their data, organizations should implement standardized data definitions and a centralized store of key data with mechanisms to check and maintain accuracy, consistency and integration, across the enterprise.

Maintaining Data Consistency

Though firms are giving the highest priority to the quality, consistency, and integrity of data these days, the recovery of data during disaster management manoeuvres still poses a problem. This is because data that is held on standby for emergency purposes is usually not consistent with the data in the production database. The release of Veridata 1.0 promises to ensure consistency and eliminate discrepancies between an organization's standby and production databases. The product comes from the transactional data management (TDM) vendor, GoldenGate Software, and compares source and target data in relational databases to ensure that they are identical. Biz Intelligence Pipeline reports:

Veridata has three components: Veridata server, client agent(s) and the Web/command interface. The server processes comparisons of data served by client agents, which reside on the database servers and connect to the production and standby data sources. Users configure, execute and report on comparisons using a command-line interface or the simple Web interface.

Effective Data Management

Data is the lifeblood of any organization. But to serve a useful purpose, it has to be accurate, consistent and properly managed. A good strategy to manage your data quality will help in optimizing profitability and costs. Effective data management should encompass data profiling, data quality, data integration, data augmentation, and data monitoring. The data should be scanned for errors, discrepancies and duplication, which should then be corrected and rectified. Data from various sources should be integrated and then enhanced. The data thus gathered and merged should be monitored from time to time to prevent inconsistencies and redundancies from creeping in.

Cognos Ties up with Similarity Systems

With data quality being a top priority for business intelligence professionals, the tie-up between Athanor from Similarity Systems and Cognos 8 from BI corporation Cognos seems to make perfect sense. Athanor will provide users methods to monitor data quality and deal with data problems at their source. Data quality metrics from Similarity Systems can be scrutinized using Cognos 8. Business Intelligence Pipeline reports:

Data quality remains a serious challenge for many BI practitioners. Most surveys of business intelligence professionals find poor data to be a top impediment to the generation of meaningful analytics.

BI and Data Quality - Hand in Hand

Data quality technology is silently emerging as a necessary complement to business intelligence (BI) applications. The need for clean data when running BI was well established when SAS Institute bought out data quality vendor DataFlux Corp. to complement its BI products.

The LexisNexis Group, which uses DataFlux's schemes and verification processes to eliminate data overlapping and duplication and to project a single view of its customers from all data sources available, has seen benefits from the combined training materials and documentation of DataFlux and SAS.

Data quality issues have a way of cropping up in a BI system, which means that data quality tools need to be in place to process the data before it goes into the decision engine, says Keith Gile, principal analyst with Mass.-based Forrester Research Incorporated, on Search Data Management. He adds that though the SAS approach __ that everything, including modeling, extract, transform and load (ETL) tools, and analysis, comes under the umbrella of BI __ is not necessarily shared by everyone, for companies looking at a wider data management strategy, data quality is vital.

Data Quality goes from Bad to Worse

In a poll conducted by Business Intelligence Pipeline in October 2004, readers were asked to rate the overall quality of their enterprise data for analytical purposes, as excellent, good, fair or poor. The responses were not too spectacular; around 19 percent awarded an "excellent" rating to their data quality, nearly 12 percent said it was "good", over 50 percent said their data was of "fair" quality, and only 19 percent rated their data quality as "poor".

A similar poll conducted towards the end of 2005 has shown the previous year's results in good light. An astounding 30 percent rated their data quality "poor", which Business Intelligence Pipeline deems, "often worthless for BI purposes". Only 4 percent said their data quality was "excellent". The number of "fair" ratings also slipped to 43 percent. The only consolation was in the "good" category, with 23 percent of readers voting that their data quality was good.

Keeping data safe and maintaining operational security

Data quality has to match up with the exacting standards that governing organizations set, particularly, financial institutions, where collecting and studying loss data is a high priority activity. Loss data is important with respect to Operational Risk and the development of regulatory capital requirements. Even though data loss is a big risk that financial institutions face, it can be controlled. The first step is to educate the staff, be it executive, managerial, IT, etc. Back-up mechanisms, disaster-recovery plans, and, proper training for the end users is essential. Human error is frequently responsible for data loss, which can be categorized as occurring due to not following procedures or lack of education. For a better understanding of Operational Risk, financial institutions should share the loss data and have a common set of metrics and definitions. Operational Risk Management involves data tracking, collection, and reporting so that the key risk indicators are identified with the tracking and monitoring of historical and current data. Businessintelligence reports:

However, the main challenge for the banks in the coming months will be to spot and locate the required data and build up a data storehouse or data warehouse for storing the ready data for analysis. Moreover, banks must ensure effectual integration of different risk types and accurate computation of the various risk measures.

Read More: Measuring Operational Risk: The Data Challenges