Computers and Tech News

Non uniform memory architecture (NUMA) machines are not a new concept. The idea has been around for some years. Recent advances in hardware technology have made machines such as the Sequent NUMA a possibility. A NUMA machine is basically a tightly coupled cluster of SMP nodes, with an extremely high-speed interconnect. The interconnect needs to be sufficiently fast to give near SMP internal speeds. The advantage of such an interconnect is that it enables the operating system to run across the whole machine as a single distributed instance, rather than the usual separate copies of the operating system running on each node. In essence it makes the distributed memory on the SMP nodes of a NUMA machine act as if it were a single shared memory.

Another new technology is the High speed memory interconnect, such as the memory channel provided by Digital on its clustered UNIX systems. The memory channel allows a cluster of SMP nodes to act more as though it has one memory address space. It also removes the requirement for a shared disk to be connected to each node, because disks that are connected to one node can be shared by other nodes over the high-speed memory channel link.

Both of these new technologies overcome bottlenecks in the current cluster systems, thereby allowing much greater scalability of disk, memory and CPU on a clustered system. These technologies, and the continual improvement of disk and I/O technology, mean that the size of disk farm that cluster systems can support continues to grow. These systems are eating into the MPP size advantage, making the decision of whether to go SMP or MPP a little easier. For the development of a data warehouse the default choice currently has to be an SMP system, unless a very good reason can be given for choosing an MPP system.

Massively parallel processing is the “new kid on the block”. Like any. new technology, MPP has had its teething problems, not least of which is the lack of tools to manage, monitor and support these massive machines. This is starting to change as the MPP market shakes itself out. Most of the myriad MPP start-up companies have failed, leaving only a few serious-players in the MPP business. However, MPP has for the most part failed to get a toehold in the IT strategies of general business. This is likely to be compounded by the advent of the new SMP architectures that are emerging, which allow much greater SMP scalability. So, for the foreseeable future, MPP is likely to be concentrated in certain niche markets.

An MPP machine is made up of many loosely coupled nodes. These nodes will be linked together by a high-speed connection. The form that this connection takes varies from vendor to vendor. Each node has its own memory, and the disks are not shared, each being attached to only one node. However, most MPP systems allow a disk to be dual connected between two nodes. This protects against an individual node failure causing disks to be unavailable. These types of environment are generally referred to as shared nothing. There are many different models of MPP. Each will have nodes that consist of one or more CPUs. For example, some MPP machines have nodes with two CPUs, where one CPU is dedicated to handling I/O. Others have nodes that are genuine SMP machines in their own right. Yet others have single CPU nodes, where some will be dedicated to handling I/O and others to processing. However, all these configurations have one thing in common: nothing is shared directly.

It is important to establish early any security and audit requirements that will be placed on the data warehouse. Clearly, adding security will affect performance, because further checks require CPU cycles and time to perform. More importantly, and often overlooked, the security and audit requirements can severely affect the design of the data warehouse. It is very difficult to add security restrictions after the data warehouse has gone live, so it is important to capture the ultimate security and audit requirements at the beginning. Where possible, future-proof the data warehouse against changes. During the design phase take into account what other data sources may be added later and the likely impact of these additional sources.

Data can be classified in a number of ways. The actual approach taken to data classification will depend on the level of security required. Data may be classified by its sensitivity. So, for example, highly sensitive data such as payroll information may be classified as highly restricted, while less sensitive but still commercially important data such as general customer information may be classified as company confidential. In high-security environments there will generally be a data classification hierarchy already available.

At the design stage it may be necessary to consider what user-side tools will be used. If these tools have special requirements, such as data being summarized in certain ways, these requirements need to be catered for. Even given this requirement, it is important that the data warehouse be designed to be accessible to any tool. In fact the tool should not be allowed to affect the basic design of the warehouse itself. This protects against changing tools requirements, and is particularly important in the data warehouse arena, where requirements evolve over time.

No one tool is likely to meet all users requirements, and it is probable that multiple tools will be used against the data warehouse. The tools should be thoroughly tested and trialed to ensure that they are suitable for the users. This testing of the tools should ideally be performed in parallel with the data warehouse design. This will allow any usability issues to be exposed, and will also help to drive out any requirements that the tool will place on the data warehouse.

As long as the network has sufficient bandwidth to supply the data feed and user requirements, the architecture of the network is irrelevant. It is, however, important at the design stage to consider some aspects of the network architecture. For example, if user access is going to be via a wide area network (WAN), this needs to be considered in the design of parts of the application, such as the query manager. Each of these issues needs to be considered carefully. For example, consider the problem of getting the data from the source systems. Even if the network bandwidth is sufficient to cope with the data transfer on a daily basis, it may not get the data to the warehouse system early enough to allow it to be loaded, transformed, processed and backed up within the overnight time window.

All such issues that can affect the data warehouse design need to be bottomed out very early on. Data extractions are any requests that cause data to be transferred out over the network. Result sets of queries that are passed to the user either directly or via a temporary table in the database are examples to data extractions. Another common data extraction is a data mart. The question that has to be answered is - Will the network bandwidth be capable of supporting the load placed on it by the identifiable or likely data extractions?

Management of the clients is beyond the scope of the data warehouse environment. The dependencies here are the other way around. Those responsible for client machine management will need to know the requirements for that machine to access the data warehouse system. Details such as the network protocols supported on the server, and the server’s Internet address, will need to be supplied.

If multiple access paths to the server system exist, this information needs to be relayed to those responsible for the client systems. For example, if the server is a cluster environment, during node fallover users may need to access a different machine address. All such dependencies need to be driven out and documented.

While shared nothing gives MPP its scalability, it is also its Achilles heel. Because nothing is shared, MPP is a much more difficult environment on which to design and develop. Data location becomes crucial. To make use of the CPU power of the MPP system while avoiding vast amounts of data shipping, data needs to be spread across many nodes. Getting this data partitioning correct is critical to the success of the system. In a data warehouse where ad hoc queries are allowed, it is almost impossible to come up with a partitioning strategy that will allow all queries to be equally performant. Node and disk hot spots are almost inevitable. Even if you can get data distributed to give good performance, what happens when you want to add new nodes? How do you cope with new data sources, or new aggregations that require new disk space? These issues make design and management of a data warehouse on an MPP system considerably more difficult than on an SMP or cluster system.

Network management is a black art. It requires specialist tools and lots of network experience. The management of the network has no direct effect on a data warehouse, except for one issue. It is important to be able to monitor network performance. The network may play a key part in data flow through a data warehouse environment. Being able to monitor this part of the data flow is necessary to enable resolution of any performance problems. As with the network, clients are external to the data warehouse system itself. There are, however, still aspects that need to be considered during the design phase.

Like a cluster, MPP machines require the use of a distributed lock manager to maintain the integrity of the distributed resources across the system as a whole. The distributed lock manager needs to be able to track which node’s memory holds the current copy of each piece of information. It also needs to be able to swiftly satisfy any requests for that information, getting the data rapidly from node to node. It is primarily in the method of implementing this communication that the MPP vendors differ.

The advantage of MPP systems is that because nothing is shared they do not suffer the same restrictions as SMP and cluster systems. For example, because a disk can be shared on a cluster, each shared disk must be connected to each node that shares it. This connection takes up a disk slot on each node and thus limits the amount of disk that can be connected. In an MPP system the amount of disk that can be supported is limited only by the number of nodes that can be connected. There is great variability in these systems, with SMP/cluster solutions supporting anywhere from 1 to 50 TB, and MPP systems supporting from 10 to 100 + TB. Likewise, the amount of CPU power that can be made available in an MPP system is dependent on the number of nodes that can be connected. The scalability of the MPP system will depend on its internal connection method. The available systems vary radically, some scaling to 64 nodes while others scale up to thousands of nodes.

A cluster is a set of loosely coupled SMP machines connected by a high-speed interconnect. Each machine has its own CPUs and memory, but they share access to disk. Thus these systems are called shared-disk systems. Each machine in the cluster is called a node. The aim of the cluster is to mimic a single larger machine. In this pseudo single machine, resources such as shared disk must be managed in a distributed fashion. A layer of software called the distributed lock manager is used to achieve this. This software is run on each machine, and communicates over the high-speed interconnect. The distributed lock manager maintains the integrity of each resource by ensuring that the resource can be changed on only one node of the cluster at a time. The distributed lock manager and the interconnect become the factors that limit the scalability of the cluster itself.

The scalability differs from hardware vendor to hardware vendor, but clusters generally will scale only to somewhere between four and eight nodes. Clustering technology was first introduced by Digital on VAX VMS machines back in the 1980s. In the last few years the main UNIX vendors have introduced clustered systems. These systems have had many teething problems but have now begun to prove themselves. However, they still suffer from some fundamental issues that affect design. One of the main problems is the inability of the UNIX clusters to support proper file system sharing. Sharing is currently limited to raw devices. This is a nuisance, as it requires, amongst other things, the database software to be installed multiple times. However, this is already changing, as some UNIX vendors are now bringing out shared-file systems.