The Web Lab Hadoop Cluster: Configuration

The Web Lab has 60 Dell servers that are used for the Hadoop cluster and related purposes. Each node has the following hardware configuration:

• Quad Core Intel Xeon Processor, 2.66GHz, with 2x4MB Cache (total 8 core)
• 16 GBytes 667MHz memory
• 6 750GB SATA disk drives (total 4.5 TByte)
• Port Summit X450a Ethernet Switch (1 gigabit/sec)

As of October 2008, these nodes are configured as a large cluster of approximately 50 nodes (wl01.cac.cornell.edu), and a small cluster of 6 nodes (wl52@cac.cornell.edu), with the remaining nodes used as file servers and spares.

The policy is to run the most current stable release of Hadoop. New software releases are usually installed first on the small cluster.

When a user is authenticated to use the Hadoop cluster, the local directory is set up on a separate file server (currently cacfs01.cac.cornell.edu) as shown in the following figure.

Last revised: October 2008