This tutorial shows how to combine four single storage servers (running Ubuntu 12.04) to a distributed replicated storage withGlusterFS. Nodes 1 and 2 (replication1) as well as 3 and 4 (replication2) will mirror each other, and replication1 and replication2 will be combined to one larger storage server (distribution). Basically, this is RAID10 over network. If you lose one server from replication1 and one from replication2, the distributed volume continues to work. The client system (Ubuntu 12.04 as well) will be able to access the storage as if it was a local filesystem. GlusterFS is a clustered file-system capable of scaling to several peta-bytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. Storage bricks can be made of any commodity hardware such as x86_64 servers with SATA-II RAID and Infiniband HBA.
1 Preliminary Note
In this tutorial I use
five systems, four servers and a client:
·
server1.example.com: IP address 192.168.0.100 (server)
·
server2.example.com: IP address 192.168.0.101 (server)
·
server3.example.com: IP address 192.168.0.102 (server)
·
server4.example.com: IP address 192.168.0.103 (server)
·
client1.example.com: IP address 192.168.0.104 (client)
Because we will run
all the steps from this tutorial with root privileges, we can either prepend
all commands in this tutorial with the stringsudo, or we become root right now by typing
sudo su
All five systems
should be able to resolve the other systems' hostnames. If this cannot be done
through DNS, you should edit the/etc/hosts file so that it looks as follows on all
five systems:
vi /etc/hosts
|
127.0.0.1 localhost.localdomain localhost
192.168.0.100
server1.example.com server1
192.168.0.101
server2.example.com server2
192.168.0.102
server3.example.com server3
192.168.0.103
server4.example.com server4
192.168.0.104 client1.example.com client1
# The following lines are desirable for IPv6 capable hosts
::1 localhost
ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
|
(It is also possible
to use IP addresses instead of hostnames in the following setup. If you prefer
to use IP addresses, you don't have to care about whether the hostnames can be
resolved or not.)
2 Setting Up The
GlusterFS Servers
server1.example.com/server2.example.com/server3.example.com/server4.example.com:
GlusterFS is available
as a package for Ubuntu 12.04, therefore we can install it as follows:
apt-get install glusterfs-server
The command
glusterfsd --version
should now show the
GlusterFS version that you've just installed (3.2.5 in this case):
root@server1:~# glusterfsd --version
glusterfs 3.2.5 built on Jan 31 2012 07:39:58
Repository revision:
git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc.
<http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under
the terms of the GNU General Public License.
root@server1:~#
If you use a firewall,
ensure that TCP ports 111, 24007, 24008, 24009-(24009 + number of bricks across
all volumes) are open onserver1.example.com, server2.example.com, server3.example.com, and server4.example.com.
Next we must add server2.example.com, server3.example.com, and server4.example.com to the trusted storage pool (please note
that I'm running all GlusterFS configuration commands from server1.example.com, but you can as well run them fromserver2.example.com or server3.example.com or server4.example.com because the configuration is repliacted
between the GlusterFS nodes - just make sure you use the correct hostnames or
IP addresses):
server1.example.com:
On server1.example.com, run
gluster peer probe
server2.example.com
gluster peer probe
server3.example.com
gluster peer probe server4.example.com
Output should be as
follows:
root@server1:~# gluster peer probe
server2.example.com
Probe successful
root@server1:~#
The status of the
trusted storage pool should now be similar to this:
gluster peer status
root@server1:~# gluster peer status
Number of Peers: 3
Hostname: server2.example.com
Uuid: 600ff607-f7fd-43f6-af8d-419df703376d
State: Peer in Cluster (Connected)
Hostname: server3.example.com
Uuid: 1d6a5f3f-c2dd-4727-a050-0431772cc381
State: Peer in Cluster (Connected)
Hostname: server4.example.com
Uuid: 0bd9d445-0b5b-4a91-be6f-02b13c41d5d6
State: Peer in Cluster (Connected)
root@server1:~#
Next we create the
distributed replicated share named testvol with two replicas (please note that the
number of replicas is half the number of servers in this case because we want
to set up distributed replication) on server1.example.com, server2.example.com,server3.example.com, and server4.example.com in the /data directory (this will be created if it
doesn't exist):
gluster volume create testvol replica 2
transport tcp server1.example.com:/data server2.example.com:/data
server3.example.com:/data server4.example.com:/data
root@server1:~# gluster volume create testvol
replica 2 transport tcp server1.example.com:/data server2.example.com:/data
server3.example.com:/data server4.example.com:/data
Creation of volume testvol has been successful.
Please start the volume to access data.
root@server1:~#
Start the volume:
gluster volume start testvol
It is possible that
the above command tells you that the action was not successful:
root@server1:~# gluster volume start testvol
Starting volume testvol has been unsuccessful
root@server1:~#
In this case you
should check the output of...
server1.example.com/server2.example.com/server3.example.com/server4.example.com:
netstat -tap | grep glusterfsd
on both servers.
If you get output like
this...
root@server1:~# netstat -tap | grep glusterfsd
tcp 0 0 *:24009 *:* LISTEN 1110/glusterfsd
tcp 0 0 localhost.localdom:1019 localhost.localdo:24007 ESTABLISHED 1110/glusterfsd
root@server1:~#
... everything is
fine, but if you don't get any output...
root@server2:~# netstat -tap | grep glusterfsd
root@server2:~#
root@server3:~# netstat -tap | grep glusterfsd
root@server3:~#
root@server4:~# netstat -tap | grep glusterfsd
root@server4:~#
... restart the
GlusterFS daemon on the corresponding server (server2.example.com, server3.example.com, and server4.example.com in this case):
server2.example.com/server3.example.com/server4.example.com:
/etc/init.d/glusterfs-server restart
Then check the output
of...
netstat -tap | grep glusterfsd
... again on these
servers - it should now look like this:
root@server2:~# netstat -tap | grep glusterfsd
tcp 0 0 *:24009 *:* LISTEN 1152/glusterfsd
tcp 0 0 localhost.localdom:1018 localhost.localdo:24007 ESTABLISHED 1152/glusterfsd
root@server2:~#
root@server3:~# netstat -tap | grep glusterfsd
tcp 0 0 *:24009 *:* LISTEN 1311/glusterfsd
tcp 0 0 localhost.localdom:1018 localhost.localdo:24007 ESTABLISHED 1311/glusterfsd
root@server3:~#
root@server4:~# netstat -tap | grep glusterfsd
tcp 0 0 *:24009 *:* LISTEN 1297/glusterfsd
tcp 0 0 localhost.localdom:1019 localhost.localdo:24007 ESTABLISHED 1297/glusterfsd
root@server4:~#
Now back to server1.example.com:
server1.example.com:
You can check the
status of the volume with the command
gluster volume info
root@server1:~# gluster volume info
Volume Name: testvol
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: server1.example.com:/data
Brick2: server2.example.com:/data
Brick3: server3.example.com:/data
Brick4: server4.example.com:/data
root@server1:~#
By default, all
clients can connect to the volume. If you want to grant access to client1.example.com (= 192.168.0.104) only, run:
gluster volume set testvol auth.allow
192.168.0.104
Please note that it is
possible to use wildcards for the IP addresses (like 192.168.*) and that you can specify multiple IP
addresses separated by comma (e.g. 192.168.0.104,192.168.0.105).
The volume info should
now show the updated status:
gluster volume info
root@server1:~# gluster volume info
Volume Name: testvol
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: server1.example.com:/data
Brick2: server2.example.com:/data
Brick3: server3.example.com:/data
Brick4: server4.example.com:/data
Options Reconfigured:
auth.allow: 192.168.0.104
root@server1:~#
3 Setting Up The
GlusterFS Client
client1.example.com:
On the client, we can
install the GlusterFS client as follows:
apt-get install glusterfs-client
Then we create the
following directory:
mkdir /mnt/glusterfs
That's it! Now we can
mount the GlusterFS filesystem to /mnt/glusterfs with the following command:
mount.glusterfs server1.example.com:/testvol
/mnt/glusterfs
(Instead of server1.example.com you can as well use server2.example.com or server3.example.com or server4.example.com in the above command!)
You should now see the
new share in the outputs of...
mount
root@client1:~# mount
/dev/mapper/server5-root on / type ext4
(rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs
(rw,noexec,nosuid,nodev)
fusectl on /sys/fs/fuse/connections type fusectl
(rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs
(rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs
(rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs
(rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
server1.example.com:/testvol on /mnt/glusterfs
type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
root@client1:~#
... and...
df -h
root@client1:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/server5-root
29G 1.1G 27G 4% /
udev 238M 4.0K 238M 1% /dev
tmpfs 99M 212K 99M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 247M 0 247M 0% /run/shm
/dev/sda1 228M 24M 193M 11% /boot
server1.example.com:/testvol
58G 2.1G 53G 4% /mnt/glusterfs
root@client1:~#
Instead of mounting
the GlusterFS share manually on the client, you could modify /etc/fstab so that the share gets mounted
automatically when the client boots.
Open /etc/fstab and append the following line:
vi /etc/fstab
|
[...]
server1.example.com:/testvol /mnt/glusterfs glusterfs
defaults,_netdev 0 0
|
(Again, instead
of server1.example.com you can as well use server2.example.com or server3.example.com orserver4.example.com!)
To test if your
modified /etc/fstab is
working, reboot the client:
reboot
After the reboot, you
should find the share in the outputs of...
df -h
... and...
mount
4 Testing
Now let's create some
test files on the GlusterFS share:
client1.example.com:
touch
/mnt/glusterfs/test1
touch
/mnt/glusterfs/test2
touch
/mnt/glusterfs/test3
touch
/mnt/glusterfs/test4
touch
/mnt/glusterfs/test5
touch /mnt/glusterfs/test6
Now let's check the /data directory on server1.example.com, server2.example.com, server3.example.com, andserver4.example.com. You will notice that replication1 as well as replication2 hold only a part of the
files/directories that make up the GlusterFS share on the client, but the nodes
that make up replication1 (server1 and server2) or replication2 (server3 andserver4) contain the same files (mirroring):
server1.example.com:
ls -l /data
root@server1:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test1
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test2
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test4
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test5
root@server1:~#
server2.example.com:
ls -l /data
root@server2:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test1
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test2
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test4
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test5
root@server2:~#
server3.example.com:
ls -l /data
root@server3:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test3
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test6
root@server3:~#
server4.example.com:
ls -l /data
root@server4:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test3
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test6
root@server4:~#
Now we shut down server1.example.com and server4.example.com and add/delete some files on the
GlusterFS share onclient1.example.com.
server1.example.com/server4.example.com:
shutdown -h now
client1.example.com:
rm -f
/mnt/glusterfs/test5
rm -f /mnt/glusterfs/test6
The changes should be
visible in the /data directory
on server2.example.com andserver3.example.com:
server2.example.com:
ls -l /data
root@server2:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test1
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test2
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test4
root@server2:~#
server3.example.com:
ls -l /data
root@server3:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test3
root@server3:~#
Let's boot server1.example.com and server4.example.com again and take a look at the /data directory:
server1.example.com:
ls -l /data
root@server1:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test1
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test2
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test4
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test5
root@server1:~#
server4.example.com:
ls -l /data
root@server4:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test3
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test6
root@server4:~#
As you see, server1.example.com and server4.example.com haven't noticed the changes that
happened while they were down. This is easy to fix, all we need to do is invoke
a read command on the GlusterFS share on client1.example.com, e.g.:
client1.example.com:
ls -l /mnt/glusterfs/
root@client1:~# ls -l /mnt/glusterfs/
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test1
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test2
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test3
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test4
root@client1:~#
Now take a look at
the /data directory
on server1.example.com and server4.example.com again, and you should see that the changes
have been replicated to these nodes:
server1.example.com:
ls -l /data
root@server1:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test1
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test2
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test4
root@server1:~#
server4.example.com:
ls -l /data
root@server4:~# ls -l /data
total 0
-rw-r--r-- 1 root root 0 2012-05-29 15:49 test3
root@server4:~#
Version
1.0
Author:
Falko Timme
source
: howtoforge
