环境：centos8虚拟机（centos-release-8.1-1.1911）
方法：

• qemu-img、qemu-nbd创建磁盘，关于nbd可以参考文章linux qemu-nbd介绍
• 使用一个虚拟机来模拟两个节点

1. 安装软件包

• 安装glusterfs源

# yum install centos-release-gluster7

得到/etc/yum.repos.d/CentOS-Gluster-7.repo

• 重新创建yum源cache

# make makecache

• 安装glusterfs

# yum install -y glusterfs glusterfs-server

2. 配置glusterfs

• 启动服务

# systemctl start glusterd

• 配置域名解析

# echo "169.254.155.112 build-centos81" >> /etc/hosts
# hostname 
build-centos81
# hostname -i
169.254.155.112

• 配置域名可信任

# gluster peer probe build-centos81

由于只有一个节点，这个可以不配置。多个节点需要相互配置可信任域名

3. 使用qemu-img创建磁盘

• 创建一个1G磁盘并挂载

# qemu-img create -f qcow2 /home/gluster1.qcow2 1G   // qemu-img创建1G磁盘
# qemu-nbd -c /dev/nbd0 /homt/gluster1.qcow2         // qemu-nbd挂载
# mkfs.ext4 /dev/nbd0                                // 格式化磁盘
# mkdir -p /home/data1/brick1                        // 创建挂载点
# mount /dev/nbd0 /home/data1/brick1                 // 挂载磁盘
# mkdir -p /home/data1/brick1/gv0                    // 创建gv0目录，后面创建glusterfs会用到

• 创建一个2G磁盘并挂载，过程同上

# qemu-img create -f qcow2 /home/gluster2.qcow2 2G   // qemu-img创建2G磁盘
# qemu-nbd -c /dev/nbd1 /homt/gluster2.qcow2         // qemu-nbd挂载
# mkfs.ext4 /dev/nbd1                                // 格式化磁盘
# mkdir -p /home/data2/brick1                        // 创建挂载点
# mount /dev/nbd1 /home/data2/brick1                 // 挂载磁盘
# mkdir -p /home/data2/brick1/gv0                    // 创建gv0目录，后面创建glusterfs会用到

4. 创建glusterfs文件系统

• 创建volume

# gluster volume create gv0 replica 2 build-centos81:/home/data1/brick1/gv0 build-centos81:/home/data2/brick1/gv0 force

因为我们是在一个系统上模拟两个节点，这里要使用force参数，否则会出现下面的错误信息：

volume create: gv0: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Bricks should be on different nodes to have best fault tolerant configuration. Use ‘force’ at the end of the command if you want to override this behavior.

• 启动volume

# gluster volume start gv0

• 查看创建的volume信息

# gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: 7bf1b0e2-1dd2-44fe-a209-bc5ca430a350
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: build-centos81:/home/data1/brick1/gv0
Brick2: build-centos81:/home/data2/brick1/gv0
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off

• 挂载glusterfs

# mount.glusterfs build-centos81:/gv0 /mnt

• 查看大小

# df -h | grep mnt
build-centos81:/gv0                  976M   13M  907M   2% /mnt

可以看到挂载后的空间大小等于两个节点中较小的一个

5. 扩容

以上创建的两个节点的glusterfs，因此每次扩容需要两个节点。创建两个磁盘，每个磁盘为2G。

• 扩容

# gluster volume add-brick gv0 build-centos81:/home/data3/brick1/gv0 build-centos81:/home/data4/brick1/gv0
# gluster volume info
 
Volume Name: gv0
Type: Distributed-Replicate
Volume ID: 7bf1b0e2-1dd2-44fe-a209-bc5ca430a350
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: build-centos81:/home/data1/brick1/gv0
Brick2: build-centos81:/home/data2/brick1/gv0
Brick3: build-centos81:/home/data3/brick1/gv0
Brick4: build-centos81:/home/data4/brick1/gv0
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
# df -h | grep mnt
build-centos81:/gv0                  2.9G   38M  2.7G   2% /mnt

6. 缩容

• 缩容

# gluster volume remove-brick gv0 build-centos81:/home/data3/brick1/gv0 build-centos81:/home/data4/brick1/gv0 start
Do you want to continue with your current cluster.force-migration settings? (y/n) y
volume remove-brick start: success
ID: 59a319e7-f055-4471-be3c-19c83b89665b

# gluster volume remove-brick gv0 build-centos81:/home/data3/brick1/gv0 build-centos81:/home/data4/brick1/gv0 status
Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
localhost                0        0Bytes             1             0             0            completed        0:00:00

# gluster volume remove-brick gv0 build-centos81:/home/data3/brick1/gv0 build-centos81:/home/data4/brick1/gv0 commit
volume remove-brick commit: success

7. 删除glusterfs

• umount挂载点

# umount /mnt

• 停止glusterfs

# gluster volume stop gv0

• 删除glusterfs

# gluster volume delete gv0
# gluster volume info
No volumes present