BeeGFS

A particular need has risen that a research group needs large amounts of storage space, yet reliable and fast.

Traditionally, central file storage is relatively slow, especially when implemented with NFS, as its designed for availability and reliability and not speed.

As an option, we've installed additional 14TB drives in a few machines, 7TB drives in a few others, and made those available as an aggregate storage volume that is faster than NFS. The disadvantage is that this architecture requires several services to be up and running else the entire volume/filesystem will become unavailable or unstable.

After considerable testing and experimentation, BeeGFS tends to perform better than other alternatives, though with the aforementioned downside of required services. For specifics on the testing done, see Throughput.

Implementation

We're going off the BeeGFS Quickstart.

We're installing on Ubuntu 22.04 LTS.
First, we'll add the repositories, then we'll add the packages and configure.

Repository Keys

Since things have changed slightly since the test (below), we need to add the GPG repository key in a different way than is noted in BeeGFS Quickstart.
wget -qOhttps://website/release/GPG-KEY | apt-key add - will no longer work as it has been DEPRECATED. Now, use the following format:

wget -qO- https://www.beegfs.io/release/beegfs_7.3.4/gpg/GPG-KEY-beegfs | \
gpg --dearmor > /etc/apt/trusted.gpg.d/beegfs.gpg

which will strip the key of its encryption, creating it as a binary. Next, we'll add the repo:

# /etc/apt/sources.list.d/beegfs-jammy.list
deb https://www.beegfs.io/release/beegfs_7.3.4  jammy non-free

This should make all beegfs packages available to us.

BeeGFS-MGMTD

We need to have a mgmtd server, to orchestrate the goings on. In Imaging, we'll make this IMAGING-01. This means that the MGMTD service needs to be constantly running and available on IMAGING-01, else the BeeGFS filesystem will appear unavailable. This is done ala:

> /opt/beegfs/sbin/beegfs-setup-mgmtd -p /opt/beegfs/mgmtd

This will create files in /opt/beegfs/mgmtd to keep track. We will need to edit the /etc/beegfs/beegfs-mgmtd.conf file to include the connauthfile directive. We'll create the connauthfile:

> dd if=/dev/random of=/etc/beegfs/connauthfile bs=128 count=1
> chown root.root /etc/beegfs/connauthfile
> chmod 400 /etc/beegfs/connauthfile

And that file we will need to be copied to every participating node, whether storage, client, or other.

BeeGFS-META

We need to have a -meta server to keep track of the files in the filesystem. This will be the service that locates data blocks throughout the volume, and tracks which system the data is on. This service also needs to be constantly available, else clients will not be able to access data.

We'll mirror this later, but for now we only need one, and install it onto IMAGING-01 with:

> /opt/beegfs/sbin/beegfs-setup-meta -p /opt/beegfs/meta/ -s 1 -m imaging-01

Then edit the /etc/beegfs/beegfs-meta.conf file to include the connauthfile directive.

BeeGFS-STORAGE

Of course we have to declare the storage, and in Imaging its no different. The following is a breakout:

Host	Device	Size
`IMAGING-01`	`/dev/sdb1`	12T
`IMAGING-02`	`/dev/sdb1`	12T
`IMAGING-03`	`/dev/nvme1n1p1`	7T
`IMAGING-04`	`/dev/nvme1n1p1`	7T

This should create a 37T volume of space, located under… we'll get to that. First, configuration with:

>  /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/14t_drive -s1 -i 101 -m imaging-01

The /mnt/14t_drive is different on -03 and -04, as those are mounted as /mnt/7t_drive. The -s1 and -i 101 here indicate that we're on IMAGING-01, the first server, and naming the storage target the first one. On IMAGING-02, it would look like -s2 -i 201, with the same -m imaging-01 since that is still our -mgmtd node. Also, edit the /etc/beegfs/beegfs-storage.conf to include the connauthfile directive.

BeeGFS-CLIENT

Of course each system is going to be its own client.

> /opt/beegfs/sbin/beegfs-setup-client -m imaging-01

Also, edit the /etc/beegfs/beegfs-client.conf to include the connauthfile directive. Additionally, we'll need the beegfs-helperd package installed, and configured via the /etc/beegfs/beegfs-helperd.conf file to include the connauthfile directive.

BeeGFS-MOUNT

Finally, each client that will be mounting the filesystem will have a list of mounted and client config files in /etc/beegfs/beegfs-mount.conf that looks similar to:

/srv/tmp /etc/beegfs/beegfs-client.conf

As you guessed it, the system will mount the filesystem onto /srv/tmp.

Daemon Launch/Enable

On IMAGING-01 (the -mgmtd and -meta node),
Launch:
systemctl start beegfs-mgmtd
systemctl start beegfs-meta
and check for errors (they may fail silently).

On all participating nodes (IMAGING-01 thru -04)
Launch:
systemctl start beegfs-storage
systemctl start beegfs-helperd
systemctl start beegfs-client
and check for errors.

Checking Validity

If everything went as expected, you should be able to verify with some beegfs-utils commands ala:

> beegfs-ctl --listnodes --nodetype=management --details
> beegfs-ctl --listnodes --nodetype=meta --details
> beegfs-ctl --listnodes --nodetype=storage --details
> beegfs-ctl --listnodes --nodetype=client --details
> beegfs-df
> beegfs-net
> beegfs-check-servers

Throughput testing

We'll go through our throughput tests that we've done before.

SCRATCH NFS WRITE

SCRATCH-NFS-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
SCRATCH-NFS-WRITE: Laying out IO file (1 file / 1024MiB)

SCRATCH-NFS-WRITE: (groupid=0, jobs=1): err= 0: pid=22026: Wed Jul 12 05:22:48 2023
  write: IOPS=27, BW=54.9MiB/s (57.5MB/s)(4090MiB/74528msec); 0 zone resets
    slat (usec): min=1115, max=10854, avg=1488.93, stdev=295.11
    clat (usec): min=3, max=51284, avg=44515.98, stdev=6766.35
     lat (usec): min=1470, max=62092, avg=46005.17, stdev=6788.76
    clat percentiles (usec):
     |  1.00th=[ 6915],  5.00th=[37487], 10.00th=[42730], 20.00th=[43254],
     | 30.00th=[45876], 40.00th=[46400], 50.00th=[46400], 60.00th=[46924],
     | 70.00th=[46924], 80.00th=[46924], 90.00th=[47449], 95.00th=[47449],
     | 99.00th=[49021], 99.50th=[50070], 99.90th=[51119], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=254076, max=1352340, per=100.00%, avg=930701.33, stdev=392448.73, samples=9
   iops        : min=  124, max=  660, avg=454.22, stdev=191.53, samples=9
  lat (usec)   : 4=0.20%
  lat (msec)   : 2=0.20%, 4=0.20%, 10=0.83%, 20=1.37%, 50=96.77%
  lat (msec)   : 100=0.44%
  cpu          : usr=0.41%, sys=4.87%, ctx=264481, majf=0, minf=12
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.1%, 32=93.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2044,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=54.9MiB/s (57.5MB/s), 54.9MiB/s-54.9MiB/s (57.5MB/s-57.5MB/s), io=4090MiB (4289MB), run=74528-74528msec

SCRATCH NFS READ

SCRATCH-NFS-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process

SCRATCH-NFS-READ: (groupid=0, jobs=1): err= 0: pid=22145: Wed Jul 12 05:25:15 2023
  read: IOPS=26, BW=52.1MiB/s (54.6MB/s)(3170MiB/60845msec)
    slat (msec): min=13, max=1759, avg=38.18, stdev=180.47
    clat (usec): min=5, max=2299.4k, avg=1134958.95, stdev=792260.44
     lat (msec): min=15, max=2312, avg=1173.14, stdev=796.95
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[  460], 10.00th=[  542], 20.00th=[  550],
     | 30.00th=[  558], 40.00th=[  558], 50.00th=[  558], 60.00th=[  558],
     | 70.00th=[ 2056], 80.00th=[ 2072], 90.00th=[ 2265], 95.00th=[ 2299],
     | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299],
     | 99.99th=[ 2299]
   bw (  KiB/s): min= 4098, max=188508, per=100.00%, avg=85947.24, stdev=41220.53, samples=74
   iops        : min=    2, max=   92, avg=41.95, stdev=20.12, samples=74
  lat (usec)   : 10=0.25%
  lat (msec)   : 20=0.19%, 50=0.19%, 100=0.57%, 250=1.70%, 500=2.65%
  lat (msec)   : 750=56.82%, 2000=0.95%, >=2000=36.68%
  cpu          : usr=0.02%, sys=6.16%, ctx=5123, majf=0, minf=16408
  IO depths    : 1=0.3%, 2=0.5%, 4=1.0%, 8=2.0%, 16=4.0%, 32=92.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=1584,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=52.1MiB/s (54.6MB/s), 52.1MiB/s-52.1MiB/s (54.6MB/s-54.6MB/s), io=3170MiB (3324MB), run=60845-60845msec

LOCAL WRITE

LOCAL-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
LOCAL-WRITE: Laying out IO file (1 file / 1024MiB)

LOCAL-WRITE: (groupid=0, jobs=1): err= 0: pid=22283: Wed Jul 12 05:28:20 2023
  write: IOPS=173, BW=348MiB/s (365MB/s)(10.0GiB/29446msec); 0 zone resets
    slat (usec): min=1087, max=4394, avg=1238.50, stdev=131.52
    clat (usec): min=3, max=42947, avg=37143.29, stdev=5337.20
     lat (usec): min=1156, max=44101, avg=38382.06, stdev=5314.32
    clat percentiles (usec):
     |  1.00th=[ 6980],  5.00th=[32375], 10.00th=[35914], 20.00th=[37487],
     | 30.00th=[38011], 40.00th=[38011], 50.00th=[38011], 60.00th=[38536],
     | 70.00th=[38536], 80.00th=[39060], 90.00th=[39584], 95.00th=[39584],
     | 99.00th=[40633], 99.50th=[41157], 99.90th=[41681], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=426192, max=1667886, per=100.00%, avg=1046875.40, stdev=424328.54, samples=20
   iops        : min=  208, max=  814, avg=510.90, stdev=207.06, samples=20
  lat (usec)   : 4=0.21%
  lat (msec)   : 2=0.10%, 4=0.20%, 10=1.02%, 20=1.58%, 50=96.89%
  cpu          : usr=1.83%, sys=31.39%, ctx=7684, majf=0, minf=17
  IO depths    : 1=0.2%, 2=0.4%, 4=0.9%, 8=1.6%, 16=3.1%, 32=93.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.1%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5118,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=348MiB/s (365MB/s), 348MiB/s-348MiB/s (365MB/s-365MB/s), io=10.0GiB (10.7GB), run=29446-29446msec

Disk stats (read/write):
  sda: ios=3456/8439, merge=0/45, ticks=445/135160, in_queue=135700, util=92.35%

LOCAL READ

LOCAL-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process

LOCAL-READ: (groupid=0, jobs=1): err= 0: pid=22386: Wed Jul 12 05:29:52 2023
  read: IOPS=170, BW=341MiB/s (357MB/s)(10.0GiB/30065msec)
    slat (usec): min=3203, max=10216, avg=5714.25, stdev=336.53
    clat (usec): min=2, max=182102, avg=171437.90, stdev=26459.08
     lat (msec): min=3, max=190, avg=177.15, stdev=26.48
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[  138], 10.00th=[  167], 20.00th=[  178],
     | 30.00th=[  178], 40.00th=[  178], 50.00th=[  178], 60.00th=[  180],
     | 70.00th=[  180], 80.00th=[  180], 90.00th=[  180], 95.00th=[  180],
     | 99.00th=[  182], 99.50th=[  182], 99.90th=[  182], 99.95th=[  182],
     | 99.99th=[  182]
   bw (  KiB/s): min=208998, max=458976, per=100.00%, avg=349013.00, stdev=63311.60, samples=60
   iops        : min=  102, max=  224, avg=170.33, stdev=30.90, samples=60
  lat (usec)   : 4=0.02%, 10=0.20%
  lat (msec)   : 4=0.20%, 10=0.21%, 20=0.23%, 50=1.21%, 100=1.60%
  lat (msec)   : 250=96.33%
  cpu          : usr=0.14%, sys=30.15%, ctx=5561, majf=0, minf=16411
  IO depths    : 1=0.2%, 2=0.4%, 4=0.9%, 8=1.6%, 16=3.1%, 32=93.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.1%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=5118,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=341MiB/s (357MB/s), 341MiB/s-341MiB/s (357MB/s-357MB/s), io=10.0GiB (10.7GB), run=30065-30065msec

Disk stats (read/write):
  sda: ios=10530/9, merge=50/1, ticks=46614/7, in_queue=46621, util=97.41%

BeeGFS WRITE

BeeGFS-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
BeeGFS-WRITE: Laying out IO file (1 file / 1024MiB)

BeeGFS-WRITE: (groupid=0, jobs=1): err= 0: pid=22709: Wed Jul 12 05:35:38 2023
  write: IOPS=34, BW=68.9MiB/s (72.2MB/s)(4132MiB/60005msec); 0 zone resets
    slat (msec): min=13, max=1582, avg=29.04, stdev=149.64
    clat (usec): min=7, max=2013.4k, avg=851876.99, stdev=690184.39
     lat (msec): min=14, max=2027, avg=880.92, stdev=696.79
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[  426], 10.00th=[  439], 20.00th=[  439],
     | 30.00th=[  443], 40.00th=[  443], 50.00th=[  443], 60.00th=[  443],
     | 70.00th=[  447], 80.00th=[ 1955], 90.00th=[ 1972], 95.00th=[ 1989],
     | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2005], 99.95th=[ 2022],
     | 99.99th=[ 2022]
   bw (  KiB/s): min= 4098, max=249978, per=100.00%, avg=108783.27, stdev=52595.86, samples=77
   iops        : min=    2, max=  122, avg=53.09, stdev=25.67, samples=77
  lat (usec)   : 10=0.19%, 20=0.05%
  lat (msec)   : 20=0.24%, 50=0.44%, 100=0.73%, 250=1.60%, 500=68.81%
  lat (msec)   : 2000=26.44%, >=2000=1.50%
  cpu          : usr=0.86%, sys=9.71%, ctx=445229, majf=0, minf=16
  IO depths    : 1=0.2%, 2=0.5%, 4=1.0%, 8=1.9%, 16=3.4%, 32=93.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2065,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=68.9MiB/s (72.2MB/s), 68.9MiB/s-68.9MiB/s (72.2MB/s-72.2MB/s), io=4132MiB (4333MB), run=60005-60005msec

BeeGFS READ

BeeGFS-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
BeeGFS-READ: Laying out IO file (1 file / 1024MiB)

BeeGFS-READ: (groupid=0, jobs=1): err= 0: pid=22827: Wed Jul 12 05:37:41 2023
  read: IOPS=67, BW=135MiB/s (142MB/s)(8106MiB/60001msec)
    slat (msec): min=14, max=215, avg=14.80, stdev= 8.33
    clat (usec): min=8, max=669906, avg=444903.42, stdev=80659.05
     lat (msec): min=14, max=684, avg=459.70, stdev=81.14
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[  359], 10.00th=[  443], 20.00th=[  447],
     | 30.00th=[  447], 40.00th=[  447], 50.00th=[  447], 60.00th=[  447],
     | 70.00th=[  447], 80.00th=[  447], 90.00th=[  464], 95.00th=[  642],
     | 99.00th=[  651], 99.50th=[  651], 99.90th=[  659], 99.95th=[  667],
     | 99.99th=[  667]
   bw (  KiB/s): min=12294, max=266370, per=99.22%, avg=137265.78, stdev=28838.77, samples=119
   iops        : min=    6, max=  130, avg=66.99, stdev=14.07, samples=119
  lat (usec)   : 10=0.05%, 20=0.15%
  lat (msec)   : 20=0.20%, 50=0.39%, 100=0.59%, 250=2.17%, 500=91.09%
  lat (msec)   : 750=5.36%
  cpu          : usr=0.06%, sys=20.48%, ctx=448073, majf=0, minf=16412
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.2%, 32=93.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=4051,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=8106MiB (8500MB), run=60001-60001msec

Tests all ran on IMAGING-02.
Scratch NFS = ~57MB/s read/write
BeeGFS = ~80MB/s write, ~140MB/s read

This is about a 50% improvement for write operations, and a 150% improvement for read operations.