# BeeGFS

A particular need has risen that a research group needs large amounts of storage space, yet reliable and fast.

Traditionally, central file storage is relatively slow, especially when implemented with NFS, as its designed for availability and reliability and not speed.

As an option, we've installed additional 14TB drives in a few machines, 7TB drives in a few others, and made those available as an aggregate storage volume that is faster than NFS.
The disadvantage is that this architecture requires several services to be up and running else the entire volume/filesystem will become unavailable or unstable.

After considerable testing and experimentation, BeeGFS tends to perform better than other alternatives, though with the aforementioned downside of required services.
For specifics on the testing done, see [Throughput](/Main/BeeGFS#throughput-testing).

## Implementation
We're going off the [BeeGFS Quickstart](https://doc.beegfs.io/latest/quick_start_guide/quick_start_guide.html#example-setup).

We're installing on Ubuntu 22.04 LTS.  
First, we'll add the repositories, then we'll add the packages and configure.

### Repository Keys
Since things have changed slightly since the test (below), we need to add the GPG repository key in a different way than is noted in BeeGFS Quickstart.  
`wget -qOhttps://website/release/GPG-KEY | apt-key add -` will no longer work as it has been DEPRECATED.
Now, use the following format:

```
wget -qO- https://www.beegfs.io/release/beegfs_7.3.4/gpg/GPG-KEY-beegfs | \
gpg --dearmor > /etc/apt/trusted.gpg.d/beegfs.gpg
```

which will strip the key of its encryption, creating it as a binary.
Next, we'll add the repo:

```
# /etc/apt/sources.list.d/beegfs-jammy.list
deb https://www.beegfs.io/release/beegfs_7.3.4  jammy non-free
```

This should make all beegfs packages available to us.

### BeeGFS-MGMTD
We need to have a `mgmtd` server, to orchestrate the goings on.
In Imaging, we'll make this `IMAGING-01`.
This means that the MGMTD service needs to be constantly running and available on IMAGING-01, else the BeeGFS filesystem will appear unavailable.
This is done ala:

```
> /opt/beegfs/sbin/beegfs-setup-mgmtd -p /opt/beegfs/mgmtd

```

This will create files in `/opt/beegfs/mgmtd` to keep track.
We will need to edit the `/etc/beegfs/beegfs-mgmtd.conf` file to include the `connauthfile` directive.
We'll create the `connauthfile`:

```
> dd if=/dev/random of=/etc/beegfs/connauthfile bs=128 count=1
> chown root.root /etc/beegfs/connauthfile
> chmod 400 /etc/beegfs/connauthfile
```

And that file we will need to be copied to every participating node, whether storage, client, or other.

### BeeGFS-META
We need to have a `-meta` server to keep track of the files in the filesystem.
This will be the service that locates data blocks throughout the volume, and tracks which system the data is on.
This service also needs to be constantly available, else clients will not be able to access data.

We'll mirror this later, but for now we only need one, and install it onto `IMAGING-01` with:

```
> /opt/beegfs/sbin/beegfs-setup-meta -p /opt/beegfs/meta/ -s 1 -m imaging-01
```
Then edit the `/etc/beegfs/beegfs-meta.conf` file to include the `connauthfile` directive.

### BeeGFS-STORAGE
Of course we have to declare the storage, and in Imaging its no different.
The following is a breakout:

| Host         | Device           | Size |
| ------------ | ---------------- | ---- |
| `IMAGING-01` | `/dev/sdb1`      | 12T  |
| `IMAGING-02` | `/dev/sdb1`      | 12T  |
| `IMAGING-03` | `/dev/nvme1n1p1` | 7T   |
| `IMAGING-04` | `/dev/nvme1n1p1` | 7T     |
This should create a 37T volume of space, located under… we'll get to that.
First, configuration with:

```
>  /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/14t_drive -s1 -i 101 -m imaging-01
```

The `/mnt/14t_drive` is different on `-03` and `-04`, as those are mounted as `/mnt/7t_drive`.
The `-s1` and `-i 101` here indicate that we're on `IMAGING-01`, the first server, and naming the storage target the first one.
On `IMAGING-02`, it would look like `-s2 -i 201`, with the same `-m imaging-01` since that is still our `-mgmtd` node.
Also, edit the `/etc/beegfs/beegfs-storage.conf` to include the `connauthfile` directive.

### BeeGFS-CLIENT
Of course each system is going to be its own client.

```
> /opt/beegfs/sbin/beegfs-setup-client -m imaging-01
```

Also, edit the `/etc/beegfs/beegfs-client.conf` to include the `connauthfile` directive.
Additionally, we'll need the `beegfs-helperd` package installed, and configured via the `/etc/beegfs/beegfs-helperd.conf` file to include the `connauthfile` directive.

### BeeGFS-MOUNT
Finally, each client that will be mounting the filesystem will have a list of mounted and client config files in `/etc/beegfs/beegfs-mount.conf` that looks similar to:

```
/srv/tmp /etc/beegfs/beegfs-client.conf
```

As you guessed it, the system will mount the filesystem onto `/srv/tmp`.

## Daemon Launch/Enable
On `IMAGING-01` (the `-mgmtd` and `-meta` node),  
Launch:  
`systemctl start beegfs-mgmtd`  
`systemctl start beegfs-meta`  
and check for errors (they may fail silently).  

On all participating nodes (`IMAGING-01 thru -04`)  
Launch:  
`systemctl start beegfs-storage`  
`systemctl start beegfs-helperd`  
`systemctl start beegfs-client`  
and check for errors.  

## Checking Validity
If everything went as expected, you should be able to verify with some `beegfs-utils` commands ala:

```bash
> beegfs-ctl --listnodes --nodetype=management --details
> beegfs-ctl --listnodes --nodetype=meta --details
> beegfs-ctl --listnodes --nodetype=storage --details
> beegfs-ctl --listnodes --nodetype=client --details
> beegfs-df
> beegfs-net
> beegfs-check-servers
```

### Throughput testing
We'll go through our throughput tests that we've done before.

#### SCRATCH NFS WRITE

```
SCRATCH-NFS-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
SCRATCH-NFS-WRITE: Laying out IO file (1 file / 1024MiB)

SCRATCH-NFS-WRITE: (groupid=0, jobs=1): err= 0: pid=22026: Wed Jul 12 05:22:48 2023
  write: IOPS=27, BW=54.9MiB/s (57.5MB/s)(4090MiB/74528msec); 0 zone resets
    slat (usec): min=1115, max=10854, avg=1488.93, stdev=295.11
    clat (usec): min=3, max=51284, avg=44515.98, stdev=6766.35
     lat (usec): min=1470, max=62092, avg=46005.17, stdev=6788.76
    clat percentiles (usec):
     |  1.00th=[ 6915],  5.00th=[37487], 10.00th=[42730], 20.00th=[43254],
     | 30.00th=[45876], 40.00th=[46400], 50.00th=[46400], 60.00th=[46924],
     | 70.00th=[46924], 80.00th=[46924], 90.00th=[47449], 95.00th=[47449],
     | 99.00th=[49021], 99.50th=[50070], 99.90th=[51119], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=254076, max=1352340, per=100.00%, avg=930701.33, stdev=392448.73, samples=9
   iops        : min=  124, max=  660, avg=454.22, stdev=191.53, samples=9
  lat (usec)   : 4=0.20%
  lat (msec)   : 2=0.20%, 4=0.20%, 10=0.83%, 20=1.37%, 50=96.77%
  lat (msec)   : 100=0.44%
  cpu          : usr=0.41%, sys=4.87%, ctx=264481, majf=0, minf=12
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.1%, 32=93.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2044,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=54.9MiB/s (57.5MB/s), 54.9MiB/s-54.9MiB/s (57.5MB/s-57.5MB/s), io=4090MiB (4289MB), run=74528-74528msec
```

#### SCRATCH NFS READ

```
SCRATCH-NFS-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process

SCRATCH-NFS-READ: (groupid=0, jobs=1): err= 0: pid=22145: Wed Jul 12 05:25:15 2023
  read: IOPS=26, BW=52.1MiB/s (54.6MB/s)(3170MiB/60845msec)
    slat (msec): min=13, max=1759, avg=38.18, stdev=180.47
    clat (usec): min=5, max=2299.4k, avg=1134958.95, stdev=792260.44
     lat (msec): min=15, max=2312, avg=1173.14, stdev=796.95
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[  460], 10.00th=[  542], 20.00th=[  550],
     | 30.00th=[  558], 40.00th=[  558], 50.00th=[  558], 60.00th=[  558],
     | 70.00th=[ 2056], 80.00th=[ 2072], 90.00th=[ 2265], 95.00th=[ 2299],
     | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299],
     | 99.99th=[ 2299]
   bw (  KiB/s): min= 4098, max=188508, per=100.00%, avg=85947.24, stdev=41220.53, samples=74
   iops        : min=    2, max=   92, avg=41.95, stdev=20.12, samples=74
  lat (usec)   : 10=0.25%
  lat (msec)   : 20=0.19%, 50=0.19%, 100=0.57%, 250=1.70%, 500=2.65%
  lat (msec)   : 750=56.82%, 2000=0.95%, >=2000=36.68%
  cpu          : usr=0.02%, sys=6.16%, ctx=5123, majf=0, minf=16408
  IO depths    : 1=0.3%, 2=0.5%, 4=1.0%, 8=2.0%, 16=4.0%, 32=92.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=1584,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=52.1MiB/s (54.6MB/s), 52.1MiB/s-52.1MiB/s (54.6MB/s-54.6MB/s), io=3170MiB (3324MB), run=60845-60845msec
```

#### LOCAL WRITE

```
LOCAL-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
LOCAL-WRITE: Laying out IO file (1 file / 1024MiB)

LOCAL-WRITE: (groupid=0, jobs=1): err= 0: pid=22283: Wed Jul 12 05:28:20 2023
  write: IOPS=173, BW=348MiB/s (365MB/s)(10.0GiB/29446msec); 0 zone resets
    slat (usec): min=1087, max=4394, avg=1238.50, stdev=131.52
    clat (usec): min=3, max=42947, avg=37143.29, stdev=5337.20
     lat (usec): min=1156, max=44101, avg=38382.06, stdev=5314.32
    clat percentiles (usec):
     |  1.00th=[ 6980],  5.00th=[32375], 10.00th=[35914], 20.00th=[37487],
     | 30.00th=[38011], 40.00th=[38011], 50.00th=[38011], 60.00th=[38536],
     | 70.00th=[38536], 80.00th=[39060], 90.00th=[39584], 95.00th=[39584],
     | 99.00th=[40633], 99.50th=[41157], 99.90th=[41681], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=426192, max=1667886, per=100.00%, avg=1046875.40, stdev=424328.54, samples=20
   iops        : min=  208, max=  814, avg=510.90, stdev=207.06, samples=20
  lat (usec)   : 4=0.21%
  lat (msec)   : 2=0.10%, 4=0.20%, 10=1.02%, 20=1.58%, 50=96.89%
  cpu          : usr=1.83%, sys=31.39%, ctx=7684, majf=0, minf=17
  IO depths    : 1=0.2%, 2=0.4%, 4=0.9%, 8=1.6%, 16=3.1%, 32=93.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.1%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5118,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=348MiB/s (365MB/s), 348MiB/s-348MiB/s (365MB/s-365MB/s), io=10.0GiB (10.7GB), run=29446-29446msec

Disk stats (read/write):
  sda: ios=3456/8439, merge=0/45, ticks=445/135160, in_queue=135700, util=92.35%
```

#### LOCAL READ

```
LOCAL-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process

LOCAL-READ: (groupid=0, jobs=1): err= 0: pid=22386: Wed Jul 12 05:29:52 2023
  read: IOPS=170, BW=341MiB/s (357MB/s)(10.0GiB/30065msec)
    slat (usec): min=3203, max=10216, avg=5714.25, stdev=336.53
    clat (usec): min=2, max=182102, avg=171437.90, stdev=26459.08
     lat (msec): min=3, max=190, avg=177.15, stdev=26.48
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[  138], 10.00th=[  167], 20.00th=[  178],
     | 30.00th=[  178], 40.00th=[  178], 50.00th=[  178], 60.00th=[  180],
     | 70.00th=[  180], 80.00th=[  180], 90.00th=[  180], 95.00th=[  180],
     | 99.00th=[  182], 99.50th=[  182], 99.90th=[  182], 99.95th=[  182],
     | 99.99th=[  182]
   bw (  KiB/s): min=208998, max=458976, per=100.00%, avg=349013.00, stdev=63311.60, samples=60
   iops        : min=  102, max=  224, avg=170.33, stdev=30.90, samples=60
  lat (usec)   : 4=0.02%, 10=0.20%
  lat (msec)   : 4=0.20%, 10=0.21%, 20=0.23%, 50=1.21%, 100=1.60%
  lat (msec)   : 250=96.33%
  cpu          : usr=0.14%, sys=30.15%, ctx=5561, majf=0, minf=16411
  IO depths    : 1=0.2%, 2=0.4%, 4=0.9%, 8=1.6%, 16=3.1%, 32=93.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.1%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=5118,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=341MiB/s (357MB/s), 341MiB/s-341MiB/s (357MB/s-357MB/s), io=10.0GiB (10.7GB), run=30065-30065msec

Disk stats (read/write):
  sda: ios=10530/9, merge=50/1, ticks=46614/7, in_queue=46621, util=97.41%
```

#### BeeGFS WRITE

```
BeeGFS-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
BeeGFS-WRITE: Laying out IO file (1 file / 1024MiB)

BeeGFS-WRITE: (groupid=0, jobs=1): err= 0: pid=22709: Wed Jul 12 05:35:38 2023
  write: IOPS=34, BW=68.9MiB/s (72.2MB/s)(4132MiB/60005msec); 0 zone resets
    slat (msec): min=13, max=1582, avg=29.04, stdev=149.64
    clat (usec): min=7, max=2013.4k, avg=851876.99, stdev=690184.39
     lat (msec): min=14, max=2027, avg=880.92, stdev=696.79
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[  426], 10.00th=[  439], 20.00th=[  439],
     | 30.00th=[  443], 40.00th=[  443], 50.00th=[  443], 60.00th=[  443],
     | 70.00th=[  447], 80.00th=[ 1955], 90.00th=[ 1972], 95.00th=[ 1989],
     | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2005], 99.95th=[ 2022],
     | 99.99th=[ 2022]
   bw (  KiB/s): min= 4098, max=249978, per=100.00%, avg=108783.27, stdev=52595.86, samples=77
   iops        : min=    2, max=  122, avg=53.09, stdev=25.67, samples=77
  lat (usec)   : 10=0.19%, 20=0.05%
  lat (msec)   : 20=0.24%, 50=0.44%, 100=0.73%, 250=1.60%, 500=68.81%
  lat (msec)   : 2000=26.44%, >=2000=1.50%
  cpu          : usr=0.86%, sys=9.71%, ctx=445229, majf=0, minf=16
  IO depths    : 1=0.2%, 2=0.5%, 4=1.0%, 8=1.9%, 16=3.4%, 32=93.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2065,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=68.9MiB/s (72.2MB/s), 68.9MiB/s-68.9MiB/s (72.2MB/s-72.2MB/s), io=4132MiB (4333MB), run=60005-60005msec
```

#### BeeGFS READ

```
BeeGFS-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
fio-3.28
Starting 1 process
BeeGFS-READ: Laying out IO file (1 file / 1024MiB)

BeeGFS-READ: (groupid=0, jobs=1): err= 0: pid=22827: Wed Jul 12 05:37:41 2023
  read: IOPS=67, BW=135MiB/s (142MB/s)(8106MiB/60001msec)
    slat (msec): min=14, max=215, avg=14.80, stdev= 8.33
    clat (usec): min=8, max=669906, avg=444903.42, stdev=80659.05
     lat (msec): min=14, max=684, avg=459.70, stdev=81.14
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[  359], 10.00th=[  443], 20.00th=[  447],
     | 30.00th=[  447], 40.00th=[  447], 50.00th=[  447], 60.00th=[  447],
     | 70.00th=[  447], 80.00th=[  447], 90.00th=[  464], 95.00th=[  642],
     | 99.00th=[  651], 99.50th=[  651], 99.90th=[  659], 99.95th=[  667],
     | 99.99th=[  667]
   bw (  KiB/s): min=12294, max=266370, per=99.22%, avg=137265.78, stdev=28838.77, samples=119
   iops        : min=    6, max=  130, avg=66.99, stdev=14.07, samples=119
  lat (usec)   : 10=0.05%, 20=0.15%
  lat (msec)   : 20=0.20%, 50=0.39%, 100=0.59%, 250=2.17%, 500=91.09%
  lat (msec)   : 750=5.36%
  cpu          : usr=0.06%, sys=20.48%, ctx=448073, majf=0, minf=16412
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.2%, 32=93.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=4051,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=8106MiB (8500MB), run=60001-60001msec
```

Tests all ran on `IMAGING-02`.   
Scratch NFS = ~57MB/s read/write   
BeeGFS = ~80MB/s write, ~140MB/s read  

This is about a 50% improvement for write operations,
and a 150% improvement for read operations.