Home A - Z
Page Index

Commit 6a3b54

2026-11-13 20:18:19 Dave LeBlanc: BeeGFS updated on Fri Nov 13 20:18:04 2026
main/beegfs.md ..
@@ 1,1 1,362 @@
# BeeGFS
+
+A particular need has risen that a research group needs large amounts of storage space, yet reliable and fast.
+
+Traditionally, central file storage is relatively slow, especially when implemented with NFS, as its designed for availability and reliability and not speed.
+
+As an option, we've installed additional 14TB drives in a few machines, 7TB drives in a few others, and made those available as an aggregate storage volume that is faster than NFS.
+The disadvantage is that this architecture requires several services to be up and running else the entire volume/filesystem will become unavailable or unstable.
+
+After considerable testing and experimentation, BeeGFS tends to perform better than other alternatives, though with the aforementioned downside of required services.
+For specifics on the testing done, see [Throughput](/Main/BeeGFS#scratch-nfs-write).
+
+## Implementation
+We're going off the [BeeGFS Quickstart](https://doc.beegfs.io/latest/quick_start_guide/quick_start_guide.html#example-setup).
+
+We're installing on Ubuntu 22.04 LTS.
+First, we'll add the repositories, then we'll add the packages and configure.
+
+### Repository Keys
+Since things have changed slightly since the test (below), we need to add the GPG repository key in a different way than is noted in BeeGFS Quickstart.
+`wget -qOhttps://website/release/GPG-KEY | apt-key add -` will no longer work as it has been DEPRECATED.
+Now, use the following format:
+
+```
+wget -qO- https://www.beegfs.io/release/beegfs_7.3.4/gpg/GPG-KEY-beegfs | \
+gpg --dearmor > /etc/apt/trusted.gpg.d/beegfs.gpg
+```
+
+which will strip the key of its encryption, creating it as a binary.
+Next, we'll add the repo:
+
+```
+# /etc/apt/sources.list.d/beegfs-jammy.list
+deb https://www.beegfs.io/release/beegfs_7.3.4 jammy non-free
+```
+
+This should make all beegfs packages available to us.
+
+### BeeGFS-MGMTD
+We need to have a `mgmtd` server, to orchestrate the goings on.
+In Imaging, we'll make this `IMAGING-01`.
+This means that the MGMTD service needs to be constantly running and available on IMAGING-01, else the BeeGFS filesystem will appear unavailable.
+This is done ala:
+
+```
+> /opt/beegfs/sbin/beegfs-setup-mgmtd -p /opt/beegfs/mgmtd
+
+```
+
+This will create files in `/opt/beegfs/mgmtd` to keep track.
+We will need to edit the `/etc/beegfs/beegfs-mgmtd.conf` file to include the `connauthfile` directive.
+We'll create the `connauthfile`:
+
+```
+> dd if=/dev/random of=/etc/beegfs/connauthfile bs=128 count=1
+> chown root.root /etc/beegfs/connauthfile
+> chmod 400 /etc/beegfs/connauthfile
+```
+
+And that file we will need to be copied to every participating node, whether storage, client, or other.
+
+### BeeGFS-META
+We need to have a `-meta` server to keep track of the files in the filesystem.
+This will be the service that locations data blocks throughout the volume, and track which system the data is on.
+This service also needs to be constantly available, else clients will not be able to access data.
+
+We'll mirror this later, but for now we only need one, and install it onto `IMAGING-01` with:
+
+```
+> /opt/beegfs/sbin/beegfs-setup-meta -p /opt/beegfs/meta/ -s 1 -m imaging-01
+```
+Then edit the `/etc/beegfs/beegfs-meta.conf` file to include the `connauthfile` directive.
+
+### BeeGFS-STORAGE
+Of course we have to declare the storage, and in Imaging its no different.
+The following is a breakout:
+
+| Host | Device | Size |
+| ------------ | ---------------- | ---- |
+| `IMAGING-01` | `/dev/sdb1` | 12T |
+| `IMAGING-02` | `/dev/sdb1` | 12T |
+| `IMAGING-03` | `/dev/nvme1n1p1` | 7T |
+| `IMAGING-04` | `/dev/nvme1n1p1` | 7T |
+This should create a 37T volume of space, located under… we'll get to that.
+First, configuration with:
+
+```
+> /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/14t_drive -s1 -i 101 -m imaging-01
+```
+
+The `/mnt/14t_drive` is different on `-03` and `-04`, as those are mounted as `/mnt/7t_drive`.
+The `-s1` and `-i 101` here indicate that we're on `IMAGING-01`, the first server, and naming the storage target the first one.
+On `IMAGING-02`, it would look like `-s2 -i 201`, with the same `-m imaging-01` since that is still our `-mgmtd` node.
+Also, edit the `/etc/beegfs/beegfs-storage.conf` to include the `connauthfile` directive.
+
+### BeeGFS-CLIENT
+Of course each system is going to be its own client.
+
+```
+> /opt/beegfs/sbin/beegfs-setup-client -m imaging-01
+```
+
+Also, edit the `/etc/beegfs/beegfs-client.conf` to include the `connauthfile` directive.
+Additionally, we'll need the `beegfs-helperd` package installed, and configured via the `/etc/beegfs/beegfs-helperd.conf` file to include the `connauthfile` directive.
+
+### BeeGFS-MOUNT
+Finally, each client that will be mounting the filesystem will have a list of mounted and client config files in `/etc/beegfs/beegfs-mount.conf` that looks similar to:
+
+```
+/srv/tmp /etc/beegfs/beegfs-client.conf
+```
+
+As you guessed it, the system will mount the filesystem onto `/srv/tmp`.
+
+## Daemon Launch/Enable
+On `IMAGING-01` (the `-mgmtd` and `-meta` node),
+Launch:
+`systemctl start beegfs-mgmtd`
+`systemctl start beegfs-meta`
+and check for errors (they may fail silently).
+
+On all participating nodes (`IMAGING-01 thru -04`)
+Launch:
+`systemctl start beegfs-storage`
+`systemctl start beegfs-helperd`
+`systemctl start beegfs-client`
+and check for errors.
+
+## Checking Validity
+If everything went as expected, you should be able to verify with some `beegfs-utils` commands ala:
+
+```bash
+> beegfs-ctl --listnodes --nodetype=management --details
+> beegfs-ctl --listnodes --nodetype=meta --details
+> beegfs-ctl --listnodes --nodetype=storage --details
+> beegfs-ctl --listnodes --nodetype=client --details
+> beegfs-df
+> beegfs-net
+> beegfs-check-servers
+```
+
+We'll go through our throughput tests that we've done before.
+
+#### SCRATCH NFS WRITE
+
+```
+SCRATCH-NFS-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
+fio-3.28
+Starting 1 process
+SCRATCH-NFS-WRITE: Laying out IO file (1 file / 1024MiB)
+
+SCRATCH-NFS-WRITE: (groupid=0, jobs=1): err= 0: pid=22026: Wed Jul 12 05:22:48 2023
+ write: IOPS=27, BW=54.9MiB/s (57.5MB/s)(4090MiB/74528msec); 0 zone resets
+ slat (usec): min=1115, max=10854, avg=1488.93, stdev=295.11
+ clat (usec): min=3, max=51284, avg=44515.98, stdev=6766.35
+ lat (usec): min=1470, max=62092, avg=46005.17, stdev=6788.76
+ clat percentiles (usec):
+ | 1.00th=[ 6915], 5.00th=[37487], 10.00th=[42730], 20.00th=[43254],
+ | 30.00th=[45876], 40.00th=[46400], 50.00th=[46400], 60.00th=[46924],
+ | 70.00th=[46924], 80.00th=[46924], 90.00th=[47449], 95.00th=[47449],
+ | 99.00th=[49021], 99.50th=[50070], 99.90th=[51119], 99.95th=[51119],
+ | 99.99th=[51119]
+ bw ( KiB/s): min=254076, max=1352340, per=100.00%, avg=930701.33, stdev=392448.73, samples=9
+ iops : min= 124, max= 660, avg=454.22, stdev=191.53, samples=9
+ lat (usec) : 4=0.20%
+ lat (msec) : 2=0.20%, 4=0.20%, 10=0.83%, 20=1.37%, 50=96.77%
+ lat (msec) : 100=0.44%
+ cpu : usr=0.41%, sys=4.87%, ctx=264481, majf=0, minf=12
+ IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.1%, 32=93.9%, >=64=0.0%
+ submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
+ complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
+ issued rwts: total=0,2044,0,0 short=0,0,0,0 dropped=0,0,0,0
+ latency : target=0, window=0, percentile=100.00%, depth=32
+
+Run status group 0 (all jobs):
+ WRITE: bw=54.9MiB/s (57.5MB/s), 54.9MiB/s-54.9MiB/s (57.5MB/s-57.5MB/s), io=4090MiB (4289MB), run=74528-74528msec
+```
+
+#### SCRATCH NFS READ
+
+```
+SCRATCH-NFS-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
+fio-3.28
+Starting 1 process
+
+SCRATCH-NFS-READ: (groupid=0, jobs=1): err= 0: pid=22145: Wed Jul 12 05:25:15 2023
+ read: IOPS=26, BW=52.1MiB/s (54.6MB/s)(3170MiB/60845msec)
+ slat (msec): min=13, max=1759, avg=38.18, stdev=180.47
+ clat (usec): min=5, max=2299.4k, avg=1134958.95, stdev=792260.44
+ lat (msec): min=15, max=2312, avg=1173.14, stdev=796.95
+ clat percentiles (msec):
+ | 1.00th=[ 72], 5.00th=[ 460], 10.00th=[ 542], 20.00th=[ 550],
+ | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558],
+ | 70.00th=[ 2056], 80.00th=[ 2072], 90.00th=[ 2265], 95.00th=[ 2299],
+ | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299],
+ | 99.99th=[ 2299]
+ bw ( KiB/s): min= 4098, max=188508, per=100.00%, avg=85947.24, stdev=41220.53, samples=74
+ iops : min= 2, max= 92, avg=41.95, stdev=20.12, samples=74
+ lat (usec) : 10=0.25%
+ lat (msec) : 20=0.19%, 50=0.19%, 100=0.57%, 250=1.70%, 500=2.65%
+ lat (msec) : 750=56.82%, 2000=0.95%, >=2000=36.68%
+ cpu : usr=0.02%, sys=6.16%, ctx=5123, majf=0, minf=16408
+ IO depths : 1=0.3%, 2=0.5%, 4=1.0%, 8=2.0%, 16=4.0%, 32=92.2%, >=64=0.0%
+ submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
+ complete : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
+ issued rwts: total=1584,0,0,0 short=0,0,0,0 dropped=0,0,0,0
+ latency : target=0, window=0, percentile=100.00%, depth=32
+
+Run status group 0 (all jobs):
+ READ: bw=52.1MiB/s (54.6MB/s), 52.1MiB/s-52.1MiB/s (54.6MB/s-54.6MB/s), io=3170MiB (3324MB), run=60845-60845msec
+```
+
+#### LOCAL WRITE
+
+```
+LOCAL-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
+fio-3.28
+Starting 1 process
+LOCAL-WRITE: Laying out IO file (1 file / 1024MiB)
+
+LOCAL-WRITE: (groupid=0, jobs=1): err= 0: pid=22283: Wed Jul 12 05:28:20 2023
+ write: IOPS=173, BW=348MiB/s (365MB/s)(10.0GiB/29446msec); 0 zone resets
+ slat (usec): min=1087, max=4394, avg=1238.50, stdev=131.52
+ clat (usec): min=3, max=42947, avg=37143.29, stdev=5337.20
+ lat (usec): min=1156, max=44101, avg=38382.06, stdev=5314.32
+ clat percentiles (usec):
+ | 1.00th=[ 6980], 5.00th=[32375], 10.00th=[35914], 20.00th=[37487],
+ | 30.00th=[38011], 40.00th=[38011], 50.00th=[38011], 60.00th=[38536],
+ | 70.00th=[38536], 80.00th=[39060], 90.00th=[39584], 95.00th=[39584],
+ | 99.00th=[40633], 99.50th=[41157], 99.90th=[41681], 99.95th=[42730],
+ | 99.99th=[42730]
+ bw ( KiB/s): min=426192, max=1667886, per=100.00%, avg=1046875.40, stdev=424328.54, samples=20
+ iops : min= 208, max= 814, avg=510.90, stdev=207.06, samples=20
+ lat (usec) : 4=0.21%
+ lat (msec) : 2=0.10%, 4=0.20%, 10=1.02%, 20=1.58%, 50=96.89%
+ cpu : usr=1.83%, sys=31.39%, ctx=7684, majf=0, minf=17
+ IO depths : 1=0.2%, 2=0.4%, 4=0.9%, 8=1.6%, 16=3.1%, 32=93.8%, >=64=0.0%
+ submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
+ complete : 0=0.0%, 4=99.8%, 8=0.1%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
+ issued rwts: total=0,5118,0,0 short=0,0,0,0 dropped=0,0,0,0
+ latency : target=0, window=0, percentile=100.00%, depth=32
+
+Run status group 0 (all jobs):
+ WRITE: bw=348MiB/s (365MB/s), 348MiB/s-348MiB/s (365MB/s-365MB/s), io=10.0GiB (10.7GB), run=29446-29446msec
+
+Disk stats (read/write):
+ sda: ios=3456/8439, merge=0/45, ticks=445/135160, in_queue=135700, util=92.35%
+```
+
+#### LOCAL READ
+
+```
+LOCAL-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
+fio-3.28
+Starting 1 process
+
+LOCAL-READ: (groupid=0, jobs=1): err= 0: pid=22386: Wed Jul 12 05:29:52 2023
+ read: IOPS=170, BW=341MiB/s (357MB/s)(10.0GiB/30065msec)
+ slat (usec): min=3203, max=10216, avg=5714.25, stdev=336.53
+ clat (usec): min=2, max=182102, avg=171437.90, stdev=26459.08
+ lat (msec): min=3, max=190, avg=177.15, stdev=26.48
+ clat percentiles (msec):
+ | 1.00th=[ 21], 5.00th=[ 138], 10.00th=[ 167], 20.00th=[ 178],
+ | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 180],
+ | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180],
+ | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182],
+ | 99.99th=[ 182]
+ bw ( KiB/s): min=208998, max=458976, per=100.00%, avg=349013.00, stdev=63311.60, samples=60
+ iops : min= 102, max= 224, avg=170.33, stdev=30.90, samples=60
+ lat (usec) : 4=0.02%, 10=0.20%
+ lat (msec) : 4=0.20%, 10=0.21%, 20=0.23%, 50=1.21%, 100=1.60%
+ lat (msec) : 250=96.33%
+ cpu : usr=0.14%, sys=30.15%, ctx=5561, majf=0, minf=16411
+ IO depths : 1=0.2%, 2=0.4%, 4=0.9%, 8=1.6%, 16=3.1%, 32=93.8%, >=64=0.0%
+ submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
+ complete : 0=0.0%, 4=99.8%, 8=0.1%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
+ issued rwts: total=5118,0,0,0 short=0,0,0,0 dropped=0,0,0,0
+ latency : target=0, window=0, percentile=100.00%, depth=32
+
+Run status group 0 (all jobs):
+ READ: bw=341MiB/s (357MB/s), 341MiB/s-341MiB/s (357MB/s-357MB/s), io=10.0GiB (10.7GB), run=30065-30065msec
+
+Disk stats (read/write):
+ sda: ios=10530/9, merge=50/1, ticks=46614/7, in_queue=46621, util=97.41%
+```
+
+#### BeeGFS WRITE
+
+```
+BeeGFS-WRITE: (g=0): rw=write, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
+fio-3.28
+Starting 1 process
+BeeGFS-WRITE: Laying out IO file (1 file / 1024MiB)
+
+BeeGFS-WRITE: (groupid=0, jobs=1): err= 0: pid=22709: Wed Jul 12 05:35:38 2023
+ write: IOPS=34, BW=68.9MiB/s (72.2MB/s)(4132MiB/60005msec); 0 zone resets
+ slat (msec): min=13, max=1582, avg=29.04, stdev=149.64
+ clat (usec): min=7, max=2013.4k, avg=851876.99, stdev=690184.39
+ lat (msec): min=14, max=2027, avg=880.92, stdev=696.79
+ clat percentiles (msec):
+ | 1.00th=[ 57], 5.00th=[ 426], 10.00th=[ 439], 20.00th=[ 439],
+ | 30.00th=[ 443], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443],
+ | 70.00th=[ 447], 80.00th=[ 1955], 90.00th=[ 1972], 95.00th=[ 1989],
+ | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2005], 99.95th=[ 2022],
+ | 99.99th=[ 2022]
+ bw ( KiB/s): min= 4098, max=249978, per=100.00%, avg=108783.27, stdev=52595.86, samples=77
+ iops : min= 2, max= 122, avg=53.09, stdev=25.67, samples=77
+ lat (usec) : 10=0.19%, 20=0.05%
+ lat (msec) : 20=0.24%, 50=0.44%, 100=0.73%, 250=1.60%, 500=68.81%
+ lat (msec) : 2000=26.44%, >=2000=1.50%
+ cpu : usr=0.86%, sys=9.71%, ctx=445229, majf=0, minf=16
+ IO depths : 1=0.2%, 2=0.5%, 4=1.0%, 8=1.9%, 16=3.4%, 32=93.0%, >=64=0.0%
+ submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
+ complete : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
+ issued rwts: total=0,2065,0,0 short=0,0,0,0 dropped=0,0,0,0
+ latency : target=0, window=0, percentile=100.00%, depth=32
+
+Run status group 0 (all jobs):
+ WRITE: bw=68.9MiB/s (72.2MB/s), 68.9MiB/s-68.9MiB/s (72.2MB/s-72.2MB/s), io=4132MiB (4333MB), run=60005-60005msec
+```
+
+#### BeeGFS READ
+
+```
+BeeGFS-READ: (g=0): rw=read, bs=(R) 2049KiB-2049KiB, (W) 2049KiB-2049KiB, (T) 2049KiB-2049KiB, ioengine=libaio, iodepth=32
+fio-3.28
+Starting 1 process
+BeeGFS-READ: Laying out IO file (1 file / 1024MiB)
+
+BeeGFS-READ: (groupid=0, jobs=1): err= 0: pid=22827: Wed Jul 12 05:37:41 2023
+ read: IOPS=67, BW=135MiB/s (142MB/s)(8106MiB/60001msec)
+ slat (msec): min=14, max=215, avg=14.80, stdev= 8.33
+ clat (usec): min=8, max=669906, avg=444903.42, stdev=80659.05
+ lat (msec): min=14, max=684, avg=459.70, stdev=81.14
+ clat percentiles (msec):
+ | 1.00th=[ 72], 5.00th=[ 359], 10.00th=[ 443], 20.00th=[ 447],
+ | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447],
+ | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 464], 95.00th=[ 642],
+ | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 659], 99.95th=[ 667],
+ | 99.99th=[ 667]
+ bw ( KiB/s): min=12294, max=266370, per=99.22%, avg=137265.78, stdev=28838.77, samples=119
+ iops : min= 6, max= 130, avg=66.99, stdev=14.07, samples=119
+ lat (usec) : 10=0.05%, 20=0.15%
+ lat (msec) : 20=0.20%, 50=0.39%, 100=0.59%, 250=2.17%, 500=91.09%
+ lat (msec) : 750=5.36%
+ cpu : usr=0.06%, sys=20.48%, ctx=448073, majf=0, minf=16412
+ IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.2%, 32=93.9%, >=64=0.0%
+ submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
+ complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
+ issued rwts: total=4051,0,0,0 short=0,0,0,0 dropped=0,0,0,0
+ latency : target=0, window=0, percentile=100.00%, depth=32
+
+Run status group 0 (all jobs):
+ READ: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=8106MiB (8500MB), run=60001-60001msec
+```
+
+Tests all ran on `IMAGING-02`.
+Scratch NFS = ~57MB/s read/write
+BeeGFS = ~80MB/s write, ~140MB/s read
+
+This is about a 50% improvement for write operations,
+and a 150% improvement for read operations.
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9