A particular need has risen that a research group needs large amounts of storage space, yet reliable and fast.

Traditionally, central file storage is relatively slow, especially when implemented with NFS, as its designed for availability and reliability and not speed.

As an option, we've installed additional 14TB drives in a few machines, 7TB drives in a few others, and made those available as an aggregate storage volume that is faster than NFS.

The disadvantage is that this architecture requires several services to be up and running else the entire volume/filesystem will become unavailable or unstable.

After considerable testing and experimentation, BeeGFS tends to perform better than other alternatives, though with the aforementioned downside of required services.

For specifics on the testing done, see [Throughput](/Main/BeeGFS#throughput-testing).

We're going off the [BeeGFS Quickstart](https://doc.beegfs.io/latest/quick_start_guide/quick_start_guide.html#example-setup).

We're installing on Ubuntu 22.04 LTS.

First, we'll add the repositories, then we'll add the packages and configure.

Since things have changed slightly since the test (below), we need to add the GPG repository key in a different way than is noted in BeeGFS Quickstart.

`wget -qOhttps://website/release/GPG-KEY | apt-key add -` will no longer work as it has been DEPRECATED.

Now, use the following format:

which will strip the key of its encryption, creating it as a binary.

Next, we'll add the repo:

This should make all beegfs packages available to us.

We need to have a `mgmtd` server, to orchestrate the goings on.

In Imaging, we'll make this `IMAGING-01`.

This means that the MGMTD service needs to be constantly running and available on IMAGING-01, else the BeeGFS filesystem will appear unavailable.

This is done ala:

This will create files in `/opt/beegfs/mgmtd` to keep track.

We will need to edit the `/etc/beegfs/beegfs-mgmtd.conf` file to include the `connauthfile` directive.

We'll create the `connauthfile`:

And that file we will need to be copied to every participating node, whether storage, client, or other.

We need to have a `-meta` server to keep track of the files in the filesystem.

This will be the service that locates data blocks throughout the volume, and tracks which system the data is on.

This service also needs to be constantly available, else clients will not be able to access data.

We'll mirror this later, but for now we only need one, and install it onto `IMAGING-01` with:

Then edit the `/etc/beegfs/beegfs-meta.conf` file to include the `connauthfile` directive.

Of course we have to declare the storage, and in Imaging its no different.

The following is a breakout:

| Host | Device | Size |

| ------------ | ---------------- | ---- |

| `IMAGING-01` | `/dev/sdb1` | 12T |

| `IMAGING-02` | `/dev/sdb1` | 12T |

| `IMAGING-03` | `/dev/nvme1n1p1` | 7T |

| `IMAGING-04` | `/dev/nvme1n1p1` | 7T |

This should create a 37T volume of space, located under… we'll get to that.

First, configuration with:

The `/mnt/14t_drive` is different on `-03` and `-04`, as those are mounted as `/mnt/7t_drive`.

The `-s1` and `-i 101` here indicate that we're on `IMAGING-01`, the first server, and naming the storage target the first one.

On `IMAGING-02`, it would look like `-s2 -i 201`, with the same `-m imaging-01` since that is still our `-mgmtd` node.

Also, edit the `/etc/beegfs/beegfs-storage.conf` to include the `connauthfile` directive.

Of course each system is going to be its own client.

Also, edit the `/etc/beegfs/beegfs-client.conf` to include the `connauthfile` directive.

Additionally, we'll need the `beegfs-helperd` package installed, and configured via the `/etc/beegfs/beegfs-helperd.conf` file to include the `connauthfile` directive.

Finally, each client that will be mounting the filesystem will have a list of mounted and client config files in `/etc/beegfs/beegfs-mount.conf` that looks similar to:

As you guessed it, the system will mount the filesystem onto `/srv/tmp`.

On `IMAGING-01` (the `-mgmtd` and `-meta` node),

Launch:

`systemctl start beegfs-mgmtd`

`systemctl start beegfs-meta`

and check for errors (they may fail silently).

On all participating nodes (`IMAGING-01 thru -04`)

Launch:

`systemctl start beegfs-storage`

`systemctl start beegfs-helperd`

`systemctl start beegfs-client`

and check for errors.

If everything went as expected, you should be able to verify with some `beegfs-utils` commands ala:

> beegfs-ctl --listnodes --nodetype=management --details

> beegfs-ctl --listnodes --nodetype=meta --details

> beegfs-ctl --listnodes --nodetype=storage --details

> beegfs-ctl --listnodes --nodetype=client --details

> beegfs-df

> beegfs-net

> beegfs-check-servers

We'll go through our throughput tests that we've done before.

Tests all ran on `IMAGING-02`.

Scratch NFS = ~57MB/s read/write

BeeGFS = ~80MB/s write, ~140MB/s read

This is about a 50% improvement for write operations,