Table of contents

• Introduction
• Requirements
• Getting Started
• Conclusion

Introduction

The purpose of this tutorial is to demonstrate configuring and using s3fs-fuse to mount a ProfitBricks S3 Object Storage bucket to the local filesystem of a server running CentOS 7. You might use this to copy backups to S3 Object Storage, or retrieve files from S3 Object Storage.

Requirements

• A CentOS 7 server with root access via sudo.
• ProfitBricks S3 Object Storage credentials and a bucket.

Getting Started

Use yum to update the OS:

sudo yum update

Install the various s3fs dependencies:

sudo sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

You will be prompted to download any of those packages that aren't currently installed along with a long list of associated dependencies. Once that completes, we'll move on to using git to retrieve the s3fs-fuse software from GitHub. Make sure you are in a directory you can write into, perhaps your homedir or /tmp.

git clone https://github.com/s3fs-fuse/s3fs-fuse.git

Now change into the directory that was just created:

cd s3fs-fuse

Compile and install it.

./autogen.sh

--- Make commit hash file -------
--- Finished commit hash file ---
--- Start autotools -------------
configure.ac:26: installing './config.guess'
configure.ac:26: installing './config.sub'
configure.ac:27: installing './install-sh'
configure.ac:27: installing './missing'
src/Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'
--- Finished autotools ----------


./configure

checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
...
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating test/Makefile
config.status: creating doc/Makefile
config.status: creating config.h
config.status: executing depfiles commands


make

make  all-recursive
make[1]: Entering directory `/tmp/s3fs-fuse'
Making all in src
make[2]: Entering directory `/tmp/s3fs-fuse/src'
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT s3fs.o -MD -MP -MF .deps/s3fs.Tpo -c -o s3fs.o s3fs.cpp
mv -f .deps/s3fs.Tpo .deps/s3fs.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT curl.o -MD -MP -MF .deps/curl.Tpo -c -o curl.o curl.cpp
...
g++  -g -O2 -Wall -D_FILE_OFFSET_BITS=64   -o test_string_util string_util.o test_string_util.o
make[2]: Leaving directory `/tmp/s3fs-fuse/src'
Making all in test
make[2]: Entering directory `/tmp/s3fs-fuse/test'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/tmp/s3fs-fuse/test'
Making all in doc
make[2]: Entering directory `/tmp/s3fs-fuse/doc'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/tmp/s3fs-fuse/doc'
make[2]: Entering directory `/tmp/s3fs-fuse'
make[2]: Leaving directory `/tmp/s3fs-fuse'
make[1]: Leaving directory `/tmp/s3fs-fuse'

sudo make install

Making install in src
make[1]: Entering directory `/tmp/s3fs-fuse/src'
make[2]: Entering directory `/tmp/s3fs-fuse/src'
 /usr/bin/mkdir -p '/usr/local/bin'
  /usr/bin/install -c s3fs '/usr/local/bin'
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/tmp/s3fs-fuse/src'
make[1]: Leaving directory `/tmp/s3fs-fuse/src'
Making install in test
make[1]: Entering directory `/tmp/s3fs-fuse/test'
make[2]: Entering directory `/tmp/s3fs-fuse/test'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/tmp/s3fs-fuse/test'
make[1]: Leaving directory `/tmp/s3fs-fuse/test'
Making install in doc
make[2]: Entering directory `/tmp/s3fs-fuse/doc'
make[2]: Nothing to be done for `install-exec-am'.
 /usr/bin/mkdir -p '/usr/local/share/man/man1'
  /usr/bin/install -c -m 644 man/s3fs.1 '/usr/local/share/man/man1'
make[1]: Leaving directory `/tmp/s3fs-fuse/doc'
make[1]: Entering directory `/tmp/s3fs-fuse'
make[2]: Entering directory `/tmp/s3fs-fuse'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/tmp/s3fs-fuse'
make[1]: Leaving directory `/tmp/s3fs-fuse'

If everything completed successfully and you received output similar to what was shown above, then the software is now installed.

Now we need to create the a credentials file to hold our ProfitBricks S3 object storage key and secret. You can get this information from inside the ProfitBricks DCD.

Once you have the values for your key and secret, place them in a text file located in your homedir (~/.passwd-s3fs), in /etc/passwd-s3fs, or in another location you prefer. The key and secret should be separated by a colon. Here is one method that could create the file:

echo MYKEY:MYSECRET > ~/.passwd-s3fs

If your key or secret contain special characters, you may need to adjust that command, or use an editor such as nano, pico, or vi to create the file.

It is a good idea to restrict the permissions by running chmod 600 on the file as well.

chmod 600 ~/.passwd-s3fs

Lets go ahead and create a directory under /mnt called s3test:

mkdir /mnt/s3test

Now we can start the s3fs process, and mount an existing bucket named examplebucket into /mnt/s3test. We are also passing a couple of options using -o.

sudo s3fs examplebucket /mnt/s3test -o passwd_file=~/.passwd-s3fs -o url=https://s3-de-central.profitbricks.com/ -o use_path_request_style -o dbglevel=info

We will be returned to the command prompt, and can now try to access /mnt/s3test and see if the contents of our S3 object storage bucket are displayed:

ls /mnt/s3test

american-flag-30.png  american-flag-small.png  german-flag-50.png     vagrant-logo.png
american-flag-50.png  german-flag-30.png       german-flag-small.png

Since we used -o dbglevel=info when starting s3fs, we can also check /var/log/messages for entries containing s3fs. This could be useful if something isn't working as you expected.

Using s3fs this way could allow you to fairly quickly gain access to files that you may want to copy onto your local system.

If you want to configure your system so that the S3 bucket is mounted when the system boots, then an entry can be added to /etc/fstab. Make sure you have the credentials stored in /etc/passwd-s3fs.

s3fs#examplebucket /mnt/s3test fuse _netdev,allow_other,use_path_request_style,url=https://s3-de-central.profitbricks.com 0 0

Once that entry is in place, then the bucket should be automatically mounted when the system boots.

Additionally, with the /etc/fstab entry in place, you can utilize mount and umount as needed:

sudo umount /mnt/s3test

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        21G  1.7G   18G   9% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  8.4M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs           379M     0  379M   0% /run/user/0

sudo mount /mnt/s3test

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        21G  1.7G   18G   9% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  8.4M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs           379M     0  379M   0% /run/user/0
s3fs            256T     0  256T   0% /mnt/s3test

Conclusion

This tutorial provided a quick introduction to using s3fs-fuse with ProfitBricks S3 Object Storage. If you would like more information, please visit the s3fs-fuse GitHub Project. If you have some questions, or get this working and want to share your use case, please comment below.