XtreemFS Installation and User Guide

Changes

• Improved Stability
  Fixed client crashes when timeouts occurred. Fixed server crashes and issues with the R/W replication.
• Full support for asynchronous writes
  If enabled, write() requests will be immediately acknowledged by the client and executed in the background. Outstanding writes are always flushed at close() or fsync*() events. This improves the write throughput, especially in case of connections with high latency. See section  for more information.
• Windows Client (Beta)
  Rewrite of Windows Client (based on C++ libxtreemfs) which uses Eldos' Callback File System product instead of Dokan.
• Re-wrote HDFS interface implementation
  Use XtreemFS as replacement for HDFS in your Hadoop setup. See section  how to use it.
• libxtreemfs for Java
  Access XtreemFS directly from your Java application. See section  how to use it.
• Re-added and improved Vivaldi support
  Vivaldi is directly integrated in the client now and there's a visualization of the coordinates available, even of the client coordinates. See section .
• Support for custom attributes in the OSD Selection policies
  Added support for OSD Selection based on Custom Attributes e.g., assign a country code to every OSD as custom attribute and limit the placement of files on OSDs based on the attribute (see ).

• new client
  We have re-written the client from scratch. The new client supports automatic fail-over for replicated files and metadata caching.
• libxtreemfs
  libxtreemfs is a convenient C++ library to use XtreemFS directly without a mounted client or the VFS layer. The new client is built on top of this library. A java version of libxtreemfs is planned.
• File system snapshots
  XtreemFS now supports snapshots. A snapshot reflects a momentary state of a volume or directory. It can be mounted and read-only accessed.
• Full file replication
  Starting with this release, XtreemFS supports full file replication. Read/write replicated files offer regular file system semantics and work with all applications.
• DIR, MRC replication
  The DIR and MRC can now be replicated using the BabuDB database replication. The replication works with a primary and backups. If the primary fails, a backup will automatically take over after a short time.
• xtfsutil
  We have replaced all user tools with a single binary. The new tool doesn't require java anymore.
• OSD drain
  With OSD drain, files can be removed from an OSD without interrupting the system. A fully drained OSD can be removed from the system without data loss.

• server status
  Each server (especially OSDs) have a persistent status which can be online or dead/removed. This status must be changed manually and is used by the scrubber tool to identify dead OSDs which have been removed from the system.
• enhanced scrubber
  The scrubber is now able to remove replicas which are stored on OSDs that are marked as dead/removed. The scrubber will create new replicas for that file if a complete replica still exists and a sufficient number of OSDs is available. In addition, the scrubber marks replicas as "complete" if they contain all objects of the original file.

• renamed binaries
  We renamed most binaries to conform with Linux naming conventions, e.g. xtfs_mount is now mount.xtreemfs. However, we added links with the old names for compatibility. For a full list see Sec. .
• "Grid SSL" mode
  In this mode, SSL is only used for authentication (handshake) and regular TCP is used for communication afterwards. For more details see Sec. .
• the xctl utility
  The new release includes a command line utility xctl for starting and stopping the services. This tool is useful if you don't want a package based installation or if you don't have root privileges.
• vivaldi
  XtreemFS now includes modules for calculating Vivaldi network coordinates to reflect the latency between OSDs and clients. An OSD and replica selection policy for vivaldi is also available. For details, see Sec. .

Chapter 1
Quick Start

1. Download XtreemFS RPMs/DEBs and install
   1. Download the RPMs or DEBs for your system from the XtreemFS website (http://www.xtreemfs.org)
   2. open a root console (su or sudo)
   3. install with rpm -Uhv xtreemfs*-1.3.x.rpm
2. Start the Directory Service:
   /etc/init.d/xtreemfs-dir start
3. Start the Metadata Server:
   /etc/init.d/xtreemfs-mrc start
4. Start the OSD:
   /etc/init.d/xtreemfs-osd start
5. If not already loaded, load the FUSE kernel module:
   modprobe fuse
6. Depending on your distribution, you may have to add users to a special group to allow them to mount FUSE file systems. In openSUSE users must be in the group trusted, in Ubuntu in the group fuse. You may need to log out and log in again for the new group membership to become effective.
7. You can now close the root console and work as a regular user.
8. Wait a few seconds for the services to register at the directory service. You can check the registry by opening the DIR status page in your favorite web browser http://localhost:30638.
9. Create a new volume with the default settings:
   mkfs.xtreemfs localhost/myVolume
10. Create a mount point:
    mkdir ~/xtreemfs
11. Mount XtreemFS on your computer:

    	mount.xtreemfs localhost/myVolume ~/xtreemfs
    	
    
12. Have fun ;-)
13. To un-mount XtreemFS:
    umount.xtreemfs ~/xtreemfs

Chapter 2
About XtreemFS

2.1  What is XtreemFS?

2.2  Is XtreemFS suitable for me?

... an open source file system. It is distributed freely and can be used by anyone without limitations.
... a POSIX file system. Users can mount and access XtreemFS like any other common file system. Application can access XtreemFS via the standard file system interface, i.e. without having to be rebuilt against a specialized API. XtreemFS supports a POSIX-compliant access control model.
... a multi-platform file system. Server and client modules can be installed and run on different platforms, including most Linux distributions, Solaris, Mac OS X and Windows.
... a globally distributed file system. Unlike cluster file systems, an XtreemFS installation is not restricted to a single administrative domain or cluster. It can span the globe and may comprise servers in different administrative domains.
... a failure-tolerant file system. As stated in the previous section, replication can keep the system alive and the data safe. In this respect, XtreemFS differs from most other open-source file systems.
... a secure file system. To ensure security in an untrusted, worldwide network, all network traffic can be encrypted with SSL connections, and users can be authenticated with X.509 certificates.
... a customizable file system. Since XtreemFS can be used in different environments, we consider it necessary to give administrators the possibility of adapting XtreemFS to the specific needs of their users. Customizable policies make it possible change the behavior of XtreemFS in terms of authentication, access control, striping, replica placement, replica selection and others. Such policies can be selected from a set of predefined policies, or implemented by administrators and plugged in the system.

... a high-performance cluster file system. Even though XtreemFS reaches acceptable throughput rates on a local cluster, it cannot compete with specialized cluster file systems in terms of raw performance numbers. Most such file systems have an optimized network stack and protocols, and a substantially larger development team. If you have huge amounts of data on a local cluster with little requirements but high throughput rates to them, a cluster file system is probably the better alternative.
... a replacement for a local file system. Even though XtreemFS can be set up and mounted on a single machine, the additional software stack degrades the performance, which makes XtreemFS a bad alternative.

2.3  Core Features

2.4  Architecture

Figure 2.1: The XtreemFS architecture and components.

• DIR - Directory Service
  The directory service is the central registry for all services in XtreemFS. The MRC uses it to discover storage servers.
• MRC - Metadata and Replica Catalog
  The MRC stores the directory tree and file metadata such as file name, size or modification time. Moreover, the MRC authenticates users and authorizes access to files.
• OSD - Object Storage Device
  An OSD stores arbitrary objects of files; clients read and write file data on OSDs.

Chapter 3
XtreemFS Services

3.1  Installation

3.1.1  Prerequisites

3.1.2  Installing from Pre-Packaged Releases

$> rpm -i xtreemfs-server-1.3.x.rpm xtreemfs-backend-1.3.x.rpm

$> dpkg -i xtreemfs-server-1.3.x.deb xtreemfs-backend-1.3.x.deb

$> rpm -i --nodeps xtreemfs-server-1.3.x.rpm \
   xtreemfs-backend-1.3.x.rpm

$> dpkg -i --ignore-depends java6-runtime \
   xtreemfs-server-1.3.x.deb xtreemfs-backend-1.3.x.deb

$> export JAVA_HOME=/usr/java6

3.1.3  Installing from Sources

$> make server

$> sudo make install-server

$> sudo /etc/xos/xtreemfs/postinstall_setup.sh

3.2  Configuration

3.2.1  A Word about UUIDs

3.2.2  Automatic DIR Discovery

3.2.3  Authentication

authentication_provider = <classname>

• org.xtreemfs.common.auth.NullAuthProvider
  uses local user and group IDs
• org.xtreemfs.common.auth.SimpleX509AuthProvider
  uses X.509 certificates; user and group IDs are extracted from the distinguished names of the certificates

3.2.4  Configuring SSL Support

/etc/xos/xtreemfs/truststore/certs

Converting PEM files to PKCS#12

$> openssl pkcs12 -export -in ds.pem -inkey ds.key \
   -out ds.p12 -name "DS"
$> openssl pkcs12 -export -in mrc.pem -inkey mrc.key \
   -out mrc.p12 -name "MRC"
$> openssl pkcs12 -export -in osd.pem -inkey osd.key \
   -out osd.p12 -name "OSD"

Importing trusted certificates from PEM into a JKS

$> keytool -import -alias rootca -keystore trusted.jks \
   -trustcacerts -file ca-cert.pem

keytool error: java.lang.Exception: Input not an X.509 certificate

Sample Setup

1. Set up your test CA.
   1. Create a directory for your CA files

      		$> mkdir ca
      		
      
   2. Create a private key and certificate request for your CA.

      		$> openssl req -new -newkey rsa:1024 -nodes -out ca/ca.csr \
      		   -keyout ca/ca.key
      		
      

      Enter something like XtreemFS-DEMO-CA as the common name (or something else, but make sure the name is different from the server and client name!).
   3. Create a self-signed certificate for your CA which is valid for one year.

      		$> openssl x509 -trustout -signkey ca/ca.key -days 365 -req \
      		   -in ca/ca.csr -out ca/ca.pem
      		
      
   4. Create a file with the CA's serial number

      		$> echo "02" > ca/ca.srl
      		
      
2. Set up the certificates for the services and the XtreemFS Client.
   Replace SERVICE with dir, mrc, osd and client.
   1. Create a private key for the service.
      Use XtreemFS-DEMO-SERVICE as the common name for the certificate.

      $> openssl req -new -newkey rsa:1024 -nodes \
         -out SERVICE.req \
         -keyout SERVICE.key
      		
      
   2. Sign the certificate with your demo CA.
      The certificate is valid for one year.

      $> openssl x509 -CA ca/ca.pem -CAkey ca/ca.key \
         -CAserial ca/ca.srl -req \
         -in SERVICE.req \
         -out SERVICE.pem -days 365
      		
      
   3. Export the service credentials (certificate and private key) as a PKCS#12 file.
      Use "passphrase" as export password. You can leave the export password empty for the XtreemFS Client to avoid being asked for the password on mount.

      $> openssl pkcs12 -export -in SERVICE.pem -inkey SERVICE.key \
         -out SERVICE.p12 -name "SERVICE"
      		
      
   4. Copy the PKCS#12 file to the certificates directory.

      $> mkdir -p /etc/xos/xtreemfs/truststore/certs
      $> cp SERVICE.p12 /etc/xos/xtreemfs/truststore/certs
      		
      
3. Export your CA's certificate to the trust store and copy it to the certificate dir.
   You should answer "yes" when asked "Trust this certificate".
   Use "jks_passphrase" as passphrase for the keystore.

   	$> keytool -import -alias ca -keystore trusted.jks \
   	   -trustcacerts -file ca/ca.pem
   	$> cp  trusted.jks /etc/xos/xtreemfs/truststore/certs
   	
   
4. Configure the services. Edit the configuration file for all your services. Set the following configuration options (see Sec. 3.2 for details).
   ssl.enabled = true
   ssl.service_creds.pw = passphrase
   ssl.service_creds.container = pkcs12
   ssl.service_creds = /etc/xos/xtreemfs/truststore/certs/service.p12
   ssl.trusted_certs = /etc/xos/xtreemfs/truststore/certs/trusted.jks
   ssl.trusted_certs.pw = jks_passphrase
   ssl.trusted_certs.container = jks
5. Start up the XtreemFS services (see Sec. ).
6. Create a new volume (see Sec. for details).
   Use

   	$> mkfs.xtreemfs --pkcs12-file-path=\
   	   /etc/xos/xtreemfs/truststore/certs/client.p12 pbrpcs://localhost/test
   	
   

   for SSL-enabled servers, or

   	$> mkfs.xtreemfs --pkcs12-file-path=\
   	   /etc/xos/xtreemfs/truststore/certs/client.p12 pbrpcg://localhost/test
   	
   

   for Grid-SSL-enabled servers.
7. Mount the volume (see Sec. for details).
   Use

   	$> mount.xtreemfs --pkcs12-file-path=\
   	   /etc/xos/xtreemfs/truststore/certs/client.p12 pbrpcs://localhost/test /mnt
   	
   

   for SSL-enabled servers, or

   	$> mount.xtreemfs --pkcs12-file-path=\
   	   /etc/xos/xtreemfs/truststore/certs/client.p12 pbrpcg://localhost/test /mnt
   	
   

   for Grid-SSL-enabled servers.

3.2.5  List of Configuration Options

admin_password optional

authentication_provider

babudb.baseDir

babudb.cfgFile optional

babudb.checkInterval optional

babudb.compression optional

babudb.debug.level optional

babudb.logDir

babudb.maxLogfileSize optional

babudb.pseudoSyncWait optional

babudb.sync

babudb.worker.maxQueueLength optional

babudb.worker.numThreads optional

capability_secret

capability_timeout optional

checksums.enabled

checksums.algorithm

debug.level optional

debug.categories optional

dir_service.host

dir_service.port

discover optional

flease.dmax_ms optional

flease.lease_timeout_ms optional

flease.message_to_ms optional

flease.retries optional

geographic_coordinates optional

hostname optional

http_port

ignore_capabilities optional

listen.address optional

listen.port

local_clock_renewal

max_client_queue

monitoring.enabled

monitoring.email.programm

monitoring.email.receiver

monitoring.email.sender

monitoring.max_warnings

monitoring.service_timeout_s

no_atime

object_dir

osd_check_interval

policy_dir optional

remote_time_sync

renew_to_caps optional

report_free_space

socket.send_buffer_size optional

snmp.enabled optional

snmp.address optional

snmp.port optional

snmp.aclfile optional

ssl.enabled

ssl.grid_ssl

ssl.service_creds

ssl.service_creds.container

ssl.service_creds.pw

ssl.trusted_certs

ssl.trusted_certs.container

ssl.trust_manager optional

ssl.trusted_certs.pw

startup.wait_for_dir

storage_layout optional

uuid

vivaldi.max_clients

vivaldi.client_timeout

vivaldi.recalculation_interval_ms

vivaldi.recalculation_epsilon_ms

vivaldi.iterations_before_updating

vivaldi.max_retries_for_a_request

vivaldi.max_request_timeout_ms

vivaldi.timer_interval_ms

3.3  Execution and Monitoring

3.3.1  Starting and Stopping the XtreemFS services

	$> /etc/init.d/xtreemfs-dir start
	$> /etc/init.d/xtreemfs-mrc start
	$> /etc/init.d/xtreemfs-osd start
	

	$> /etc/init.d/xtreemfs-dir stop
	$> /etc/init.d/xtreemfs-mrc stop
	$> /etc/init.d/xtreemfs-osd stop
	

3.3.2  Web-based Status Page

Figure 3.1: OSD status web page

3.3.3  DIR Service Monitoring

3.3.4  Monitoring Services with SNMP

snmp.enabled = true
snmp.address = localhost
snmp.port = 34638
snmp.aclfile =  /etc/xos/xtreemfs/snmp.acl

acl = {
 {
 communities = public
 access = read-only
 managers = localhost
 }
}

 $> snmpget -v2c -cpublic localhost:34638 1.3.6.1.4.1.38350.1.1.0 

3.3.5  Monitoring with Ganglia

[(i)]
1. your ganglia installation supports python modules
2. you installed python and the pysnmp³ library which is used by the plugins

modules {
  module {
    name = "xtfs-dir-plugin"
    language = "python"
    param Host {
      value = localhost
    }
    param Port {
      value = 34638
    }
    param CommunityString {
      value = public
    }
  }
}

collection_group {
  collect_every = 300
  time_threshold = 100
  metric {
    name = "dir_jvm_used_mem"
    title = "used memory of the jvm"
    value_threshold = 1
  }
  metric {
    name = "dir_jvm_free_mem"
    title = "free memory of the jvm"
    value_threshold = 1
  } 
  metric {
    name = "dir_client_connections"
    title = "number of Clients"
    value_threshold = 1
  }
  metric {
    name = "dir_pending_requests"
    title = "number of pending requests"
    value_threshold = 1
  }
  metric {
    name = "addr_mapping_count"
    title = "number of address mappings"
    value_threshold = 1
  }
  metric {
    name = "service_count"
    title = "number of services"
    value_threshold = 1
  }
  metric {
    name = "dir_status"
    title = "Status DIR"
  }
  metric {
    name = "dir_uuid"
    title = "DIR UUID"
  }
}

3.4  Troubleshooting

• You have root permissions when starting the service. Running the init.d scripts requires root permissions. However, the services themselves are started on behalf of a user xtreemfs.
• DIR has been started before MRC and OSD. Problems may occur if a script starts multiple services as background processes.
• There are no firewall restrictions that keep XtreemFS services from communicating with each other. The default ports that need to be open are: 32636 (MRC, TCP), 32638 (DIR, TCP), and 32640 (OSD, TCP & UDP).
• The MRC database version is correct. In case of an outdated database version, the xtfs_mrcdbtool commands of the old and new XtreemFS version can dump and restore the database, respectively (see Sec. ).
• A network interface is available on the host. It may be either bound to an IPv4 or IPv6 address.

Chapter 4
XtreemFS Client

4.1  Installation

4.1.1  Prerequisites

4.1.2  Installing from Pre-Packaged Releases

$> rpm -i xtreemfs-client-1.3.x.rpm

$> dpkg -i xtreemfs-client-1.3.x.deb

4.1.3  Installing from Sources

$> make client

• RPM-based distros:

  cmake >= 2.6
  gcc-c++ >= 4.1
  fuse >= 2.6
  fuse-devel >= 2.6
  boost-devel >= 1.35
  openssl-devel >= 0.9.8
  libattr-devel >= 2
  
  
• DEB-based distros:

  cmake (>= 2.6)
  build-essential (>=11)
  libfuse-dev (>= 2.6)
  libssl-dev (>= 0.9)
  libattr-dev (>= 2)
  libboost-system1.35-dev or later
  libboost-thread1.35-dev or later
  libboost-program-options1.35-dev or later
  libboost-regex1.35-dev or later
  
  

$> sudo make install-client

4.2  Volume Management

4.2.1  Creating Volumes

$> mkfs.xtreemfs my-mrc-host.com/myVolume

4.2.2  Deleting Volumes

$> rmfs.xtreemfs my-mrc-host.com/myVolume

4.2.3  Listing all Volumes

$> lsfs.xtreemfs my-mrc-host.com

4.3  Accessing Volumes

4.3.1  Mounting and Un-mounting

$> su
Password:
#> modprobe fuse
#> exit

$> mount.xtreemfs remote.dir.machine/myVolume /xtreemfs

xtreemfs@localhost/xtreemfs on /xtreemfs type fuse (...)

$> umount.xtreemfs /xtreemfs

$> umount /xtreemfs

4.3.2  Mount Options

4.3.3  Asynchronous Writes Support

• Maximum number of pending requests. This can be configured with the option --async-writes-max-reqs. Do not set this value above 100 or increase the OSD configuration parameter max_client_queue accordingly.
• Maximum buffer size. The buffer size is determined as product of the number of pending requests and the assumed maximum request size. On Linux, the assumed maximum request size is set to 128 kB i.e., the default buffer size on Linux is 1280 kB (10 * 128 kB). If you like, you can change the assumed maximum request size with the hidden client option --async-writes-max-reqsize-kb. However, note that FUSE on Linux does not support write() requests larger than 128 kB.

4.4  Troubleshooting

$> mount.xtreemfs -f -d DEBUG remote.dir.machine/myVolume /xtreemfs

Chapter 5
XtreemFS Tools

5.1  Installation

5.1.1  Prerequisites

5.1.2  Installing from Pre-Packaged Releases

$> rpm -i xtreemfs-tools-1.3.x.rpm xtreemfs-backend-1.3.x.rpm

$> dpkg -i xtreemfs-tools-1.3.x.deb xtreemfs-backend-1.3.x.deb

$> rpm -i --nodeps xtreemfs-tools-1.3.x.rpm \
   xtreemfs-backend-1.3.x.rpm

$> dpkg -i --ignore-depends java6-runtime \
   xtreemfs-tools-1.3.x.deb xtreemfs-backend-1.3.x.deb

$> export JAVA_HOME=/usr/java6

5.1.3  Installing from Sources

$> make server

$> sudo make install-tools

5.2  Admin Tools

5.2.1  MRC Database Backups and Conversion

1. Create an XML representation of the old database with the old MRC version.
2. Update the MRC to the new version.
3. Restore the database from the XML representation.

$> xtfs_mrcdbtool -mrc pbrpc://my-mrc-host.com:32636 \
   dump /tmp/dump.xml

$> xtfs_mrcdbtool -mrc pbrpc://my-mrc-host.com:32636 \
   restore /tmp/dump.xml

5.2.2  Scrubbing and Cleanup

$> xtfs_scrub -dir pbrpc://my-dir-host.com:32638 -repair myVolume

• ensure that the file size stored with the metadata reflects the actual file size on the OSD(s),
• check whether an invalid checksum in the OSD indicates a corrupted file content,
• check if one or more read-only replicated replicas are on an OSD marked as "removed" (which can be done with the xtfs_chstatus tool) and restore the desired numbers of replicas by adding new replicas, if necessary

$> xtfs_cleanup -dir pbrpc://localhost:32638 \
   uuid:u2i3-28isu2-iwuv29-isjd83

5.2.3  Setting the OSD Status

$> xtfs_chstatus -dir pbrpc://localhost:32638 \
   u2i3-28isu2-iwuv29-isjd83 online

5.2.4  Draining OSDs

$> xtfs_remove_osd -dir localhost:32638 \
   uuid:8bca70da-c963-43c7-b30b-d0d605d39fa7

How it Works

1. The OSD is marked as "locked". This restrains it from being assigned to newly created files.
2. A new replica is added to each file stored on the OSD. The OSD used for this replica is selected according to the volume's OSD selection policy (see Section ).
3. Once the new replica has become consistent, i.e. all content has been copied from the original replica, the original replica (which is stored on the OSD that is supposed to be drained) is removed.
4. As soon as all replicas have been removed, the draining procedure has finished.

Limitations

5.3  User Tools

5.3.1  xtfsutil for Files

$> cd /xtreemfs
$> echo 'Hello World' > test.txt
$> xtfsutil test.txt

Path (on volume)     /test.txt
XtreemFS file Id     1089e4fb-9eb9-46ea-8acf-91d10c2170e3:2
XtreemFS URL         pbrpc://localhost:32638/xtreemfs/test.txt
Owner                user
Group                users
Type                 file
Replication policy   WqRq
XLoc version         0
Replicas:
  Replica 1
     Striping policy     STRIPING_POLICY_RAID0 / 1 / 128kB
     Replication Flags   partial
     OSD 1               test-osd1/127.0.0.1:32641
  Replica 2
     Striping policy     STRIPING_POLICY_RAID0 / 1 / 128kB
     Replication Flags   partial
     OSD 1               test-osd0/127.0.0.1:32640
  Replica 3
     Striping policy     STRIPING_POLICY_RAID0 / 1 / 128kB
     Replication Flags   partial
     OSD 1               test-osd2/127.0.0.1:32642

Changing the Replication Policy

$> xtfsutil --set-replication-policy ronly /xtreemfs/test.txt

none

File is not replicated.

ronly (readonly)

File is read-only replicated and will be marked as read-only, i.e. the file cannot be modified as long as the replication policy is set to ronly.

WqRq (quorum), WaR1 (all)

The file will be read-write replicated and can be modified. Please refer to the Section for more information on both policies and their properties.

Adding and Removing Replicas

$> xtfsutil --add-replica AUTO /xtreemfs/test.txt

$> xtfsutil --list-osds /xtreemfs/test.txt
OSDs suitable for new replicas: 
  test-osd1
  test-osd2

$> xtfsutil --delete-replica test-osd1 /xtreemfs/test.txt

5.3.2  xtfsutil for Volumes

$> xtfsutil /xtreemfs

Path (on volume)     /
XtreemFS file Id     1089e4fb-9eb9-46ea-8acf-91d10c2170e3:1
XtreemFS URL         pbrpc://localhost:32638/replicated
Owner                user
Group                users
Type                 volume
Free/Used Space      24 GB / 6 bytes
Num. Files/Dirs      1 / 1
Access Control p.    2
OSD Selection p.     1000,3002
Replica Selection p. default
Default Striping p.  STRIPING_POLICY_RAID0 / 1 / 128kB
Default Repl. p.     WqRq with 3 replicas

Changing the Default Striping Policies

$> xtfsutil --set-dsp -p RAID0 -w 4 -s 256 /xtreemfs

Changing the Default Replication Policy

$> xtfsutil --set-drp --replication-policy WqRq \
            --replication-factor 3 /xtreemfs

none

New files are not replicated.

ronly (readonly)

Files are initially created without replicas and can be modified until they are closed. On close, the file is set to read-only and the replicas are created. Replicas are partial by default. Full replicas will be created if the --full flag is set.

WqRq (quorum), WaR1 (all)

New files are read-write replicated and can be modified. Please refer to the Section for more information on both policies and their properties.

5.3.3  Changing OSD and Replica Selection Policies

 $> xtfsutil --set-rsp dcmap /xtreemfs

 $> xtfsutil --set-osp dcmap /xtreemfs

default

The default OSD selection policy selects a random subset of OSDs that are responsive and have more than 2GB of free disk space.

fqdn

Selects OSDs based on the size of the post-fix match of the fully qualified domain names and on the free space.

dcmap

Selects OSDs based on the distance defined in the datacenter map and on the free space.

vivaldi

Selects OSDs based on the distance of the Vivaldi coordinates between client and OSD and on the free space.

5.3.4  Setting and Listing Policy Attributes

 $> xtfsutil --set-pattr 1001.domains \
    --value "*.xtreemfs.org bla.com" /xtreemfs

 $> xtfsutil --set-pattr 1001.domains --value "" /xtreemfs

 $> xtfsutil --list-pattrs /xtreemfs

5.3.5  Modifying Access Control Lists

 $> xtfsutil --set-acl u:someone:rx /xtreemfs

 $> xtfsutil --del-acl u:someone /xtreemfs

5.3.6  Snapshots

$> xtfsutil --enable-snapshots /path/to/mounted/volume

$> xtfsutil --create-snapshot /path/to/mounted/volume/directory \
   mySnapshot

$> xtfsutil --create-snapshot-non-recursive \
   /path/to/mounted/volume/directory mySnapshot

$> xtfsutil --list-snapshots /path/to/mounted/volume

$> mount.xtreemfs localhost/volume@mySnapshot \
   /path/to/mounted/volume2

$> xtfsutil --delete-snapshot /path/to/mounted/volume mySnapshot

$> xtfs_cleanup -dir localhost:32638 -v \
   uuid:8bca70da-c963-43c7-b30b-d0d605d39fa7

5.4  Test Tools

Chapter 6
Replication

6.1  Read/Write File Replication

6.1.1  Technical Details

6.1.2  Limitations

6.1.3  Setup

6.2  Read-Only File Replication

6.2.1  Limitations

6.2.2  Setup

6.3  MRC and DIR Replication

6.3.1  Technical Details

6.3.2  Setup

• Enable the replication plug-in on all replicated MRC/DIR instances
• Configure replication parameters across all instances
• Start up all replicated MRC/DIR instances

Enabling and Configuring MRC Replication

babudb.sync = FSYNCDATA

babudb.plugin.0 = /etc/xos/xtreemfs/server-repl-plugin/mrc.properties

...

# participants of the replication including this replica
babudb.repl.participant.0 = first-MRC-replica
babudb.repl.participant.0.port = 35676
babudb.repl.participant.1 = second-MRC-replica
babudb.repl.participant.1.port = 35676
babudb.repl.participant.2 = third-MRC-replica
babudb.repl.participant.2.port = 35676

...

# number of servers that at least have to be up to date
babudb.repl.sync.n = 2

...

Enabling and Configuring DIR Replication

dir_service.host = first-DIR-replica
dir_service.port = 32638
dir_service1.host = second-DIR-replica
dir_service1.port = 32638
dir_service2.host = third-DIR-replica
dir_service2.port = 32638

mount.xtreemfs first-DIR-replica,second-DIR-replica,third-DIR-replica/myVol

Startup and Access

Chapter 7
Policies

7.1  Authentication Policies

7.1.1  UNIX uid/gid - NullAuthProvider

7.1.2  Plain SSL Certificates - SimpleX509AuthProvider

7.2  Authorization Policies

• Authorize All Policy (policy Id 1)
  No authorization - everyone can do everything. This policy is useful if performance of metadata operations matters more than security, since no evaluation of access rights is needed.
• POSIX ACLs & Permissions (policy Id 2)
  This access policy implements the traditional POSIX permissions commonly used on Linux, as well as POSIX ACLs, an extension that provides for access control at the granularity of single users and groups. POSIX permissions should be used as the default, as it guarantees maximum compatibility with other file systems.
• Volume ACLs (policy Id 3)
  Volume ACLs provide an access control model similar to POSIX ACLs & Permissions, but only allow one ACL for the whole volume. This means that there is no recursive evaluation of access rights which yields a higher performance at the price of a very coarse-grained access control.

7.3  OSD and Replica Selection Policies

7.3.1  Attributes

7.3.2  Predefined Policies

Filtering Policies

• Default OSD filter (policy ID 1000)
  Removes OSDs from the list that are either dead or do not have sufficient space. By default, the lower space limit for an OSD is 2GB, and the upper response time limit is 5 minutes.
  
  Attributes:
  • free_capacity_bytes: the lower space limit in bytes
  • offline_time_secs: the upper response time limit in seconds
  • custom attributes (see Section )
• FQDN-based filter (policy ID 1001)
  Removes OSDs from the list that do not match any of the domains in a given set. By default, the set of domains contains `*', which indicates that no domains are removed.
  
  Attributes:
  • domains: a comma or space-separated list of domain names. The list may include leading and trailing `*'s, which will be regarded as wildcard characters.
• UUID-based filter (policy ID 1002)
  Removes OSDs from the list that do not match any of the UUIDs in a given set. By default, the set of UUIDs contains `*', which indicates that no domains are removed.
  
  Attributes:
  • uuids: a comma or space-separated list of OSD UUIDs. The list may include leading and trailing `*'s, which will be regarded as wildcard characters.

Grouping Policies

• Data center map-based grouping (policy ID 2000)
  Removes all OSDs from the OSD set that have been used in the file's replica locations list already and selects the subset of OSDs that is closest to the client and provides enough OSDs for the new replica in a single data center.
  This policy uses a statically configured datacenter map that describes the distance between datacenters. It works only with IPv4 addresses at the moment. Each datacenter has a list of matching IP addresses and networks which is used to assign clients and OSDs to datacenters. Machines in the same datacenter have a distance of 0.
  This policy requires a datacenter map configuration file in
  /etc/xos/xtreemfs/datacentermap on the MRC machine which is loaded at MRC startup. This config file must contain the following parameters:
  • datacenters=A,B,C
    A comma separated list of datacenters. Datacenter names may only contain a-z, A-Z, 0-9 and _.
  • distance.A-B=100
    For each pair of datacenters, the distance must be specified. As distances are symmetric, it is sufficient to specify A to B.
  • A.addresses=192.168.1.1,192.168.2.0/24
    For each datacenter a list of matching IP addresses or networks must be specified.
  • max_cache_size=1000
    Sets the size of the address cache that is used to lookup IP-to-datacenter matches.
  A sample datacenter map could look like this:

  	datacenters=BERLIN,LONDON,NEW_YORK
  	distance.BERLIN-LONDON=10
  	distance.BERLIN-NEW_YORK=140
  	distance.LONDON-NEW_YORK=110
  	BERLIN.addresses=192.168.1.0/24
  	LONDON.addresses=192.168.2.0/24
  	NEW_YORK.addresses=192.168.3.0/24,192.168.100.0/25
  	max_cache_size=100
  	
  
• FQDN-based grouping (policy ID 2001)
  Removes all OSDs from the OSD set that have been used in the file's replica locations list already and selects the subset of OSDs that is closest to the client and provides enough OSDs for the new replica in a single domain.
  This policy uses domain names of clients and OSDs to determine the distance between a client and an OSD, as well as if OSDs are in the same domain.

Sorting Policies

• Data center map-based sorting (policy ID 3000)
  Sorts the list of OSDs in ascending order of their distance to the client, according to the data center map.
• DNS based OSD Selection (policy ID 3001)
  The FQDN of the client and all OSDs is compared and the maximum match (from the end of the FQDN) is used to sort the OSDs. The policy sorts the list of OSDs in descending order by the number of characters that match. This policy can be used to automatically select OSDs which are close to the client, if the length of the match between two DNS entries also indicate a low latency between two machines.
• Shuffling (policy ID 3002)
  Shuffles the given list of OSDs.
• Vivaldi network coordinates based sorting (policy ID 3003)
  Sorts the list of OSDs in ascending order of their distance to the client, according to the vivaldi coordinates of the client and OSDs. This policy requires to enable vivaldi in the client (see section  for details).

7.3.3  OSD Selection based on Custom Attributes

  uuid = OSD1
  ...
  config.country = DE

  uuid = OSD2
  ...
  config.country = UK

  %> xtfsutil --set-pattr 1000.country --value US DE <mountpoint>
  

7.3.4  Vivaldi

1. The vivaldi replica and OSD selection policies must be set at the MRC for the volume(s) (see section 5.3.3 for details).
2. The clients must be configured to calculate their own coordinates relative to the OSDs.

 $> mount.xtreemfs --vivaldi-enable \
    remote.dir.machine/myVolume /xtreemfs

Figure 7.1: Vivaldi visualization showing two OSDs and one client.

7.4  Striping Policies

7.5  Plug-in Policies

• org.xtreemfs.common.auth.AuthenticationProvider
  interface for authentication policies
• org.xtreemfs.mrc.ac.FileAccessPolicy
  interface for file access policies
• org.xtreemfs.mrc.osdselection.OSDSelectionPolicy
  interface for OSD and replica selection policies

 public static final long POLICY_ID = 4711;

Appendix A
Support

Appendix B
Hadoop Integration

B.1  Introduction

Figure B.1: Hadoop cluster setup recommendation

B.2  Quick Start

• XtreemFS (www.XtreemFS.org)
• XtreemFSHadoopClient.jar (Download it from www.XtreemFS.org or build it by yourself by excecuting make hadoop-client)
• Hadoop (v 1.0.1 or newer) (hadoop.apache.org)
• JDK 1.6+ (Oracle/SUN)

1. Install and start XtreemFS:
   Follow the instructions given by the quick start guide for XtreemFS, available at Sec.1. Notice that the DIR is reachable at localhost:32638, beause this information will be important later.
2. Download and extract Hadoop
3. Configure Hadoop to use XtreemFS instead of HDFS:
   1. After downloading and extracting Hadoop you have to make sure it will find the XtreemFSHadoopClient package. The easiest way to do so is to download XtreemFSHadoopClient.jar into Hadoop's lib directory, or to symlink it there.
      Alternativly, you can add it to Hadoop's classpath. It can be edited in hadoop-env.sh which can be found in the conf directory of Hadoop. Assuming you put XtreemFSHadoopClient.jar to the other jar-libraries located at '/usr/share/java/', the resulting line should look like this.

      	export HADOOP_CLASSPATH="/usr/share/XtreemFSHadoopClient.jar"
      	
      
   2. Now you have to specify some properties in core-site.xml (in the conf directory of Hadoop). If this file does not exist you can safely create it.

      <configuration>
      
        <property>
          <name>fs.xtreemfs.impl</name>
          <value>org.xtreemfs.common.clients.hadoop.XtreemFSFileSystem</value>
          <description>The file system for xtreemfs: URIs.</description>
        </property>
      
        <property>
          <name>fs.default.name</name>
          <value>xtreemfs://localhost:32638</value>
          <description>Address for the DIR.</description>
        </property>
      
        <property>
          <name>xtreemfs.volumeName</name>
          <value>volumeName</value>
          <description>Name of the volume to use within XtreemFS.</description>
        </property>
        
        <property>
          <name>io.file.buffer.size</name>
          <value>131072</value>
          <description>Default buffer size when accessing files.</description>
        </property>
      
      </configuration>
      	
      

      1. The first property is required to register the XtreemFSHadoopClient at Hadoop. Now you are able to access XtreemFS by the Hadoop binary using the fs argument.
      2. The next property makes Hadoop use the DIR instead of a NameNode, therefore address and port of the DIR has to be populated. In this case the DIR is located at localhost:32638.
      3. The third property specifies the name of the volume to use within XtreemFS. Make sure, that the volume (here named volumeName) does exist. If the volume is not available Hadoop will not be able to use XtreemFS.
      4. The last property tells Hadoop the buffer size that will be used when reading/writing files. This size should be exactly the default stripesize of your volume. Note that this value is measured in bytes while the stripe size of your volume usually is measured in kilo bytes.
      Hint: If you want to provide user rights to your Hadoop installation according to the POSIX file-access-policy, you have to set the following additional properties:

        <property>
          <name>xtreemfs.client.userid</name>
          <value>hadoopUserID</value>
          <description>UserID to be used by Hadoop while accessing XtreemFS.</description>
        </property>
      
        <property>
          <name>xtreemfs.client.groupid</name>
          <value>hadoopGroupID</value>
          <description>GroupID to be used by Hadoop while accessing XtreemFS.</description>
        </property>
      	
      
4. To provide the minimum JobTracker configuration for Hadoop you also have to add the following property to the conf/mapred-site.xml:

   <configuration>
   
     <property>
       <name>mapred.job.tracker</name>
       <value>localhost:9001</value>
       <description>Listening address for the JobTracker.</description>
     </property>
   
   </configuration>
   	
   

   This specifies the address where the JobTracker will be running at.
5. Finally you are now able to start the JobTracker and the TaskTracker by running 'bin/start-mapred.sh' from within the Hadoop root-directory.

Appendix C
Client Library libxtreemfs

C.1  General Interface

• Client
  A Client object is initialized with the list of DIR replicas of a single XtreemFS installation. It provides methods to resolve an UUID and for volume maintenance e.g., creating, deleting and listing volumes. Most users will need only the OpenVolume method which returns a Volume object for a given XtreemFS volume name.
• Volume
  A Volume object provides methods for the creation, deletion and renaming of files on the particular volume. These methods require the path to the file or directory as parameter and usually result into metadata operations executed on a MRC server. In order to read and write files, the file has to be opened with the OpenFile method which returns a FileHandle object.
• FileHandle
  The FileHandle object has to be used to read or write data to/from a file. There are also methods for acquiring and releasing advisory locks. All these operations usually will be executed on the OSD which holds the replica of the file.

C.2  Using the C++ libxtreemfs

1. Install required libraries and dev(el) packages.
   In particular, you need boost and libssl-dev. On a Unix plattform, you also need libfuse-dev or you have to comment out the respective portions in the CMake configuration manually. At last, you have to install "cmake" which generates the build system.
2. Check out the XtreemFS trunk.

   svn checkout http://xtreemfs.googlecode.com/svn/trunk/ \
                xtreemfs-read-only 
   
   
3. Compile the Client.
   Currently you can only compile or all nothing, so the FUSE client mount.xtreemfs will be automatically compiled and the binary "example_libxtreemfs", too.

   cd xtreemfs-read-only
   make client_debug
   
   

   The Makefile target "client_debug" is identical to the default "client" except that it also enables debug symbols.
4. Run the example_libxtreemfs.

   cd cpp/build
   ./example_libxtreemfs
   
   
5. Use the example code as basis for own modifications.
   Now modify cpp/src/example_libxtreemfs/example_libxtreemfs.cpp to suit your needs.
   "make client_debug" was only needed the first time for letting CMake generate the Makefiles to build everything. From now on, just run "make" inside cpp/build and it will compile the changed code.

C.3  Using the libxtreemfs for Java

Appendix D
Command Line Utilities

lsfs.xtreemfs (formerly xtfs_lsvol)

Lists the volumes on an MRC.

mkfs.xtreemfs (formerly xtfs_mkvol)

Creates a new volume on an MRC.

mount.xtreemfs (formerly xtfs_mount)

The XtreemFS client which mounts an XtreemFS volume locally on a machine.

rmfs.xtreemfs (formerly xtfs_rmvol)

Deletes a volume.

umount.xtreemfs (formerly xtfs_umount)

Un-mounts a mounted XtreemFS volume.

xstartserv

Tool for manually starting/stopping XtreemFS servers, e.g. for testing and development.

xtestenv

Tool for automatic set-up of a test environment and for executing the autotests.

xtfsutil

XtreemFS's swiss army knife.

xtfs_cleanup

Deletes orphaned objects on an OSD, restores orphaned files and removes obsolete file versions.

xtfs_chstatus

Changes the status of an OSD.

xtfs_snap

Creates, lists and deletes snapshots.

xtfs_mrcdbtool

Dumps and restores an XML representation of the MRC database.

xtfs_remove_osd

Relocates all files stored on an OSD in order to remove the OSD from the system.

xtfs_scrub

Examines all files in a volume for wrong file sizes and checksums and corrects wrong file sizes in the MRC.

Index

Footnotes: