Hoopoe storage services
Files play a big role in Hoopoe, as they contain data to be processed by tasks
users submit.
For this purpose, Hoopoe supports several storage mechanism providing a high
level of flexbility.
HoopoeFS (File-System)
HoopoeFS is a native service in Hoopoe, providing a simple storage service. The
API supports basic file-system objects (files, directories etc.) and was created
with simplicity in mind so users can take care of the computing.
For example, working with files and directories support the very same
operations:
- Create
- Delete
- Rename
- Get information
Where files also support I/O operations:
HoopoeFS has no limitations for the size of files a user can upload. All
functions in the API are meant for 64 bit support to work with very large files.
An advantage of using HoopoeFS is that the storage is placed internally in
the cluster, for immediate access by Hoopoe without requiring further
transfers.
Amazon S3
Amazon S3 is a
storage service provided by Amazon Web Services and is supported in Hoopoe
for convenience of users.
Following S3 model, when submitting a task, one provides further
information about the origin of the data (bucket, object), to be
accessible by Hoopoe.
When a computation finishes, the resulting data is also uploaded to the
object specified by the user (following the bucket-object principle).