Fujifilm archiving software combines S3 with record write speed and maximum security.
The digital data explosion, increasing regulations for the long-term retention of data and the risk of cyber-attacks: How does Fujifilm's Software-defined Tape meet archiving requirements whilst adapting to the specific features of object storage?
Fujifilm has launched an archiving software specially designed for writing objects on tape: Software-defined Tape. It operates as an S3 gateway between an object storage solution and a tape system. With its record writing speed and high flexibility, the software is compatible with a wide range of capacities. It can carry up to several hundred PBs and achieves an operational transfer rate of 205MB/s for 10MB files, which is 10 times faster than regular tape systems and 20 times faster than hard drives. This software allows the user to install their own private Glacier-type cloud on-premise, while strengthening the level of data security and optimising storage costs.
The Holy Grail of Data Storage: Software-defined Tape maintains a record write speed
Fujifilm continuously explores new fields to make tape more efficient and user friendly. Whilst addressing the question of object storage, they solved the fundamental question facing all storage technologies: write speed. When backing up a large number of small files, the write speed can be affected and system downtime can also occur. This is exacerbated as the file sizes decrease. The linear structure of object storage makes this even more significant because each object is saved individually. As a result, the deterioration of speed can be more severe. This is contrary to a traditional file system, which by its structure allows files to be grouped within archival folders or in tar-type containers, for example.
Tape drives have special tools designed to mitigate this phenomenon such as the buffer, which is effective for files over 500MB and IBM's Skip Sync system for files between 200MB and 500MB. Nevertheless, writing a large number of very small files will slow down backup operations. This can result in a drop in operating speed below 10-15MB/s on an LTO drive and even below 5MB/s on disk.
In order to solve the problem of write speed, the principle of Software-defined Tape is to prevent the accumulation of write stops between small files. The software groups the objects into packs before storing them on tape. It generates a "single 1GB to 10GB file" which is written to tape in one stroke. The OTFormat, the new tape writing technology developed by Fujifilm, makes it possible to write these packed objects to tape and significantly reduces the number of write stops. Software-defined Tape succeeds in maintaining a spectacular 205MB/s transfer rate for the back-up of 10MB files.
Fujifilm chose the popular and easy-to-use S3 interface
In order to facilitate the communication between tape systems and the largest number of object storage solutions, Software-defined Tape uses the native S3 and S3 Glacier protocols. The S3 protocol, already widely adopted by object storage solutions, has become the standard for the transmission of objects. It is a free and open data transfer protocol. Furthermore, the publication of its APIs makes it easy to program. In this way, the popularity of Amazon's cloud services and the availability of the S3 APIs motivate the software manufacturers and the users to implement them within their solutions and infrastructures. The S3 Glacier protocol allows the users to seamlessly restore their objects from tape to their object based disk storage.
Increased security for data and metadata
Losing metadata can seriously affect access to data. As a result, Fujifilm has chosen to focus on metadata protection alongside all the specific features of object storage.
Metadata is used as search criteria to find the objects that need to be restored. Without metadata, it is difficult and sometimes even impossible to know which object corresponds to which data. In such cases, accessing data can be very problematic.
In addition to the copy of metadata kept on the server, Software-defined Tape systematically and automatically commands the writing of four copies of metadata on the same tape as the data. The metadata benefits, as does the data, from all the advantages of tape in aspects of security: protection against viruses and hackers, a data integrity 10,000 times higher than that of hard disk, an archive life of over 30 years and seven times longer than on hard disk, which is weakened by its many mechanical parts. These are just a few of the advantages of tape over other storage technologies.
For the users' peace of mind, Software-defined Tape adapts to all capacities up to several hundred Petabytes The Software-defined Tape system scalability and the software subscription scheme flexibility contribute to Fujifilm's quest for simplicity. It supports up to 128 billion objects, 64 servers and 256 tape drives. Software-defined Tape is compatible with LTO7 and LTO8 drives. In early 2021 it will be compatible with the IBM TS1155 and TS1160 3592 drives and with LTO9 later in the same year.
When it comes to hardware brands, Fujifilm software is agnostic and works with most of the libraries available on the market, from the 280 slot IBM TS4300 library to several IBM TS4500 libraries each operating up to 23,700 LTO slots. This also gives users the freedom to choose their servers provided they match the necessary performance requirements.
The software subscription scheme is based on both the archiving capacity and the length of the contract. Its flexibility enables the user to cover any unexpected data growth. The software license includes the number of tapes required to write from one to three copies according to the user’s own choice. Software-defined tape automatically and simultaneously, performs these copies. For a 10PB subscription over a 5-year period, two sets of data are included (one copy), as standard and 1,700 LTO8 tapes, that is to say 20PB of tapes, will be provided with the software.
Furthermore, for the user’s convenience, Fujifilm has added features which enable them to manage their archives directly from the object storage application GUI. This ensures the operational continuity through a path-failover system.
The philosophy of this archiving software can be summarised by Fujifilm's strong desire to make life easier for the users. Software-defined Tape meets both the specific challenges raised by object storage and the companies’ high requirements in the field of long-term data retention, especially in matters of data security and cost optimisation. Indeed, the total cost of using tape is, on average, 5 times lower than hard disk, in particular due to its low ecological footprint. To go a step further we can say that Fujifilm leveraged its first leap into archiving software in order to revolutionise the issue of write speed.