All we know about APFS, The next-generation Apple file system

The most important news announced by Apple, something that will not be immediately visible to users, but that is fundamental for the future of all its platforms: the new APFS file system. But let’s start from the beginning, highlighting some of the peculiarities of the new storage mechanism.

What is a file system?

Computers allow you to create, manipulate, store and retrieve data. The file system is basically the supporting infrastructure that makes this possible in an operating system. The file system allows you to organize, store, locate and manage information stored permanently on a storage device such as a hard drive or a USB drive.

The file system is responsible for the management of files (nothing but a logical sequence of information written on ” blocks ” ), the way they are stored, can take into account any restrictions, such as the inability to write in certain folders, show a hierarchical organization with folders and sub folders (a sort of illusion created for the user’s convenience: the file system always stores everything and anyway in the first free blocks) and other problems that those who develop applications delegate to the operating system.

There are various ways to store files; in the course of time, various ways have been studied to do this, fundamental with the evolution of the types of mass memories available, with the evolution of storage systems (we have moved from floppy disks to even tens of TB units) with, they progress (from hard drives to SSDs), with the size of user-created files grown enormously.

HFS, HFS+ and other derivatives

Hierarchical File System (HFS) is the name by which Apple calls its file system. It was introduced in 1985 as a new file system for Macintosh computers, replacing the previous Macintosh File System (MFS), used by the first Mac models, designed for a ” flat ” storage structure. It had several advantages over DOS system FAT: it allowed, for example, to use names up to 31 characters long.

It supported metadata and the separate storage of data and resources concerning the same file. Towards the end of 1997, with Mac OS 8.1, HFS+ was officially presented as a successor to the previous one. HFS+ solved fragmentation problems, unnecessarily occupied space from files as partitions increased, integrating support for file names up to 255 characters long.

Apple has continued to improve its file system; in November 2002 he added journaling improving the availability and resilience of computer errors. Journaling protects file system integrity in the event of an unexpected shutdown or power failure; it also optimizes the uptime of connected servers and storage devices, speeding up repair of affected volumes when the system reboots.

With Mac OS 10.3 came another version of HFS Plus called HFSX. The volumes in question are almost identical to HFS Plus, but they support case sensitivity for file and folder names, distinguishing file names with uppercase and uppercase (especially useful in the server environment). With Mac OS X 10.4, support for some extended attributes was integrated and support for security control based on access control lists (ACLs) was introduced.

New-Generation file system

Apple has long known that HFS+ is now a dated system. It is not ” disastrous ” as someone tries to paint it but rather, considering that its roots sink 30 years ago, it must be recognized that its designers were forward-looking, able to create a memorable storage mechanism, solid and durable.

Storage systems have changed a lot in recent years and as we have already mentioned, we have moved from floppy disks, SSD drives, RAID disks, storage systems that used magnetic plates, to units that use fast random access memories.

In 2007, Apple began to explore next-generation file system support by focusing on ZFS, a file system designed by a Sun team that has many features missing from older generation file systems. In some preliminary versions of Mac OS X 10.5, the support in reading and writing was integrated to the ZFS, but the project was definitively abandoned in 2009 (for reasons never fully clarified but probably for incompatibility with the type of license imposed by Sun for the use of his file system).

There are third-party tools that allow you to manage ZFS on the Mac, but these utilities have never been successful except for narrow circles of experts and enthusiasts.

APFS arrives

Apple has not been with hands in hand in recent years and the team that deals with the file system in Cupertino (led by Dominic Giampaolo, an engineer who created, among other things, BeFS for BeOS and integrated journaling in HFS), has developed a new file system, APFS, which is supported by macOS Sierra 10.12 but whose completion is testing and will take many months.

Apple talks about 2017 but it could be an optimistic goal: as you can imagine, a file system with potential disastrous bugs is not acceptable: you put at risk the data of millions of users and before marking it as definitive, safe and reliable, it will take months and months of testing in the most disparate conditions possible.

Why ” APFS ” and not ” AFS ” ?

The name ” APFS ” has been chosen instead of ” AFS ” (logical acronym for ” Apple File System ” ) because there already exists a file system with the latter name. AFS is the acronym of Andrew File System: the latter is developed by Carnegie Mellon University and is used in the field of distributed processing. To avoid legal issues, Apple probably preferred to call the new file storage mechanism with a non-existent acronym.

---

Peculiarities of APFS

There are several peculiarities of APFS. Below is a description of some of the most interesting features (for more technical details we refer to the technical support articles on the Apple website dedicated to developers).

Encryption

A first feature of the APFS is encryption. It is an essential requirement because of new needs created in the security field and with the spread of portable devices (notebooks, smartphones, tablets). From a technical point of view, APFS supports both encrypted and unencrypted units, single key encryption (symmetric encryption) for medatates and user data, multiple key encryption with different types of metadata protection, file and file sections (variable or ” extents ” ).

Snapshot

Snapshot are one of the most interesting features. In short, it is possible to ” photograph ” the overall status of a unit at a given time, an option to restore the system in case of problems. For example: it will be possible to install an update of the operating system and, in case of problems, go back exactly to the same situation from which we left before the update. The system is efficient from the data-management point of view, taking into account the actual changes made.

The mechanism is complementary to the functionality provided by Time Machine (which currently does not support writing to APFS initialized media). In future versions of OS X, there will be two functions for data recovery: with Time Machine, it will be possible to recover deleted files and folders over time; with the snapshots, it will be possible to restore the entire state of the system saved periodically (according to the settings chosen by the user).

Shared Space

APFS supports the shared space of units and volumes, consisting of units, or volumes, joined together. Space sharing allows multiple file systems to share free space on a physical volume. Compared to the traditional and rigid partition mechanism, with which it is necessary to allocate a certain fixed space for the filesystem, APFS volumes can grow or decrease (be resized) without repartitioning the volumes.

Each volume in an APFS ” container “, reports the same free space available, corresponding to the owner of the available space in the container. For example: an APFS container with a capacity of 100GB that includes the volume ” A ” in which 10GB is exploited and the volume ” B ” that uses 20GB, the total free space is indicated in 70GB (100GB – 10GB – 20GB).

Efficiency in managing the allocated space

APFS takes into account identical files, avoiding unnecessary duplication. If there are folders with multiple identical files, the system will store only one type of file for everyone; if a single file is modified with respect to the others until then identical, it will allocate new space on the fly with the ” copy in write “, an optimization technique that aims to reduce duplication of system resources.

The main advantage of the copy-on-write technique is that if a duplicate resource is subsequently freed or deleted without meanwhile neither the original nor the copy ever been modified, the system has effectively avoided unnecessary duplication, resulting in time savings. The copy-on-write is all the more advantageous the more expensive the duplication operation is and the more infrequent the write operations.

Data integrity

The main purpose of a file system is to ensure data integrity. During file read and write operations, crashes, blocks, and unexpected machine shutdown may occur. In traditional file systems, a system startup flag lets you know if a computer or device has been properly closed or not, and to invoke any verification and control utilities. A specific copy-on-write metadata management scheme improves file system protection against crashes, while reducing the tasks to which journaling is invoked in case of problems.

Performance

Apple claims that the new file system is particularly optimized for Flash / SSD storage drives. For this purpose, it has foreseen a specific layer called Flash Translation Layer (FTL). Increased speed in reading and writing files should be guaranteed by specific features designed to reduce I / O latencies.

The system always tends to give priority to the operations performed by the user; in theory – we should see much less the colored wheel that appears when the system is particularly busy in read / write operations with the disk. Also taking into account how much space a folder set occupies will be easier: Fast Directory Sizing allows you to quickly calculate the space occupied by folders and subfolders.

Volume Format Comparison

---

Criticism

All good for APFS? Yes, but the notes are not lacking. Among the criticisms of the new file system is the absence of a checksum system for user data. A check summing mechanism is provided but not for user generated files. This implementation, according to some experts, although it would imply for each file, more space occupied, would allow to implement a sequence of bits that, associated with the transmitted package, could be used to verify the integrity of any file.

Another criticism of the APFS is that it continues to be based on the concept of logical blocks (sectors), a mechanism which, according to some, no longer makes sense with the advent of new-generation memories such as Intel’s Xpoint 3D. Hard disks and SSD drives store files in variable 512 byte logical blocks (in older systems) to 4K sectors (4,096 bytes).

As disk drives increased, the size of the sectors proved to be a drag on the design of large-capacity disk drives and improving the efficiency of error correction. In any case, being a file system under development, the arrival of new functionalities in the months and years to come cannot be excluded. As we have seen, previous limitations of HFS have been resolved in subsequent implementations.

Conclusions

There is no perfect file system and above all, it is not possible to create it in a short time. Apple, although it has only officially announced APFS now, has been working on the system for at least a couple of years. Cupertino has laid the foundations for a file system that will probably resist well at least another 30 years. Many tests will be needed (Apple hypothesizes the availability in 2017, but we would not be surprised if it decided to move the definitive specifications even further in time).

When the new filesystem is available it can be tested for performance, reliability, security, etc. It is not clear if the file system will be available as an open-source implementation; in the developer documentation, it is indicated that – at the moment – this implementation is not available; it is not to be excluded that Apple intends to do something in this sense. Open source availability could mean adoption or support even on non-Apple operating systems, accelerating and favoring mass adoption.