I skimmed through the article, but I have a question that I hope someone can answer. I have a sparse disk image created on a NAS (which runs Linux), and I use it to backup some stuff (not a VM) from the Mac in the native format (the macOS APFS file system).
Would this new format, ASIF, make this faster and better whenever I switch to macOS Tahoe? I hope there wouldn’t be any gotchas with respect to storing this disk image on a NAS.
Yeah I’m interested in this as well. There’s something about the way time machine works over SMB that is absolutely unfashionably dog-slow. I suspect SMB performance on mac is just not very good in general tbf
(Yes, you can store a filesystem in a file - and that's a trivial sort of disk image, but one with some serious drawbacks like "you have to allocate all of the space up front". We can do better.)
Some of the most popular disk image formats are basically a sparse file abstraction for non-sparse files and nothing more. You have a bunch of blocks, a table mapping each block to its virtual location, and a couple convenience headers.
If those count as a disk image when you put a filesystem inside, then I say a normal file is also a disk image when you put a filesystem inside.
Especially because the sparse mapping is optional. For example, lots of VHDs are a raw file plus a 512 byte footer.
They also announced new APIs for native Linux containerization on MacOS with a specific focus on security and performance. This seems like it may be in support of that as well. Anything they can do to improve the performance of containers is a huge win. https://youtu.be/JvQtvbhtXmo?si=3OphClGvylHggmSW
What blew my mind recently was that I could store an APFS sparsebundle on a NAS drive, then mount it over NFS and use it as a plain old APFS volume. Despite the filesystem layering, it works pretty much like a local volume, albeit with a bit of performance degradation. Seems preferable to something like iSCSI for using APFS with network storage.
Not sure why that would blow your mind, but Apple's old Time Capsule basically mounted a sparsebundle over the network too. And yes I would also guess this new format would work better.
My experience with NFS file management has been less than stellar, so running a full, virtualized, and performant APFS volume on top of it feels like a bit of a magic trick.
Interesting. My employer uses NFS extensively and issues are very rare, and such issues can always be traced to underlying network issues not NFS itself. It's a good choice if you have a good network.
I have a meta question not directly related to the article but more about HN itself. I posted this exact link 9h before this submission was posted[1]. How is it possible that there is a new entry for the submission given the link is the same?
In practice, if it gets any real amount of votes or comments, you have to wait a year to repost. If it doesn't get any attention, it can be reposted quickly (though I think it should be a day later).
It depends. I tried posting a few links I came across over the past few days, and it just showed me dups but they were from days or weeks or sometimes months ago.
Not sure why this comment is downvoted. Yes, I got redirected for a dupe earlier today for something posted 7 months ago. And even though the content substantially changed, it can’t be posted again.
My guess is that “fastest gun in the west” might be a bit anti-pattern with respect to community.
Because in ye olden times, mild URL shenanigans seemed to have been a common hack to bypass more strict dupe detection.
And the community probably doesn’t really benefit from aggressive karma seeking —back then being first would give you a point for every resubmission. [1]
But that’s all speculation based on a supposition that what is more likely to be submitted by other users is not a better criterion for choosing to submit.
But I could very well be wrong and probably am.
[1] and number of submissions is probably at best a noisy signal for front page placement and might be negatively correlated with curiosity…I mean even here, what Apple is doing doesn’t stray too far from yesterday’s big press release by one of the most valuable corporations in the world.
Might be something weird with this domain. Look at the list of submissions, yours isn't the first dupe that was accepted in a relatively short time window.
3rd party supporting a file system would be one of the last things on a list of all software I’d ever want a 3rd party writing instead of the OS maker.
Nightmare to evaluate the options, pure stress testing the options, difficult to know if it didn’t mess something up.
>3rd party supporting a file system would be one of the last things on a list of all software I’d ever want a 3rd party writing instead of the OS maker.
Given how many people use FUSE, Paragon NTFS for Mac, and similar tools, you're hardly totally representative.
Third party read-write NTFS drivers took FOREVER to become really robust. I remember hearing horror stories not infrequently up until maybe a decade or less ago.
MacOS users' awareness of a mostly linux centric piece of tech is pretty damn irrelevant here. The point is that FUSE is a pretty mature piece of technology, and we know that it can be used productively without being that nightmare scenario you described. There is no reason why Apple's FSKit can't be equally successful.
How many people use software like this because they have no choice? I used Paragon NTFS, but the entire time, I thought it was ridiculous that MacOS can't read NTFS on its own.
Like 99% of the computer using world until less than a decade ago, when almost all drivers were kernel extensions and things like kexts were very much used?
That being said: FSKit is a userspace API. In that respect, it's a lot better than filesystem code running in the kernel - it can't crash your computer or corrupt data on other filesystems, and it's much more tightly sandboxed if it gets exploited.
Exactly! Third party file systems support in user space is exactly what I want to see. It seems to me that third party kernel code has always caused me problems. By moving the FSKit to user-space, I’m quite happy to try something, knowing that it won’t affect the rest of my system.
What would Apple's incentive be to support Btrfs, Ext4, XFS or ZFS ?
Btrfs, Ext4 and XFS are all under GPLv3, which may or may not be a problem for Apple, but "just in case".
They tried with ZFS, but couldn't strike a deal with Sun/Oracle, so instead invented APFS.
Apple already delivers a stable filesystem. It may not be "best of breed", but it works, as billions of devices runs on it every day with zero problems.
I'd be happy for VeraCrypt not to have to rely on MacFuse which requires I go turn off some very low-level protection to even use. It sounds like this makes that possible.
I don't really understand your objection to be honest. Drivers for storage are common on other platforms
This article describes a new disk image format (on which a filesystem can be put, APFS in the article), not a filesystem, or did I misunderstand?
edit: added the word "image", which I apparently forgot to type. Mentioning the edition because otherwise it would make an answer to this comment difficult to understand.
Only Mac users IMHO are well-familiar with working with disk images. They are not as diverse or well-supported on other OS’es, while nearly every Mac app (prior to the App Store) was installed by dragging it out of a mounted disk image.
The post only benchmarks against UDIF. Whether this brings something new is a very good question that is not answered by the post.
How complex is the format? Did they do anything clever? Did they engage in NIH? For all I know they switched the AES mode and made the blocks bigger and that's the entirety of the performance boost.
That's of course also a reasonable question to ask! We all hope by asking these questions some Apple employee using a throwaway account will provide an answer on HN. HN is that magical place where such questions have the best chance of getting answered.
I don't think the question would make much sense. Why did Apple do X and not Y (unrelated to X) is not an interesting question. It would seem close to whataboutism. They were willing to spend money on X. They are not willing to spend money on Y. Or, they didn't think about doing Y. Or, they needed X. They haven't needed Y. Apple want fast VMs and also they haven't needed Btrfs. What insight can you get? There's no relation between the two. You'd get the same insight by asking "Why didn't Apple implement Btrfs?", but by linking the two in the same question, you are kinda implying there's a link.
But the question would have been highly relevant had Apple developed a new FS, and disk images and FS do seem related at first. I didn't want to assume whataboutism, so I figured OP was possibly confused because this is likely, and I wanted to give them a hint without bluntly asking whether they confused things. I could have, really there's no harm in being confused, nor in asking whether someone was confused.
Does anybody else think that it would make sense for Apple and Microsoft to just get in a room and horse trade a few things like this, if they cared about user experience? Cross-license both APFS and NTFS, and share any internal documentation under NDA so that external drives can use modern formats with safety features like journaling without locking users in.
I suspect there wouldn't be an agreement on a minimum set of features for a modern filesystem, even just for external disks, even if you limited it to flash storage devices to avoid all the complexities of spinning platter latency.
After all, there's no such agreement on Linux either - we just have all the Linux vendor options available.
Let me clarify, I don't expect two vendors like that to merge filesystem specs. I just think they should have first-class support for reading and writing on each others' default (NTFS and APFS) filesystems because the alternative is that a user who has a hard drive full of their important documents can't switch between Mac and PC without buying 100% more storage with which to do a complicated filesystem-migration exercise -- and let's be honest, only an absolute nerd (like us) would even understand what that is. Others would just plug in the NTFS disk to the Mac, see the popup saying "unreadable," shrug and give up (or worst case, format it not understanding that means erasing).
This is the kind of thing a reasonable government would just tell them they had to do by virtue of being a fully locked-in duopoly, just like they should tell Apple and Google that users should be able to choose to install an alternat app store.
This is a disk image format so why not VHD? It seems like it's open enough and supports what a virtual disk needs what do we gain with yet another file that's a disk type?
ext4 and Btrfs are only well supported on Linux; they are not universal standards.
NTFS was only supported well on Windows until recently; but extensions like NTFS Encryption (BitLocker) are still Windows only. Mac still does not let you write to an NTFS volume.
APFS and HFS+ are obviously Apple file systems.
FreeBSD does not support ext4 or Btrfs well; but instead prefers UFS2 or ZFS despite also being an open-source Unix-inspired OS.
The world runs on proprietary or non-universal file systems with CDFS (ISO 9660), FAT, and exFAT being the sole exceptions.
Is there a single filesystem in the world (besides "simple" ones like FAT) that both has an open standard AND is licensed in a "usable" codebase (MIT or other "non-copyleft" license)?
AFAIK that's an incorrect meme that just won't die. The performance issues you're thinking of have nothing to do with the filesystem itself, but with the I/O subsystem in Windows more generally. If you have evidence otherwise please share.
I'm on my phone and this is a longer discussion than I can have here, but the performance problems they're thinking of and the ones people usually rant about on these forums are not the same ones (or same magnitude). Before being so confident it's NTFS itself that's the issue, try ReFS (and FAT32 for that matter) and tell me if you see the performance problems you've encountered actually improve a lot. And then narrow down the cause. You might be surprised how much of it had to do e.g. with filter drivers or other things. And don't forget you're still testing one particular implementation on one OS, which doesn't say anything about a different one for the same filesystem on a different OS.
There are tools like voidtools Everything2 and WizTree that directly read NTFS from disk device bypassing windows FS apis and are blazingly fast (faster than find/du on ext4 in Linux).
correct. but it's also a management system so perhaps they only want a files system? no idea why more don't use zfs. especially after the auto expansion update earlier in the year.
That's a major benefit of ZFS for sure, but I think being copy on write is another major benefit for single disk systems. ZFS is the only one I know of that has a full feature set and is supported on every major OS.
if GPL is a non-starter for you, youre missing the point of the open standard. apple already discloses a litany of various GPL it ships. XFS would be no different.
I intentionally excluded unofficial or third party software, as almost anything is supported if you’re brave enough. The quality of said drivers also wildly varies.
Until 4 years ago, nothing was good enough for the upstream Linux kernel.
I mean, yeah, you could say that. Something being in the kernel is a good benchmark of quality. But IMO open-source is different. For instance, Terraform had no stable release from 2014 till 2021, that didn't stop enterprises from using it on scale.
I just ran into a use case yesterday. I wanted to copy some files from either my Mac or my Windows machine onto the MicroSD card for my SteamDeck, which is ext4.
I wanted to just plug in the card and copy files, but couldn’t.
Even FreeBSD can't be bothered to do a good job supporting Linux file systems. They're basically proprietary siloed file systems like the rest, even if code is available. Linux, meanwhile, can't be bothered to support either of FreeBSD's file systems officially. UFS2 is hardly patented anymore, but Linux doesn't care beyond read-only support.
Being open-source, and even being in a popular open-source project, does not make a standard, or even imply inferiority to those who do not implement it.
There might be some license issues, but the dirty secret is filesystem portability isn't terribly important, and for the users for whom it is - ExFAT and friends are usually good enough.
exFAT is widely used but it not being journaled has led to so many thousands (if not more) people losing tons of data, many of which wouldn't have lost so much data had they used a journaled filesystem (or even one with redundant file tables.)
If you need to connect a portable drive to machines of different OS's, there is no safe filesystem that supports read and write on both Windows and MacOS.
Alternatively, cloud storage works until the files are larger than the space you have left on Drive/Dropbox/OneDrive/etc., and local network sharing (on certain networks at least) is more complicated than what the average user is willing to put up with. In practice, many use USB flash drives or external HDDs/SSDs with exFAT. Yeah, people should have more than one backup, but we know in the real world that's not what many do. That requires them spending more time (e.g. configuring local network sharing or setting up an old machine lying around to be a NAS) or money (more on cloud storage.) In practice, having a cross-platform, journaled filesystem would lead to a lot less data loss.
Aside from exFAT, the only alternative with native cross-platform R/W capability is FAT32, but while it has a redundant file allocation table (unlike exFAT), it has a max file size of 4GB, which limits its usefulness for many workflows.
A. You're flagged for graphic, inappropriate, and triggering language.
B. Oracle sued Google for a decade over whether a minor number of APIs were copyrighted in Java; do you think Apple's eager to embrace their technology regardless of license? Heck, even Ubuntu wasn't willing to make that bet.
C. Guess how much official licenses for those fancy file systems cost.
D. ZFS, and other fancy file systems, are not known for low RAM and CPU usage. In a server, this does not matter; but it's pretty important for anything on a battery.
I think this is actually a human struggle - I add Conclusions to my blog posts / dev guides because it feels awkward to just end with the last step.
But it also always feels like I am just awkwardly restating things I just said. It's not a thesis, an article generally not complicated enough to need a re-synthesizing of main points at the end.
I have an unusual way of communicating sometimes that used to be quirky and fun. Now I simply get accused of being a bot. Before LLMs I never got accused of being a robot.
I think the idea of good writing is soon going to morph with writing that doesn’t feel like it was made with AI.
It drives me crazy that em-dashes are supposedly a smell for AI now. The em-dash is a great and underused bit of punctuation.
I am just glad I am not in college dealing with all the AI and faulty AI detection crap happening. Ridiculous false accusations of plagiarism were already awful enough before this.
> It drives me crazy that em-dashes are supposedly a smell for AI now.
They're an excellent tell for comments written on places like HN, Reddit, and other forms of social media. The average person (myself included) simply does not write very well and I would wager the majority of English speakers are not able to properly use an em-dash.
I expect them in books and suspect that's where the AI models are getting them from, but those are being written largely by professionals.
It's at a point where the moment I see an em-dash on a social media post I cease reading it and move on to the next.
Same here. It's almost as if LLMs emit em-dashes because the content they were trained on contains them. Wait until skeptics realize how often LLMs use the word "the".
If you're going to be this superstitious, then you might as well lean into it and have a little fun. Better throw some herbs into the air and do a little jig for good measure. Personally, I'm going to accept that I won't be able to tell, because models are trained and prompted to appear human.
It's a solved problem in the sense that Apple solved it more than a decade ago. (The sparsebundle format was introduced in Mac OS X Leopard from 2007. The sparseimage format was even older; I saw discussions of using it with Panther.) Apple solved it again but made it faster.
It's a shame that every new cool product/dataformat/cable/cpu/whatever researched by Apple has very little (or no) public documentation. Sure, there are lots of hackers who can test and reverse engineer those pretty quickly, but it's just unnecessary work. I don't know why Apple is so revered in hacker circles, to be honest. Not even Microsoft does this shit anymore, they're open sourcing a lot of research this decade, but they're still seen with extreme distrust. Whereas Apple was always secretive and used underhanded tactics, but it is still loved.
You're joking, right? You can see at a glance that this mess is only published for compliance reasons. There is no documentation at all, and most of the code consists of numerous versions of open-source libraries that have been subtly modified due to their licensing requirements, which necessitate disclosure of modifications. Good luck building any flavor of that! See: https://news.ycombinator.com/item?id=35197308
I use sparse disk images extensively as a tool to make certain directories require specific passwords. I simply create an encrypted sparse disk image and set a password on them. Seems like this will speed up my use case.