Reading the disk

Reading the disk

The first problem is speed. There are nearly six million files on my machine, and a tool that takes twenty minutes to tell you anything is a tool you never open.

The usual way to walk a directory is to list it and then call stat on every child, which is one trip into the kernel per file. macOS has a better call for this, getattrlistbulk, which fills a buffer with many entries at once and returns the name, type, size, modification date, link count and inode for all of them together. Reclaim reads into a 256KB buffer, so a folder with a thousand files in it costs a handful of syscalls rather than a thousand.

On top of that it runs one worker per core, minus one, over a shared queue of directories still to visit. Workers take a directory, read it, and push whatever subdirectories they found back onto the queue. The count of active workers is tracked separately from the queue length, because an empty queue with someone still working means wait, while an empty queue with nobody working means the disk is done.

Sizes are added up afterwards, in a single pass, once all the workers have stopped. Totalling as you go would mean every worker fighting for a lock on the same handful of top-level folders.

Together that reads about five million files in roughly thirty seconds.

Before a scan, and during one. The list reorders live as folders are totalled, so the big ones surface within a second or two.