Missing memory in Solaris 10 with ZFS

ZFS Adaptive Replacement Cache (ARC) tends to use up to 75% of the installed physical memory on servers with 4GB or less and upto everything except 1GB of memory on servers with more than 4GB of memory to cache data in a bid to improve performance.

This can significantly affect performance on mission critical servers running Databases etc.

To identify how much memory uses:

# kstat -m zfs | grep size
 
        data_size                       18935877120
        hdr_size                        66041496
        l2_hdr_size                     0
        l2_size                         0
        other_size                      11310112
        size                            19013228728

Here “19013228728” (approx 18G) indicates the total memory used by ZFS.

Alternatively, the following mdb command show ZFS ARC usage:

# echo "::arc" | mdb -k|grep size
size                      =      2048 MB
hdr_size                  =  12493584
data_size                 = 2048608256
other_size                =  86475456
l2_size                   =         0
l2_hdr_size               =         0

It makes sense to cap the maximum ZFS ARC can use on servers where memory requirement for other services is more memory.

To set the maximum limit for ZFS ARC, edit /etc/system file and add the following line

set zfs:zfs_arc_max=2147483648

where 2147483648 restricts the usage to a maximum of 2GB physical memory. Unfortunately, this requires a reboot for the setting to take effect and cannot be dynamically changed.