Sep 212011

Ever wondered all the precious memory installed on your server has gone? Among many other reasons, if you are running Solaris 10 and use ZFS file system then there may be your answer.

ZFS Adaptive Replacement Cache (ARC) tends to use up to 75% of the installed physical memory on servers with 4GB or less and upto everything except 1GB of memory on servers with more than 4GB of memory to cache data in a bid to improve performance.

This can significantly affect performance on mission critical servers running Databases etc.

To identify how much memory uses:

# kstat -m zfs | grep size
        data_size                       18935877120
        hdr_size                        66041496
        l2_hdr_size                     0
        l2_size                         0
        other_size                      11310112
        size                            19013228728

Here “19013228728” (approx 18G) indicates the total memory used by ZFS.

Alternatively, the following mdb command show ZFS ARC usage:

# echo "::arc" | mdb -k|grep size
size                      =      2048 MB
hdr_size                  =  12493584
data_size                 = 2048608256
other_size                =  86475456
l2_size                   =         0
l2_hdr_size               =         0

It makes sense to cap the maximum ZFS ARC can use on servers where memory requirement for other services is more.

To set the maximum limit for ZFS ARC, edit /etc/system file and add the following line

set zfs:zfs_arc_max=2147483648

where 2147483648 restricts the usage to a maximum of 2GB physical memory. Unfortunately, this requires a reboot for the setting to take effect and cannot be dynamically changed.

  One Response to “Missing memory in Solaris 10 with ZFS”

  1. Below is the Server specifications:
    SPARC Enterprise M4000 Server
    Memory size: 65536 Megabytes

    Current status:

    root@p # kstat -m zfs | grep size
    buf_size 142598208
    data_size 15926845248
    l2_hdr_size 0
    other_size 1064798792
    size 17134242248
    root@p # echo “::arc” | mdb -k|grep size
    size = 16340 MB
    buf_size = 135 MB
    data_size = 15189 MB
    other_size = 1015 MB
    l2_hdr_size = 0 MB
    root@p # /usr/local/bin/top |grep Memory
    Memory: 64G phys mem, 3603M free mem, 65G total swap, 65G free swap
    root@p #

    root@p # cat /etc/system |grep zfs
    set zfs:zfs_arc_max=17179869184
    root@p #

    Kindly advise if zfs:zfs_arc_max is ok for the current server usage and specifications ?

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>