ZFS Adaptive Replacement Cache (ARC) tends to use up to 75% of the installed physical memory on servers with 4GB or less and upto everything except 1GB of memory on servers with more than 4GB of memory to cache data in a bid to improve performance.

This can significantly affect performance on mission critical servers running Databases etc.

To identify how much memory uses:

# kstat -m zfs | grep size
 
        data_size                       18935877120
        hdr_size                        66041496
        l2_hdr_size                     0
        l2_size                         0
        other_size                      11310112
        size                            19013228728

Here “19013228728” (approx 18G) indicates the total memory used by ZFS.

Alternatively, the following mdb command show ZFS ARC usage:

# echo "::arc" | mdb -k|grep size
size                      =      2048 MB
hdr_size                  =  12493584
data_size                 = 2048608256
other_size                =  86475456
l2_size                   =         0
l2_hdr_size               =         0

It makes sense to cap the maximum ZFS ARC can use on servers where memory requirement for other services is more memory.

To set the maximum limit for ZFS ARC, edit /etc/system file and add the following line

set zfs:zfs_arc_max=2147483648

where 2147483648 restricts the usage to a maximum of 2GB physical memory. Unfortunately, this requires a reboot for the setting to take effect and cannot be dynamically changed.

1. Using fcinfo

In Solaris 10, you can use the “fcinfo” utility which is a 2 step process to identify the Luns presented to the server.

1. Find the HBA port WWN number on the server:

# fcinfo hba-port

This command will show the HBA Port WWN among other information like the Node WWN, Device name, manufacturer, firmware, driver details etc.

root@myserver:/root # fcinfo hba-port
HBA Port WWN: 21000022ff0553e9
OS Device Name: /dev/cfg/c5
Manufacturer: QLogic Corp.
Model: 371-4324-02
Firmware Version: 05.03.02
FCode/BIOS Version: BIOS: 2.02; fcode: 2.03; EFI: 2.01;
Serial Number: 0402R00-1102885753
Driver Name: qlc
Driver Version: 3.00p
Type: L-port
State: online
Supported Speeds: 2Gb 4Gb 8Gb
Current Speed: 4Gb
Node WWN: 20000022ff0553e9

Here 21000022ff0553e9 is the WWN number for the HBA port.

2. List SCSI targets for the remote port attached to the HBA port attached to the above.

# fcinfo -sl -p HBA-Port-WWN

root@myserver:/root # fcinfo remote-port -sl -p 21000022ff0553e9
Remote Port WWN: 202400a0b8744072
Active FC4 Types:
SCSI Target: yes
Node WWN: 200400a0b8744072
Link Error Statistics:
Link Failure Count: 1
Loss of Sync Count: 2
Loss of Signal Count: 72
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
LUN: 0
Vendor: SUN
Product: LCSM100_F
OS Device Name: /dev/rdsk/c5t202400A0B8744072d0s2
LUN: 2 Vendor: SUN
Product: LCSM100_F
OS Device Name: /dev/rdsk/c5t202400A0B8744072d2s2
LUN: 31 Vendor: SUN
Product: Universal Xport
OS Device Name: /dev/rdsk/c5t202400A0B8744072d31s2

Here, LUN0 and LUN2 are the actual disks and LUN31 is the Command device for the Storage controller

2. Using cfgadm

You can also find the above information using the “cfgadm” utility. One of the following commands should reveal the LUNS attached to the controller. Although the later works only on Solaris 10.

# cfgadm -al -o show_FCP_dev c5
 
or
 
# cfgadm -al -o show_SCSI_LUNS c5

where c5 is the controller. If you do not mentioned the controller, it then displays for every single disk controller on the server.

root@myserver:/root # cfgadm -al -o show_FCP_dev c5
Ap_Id Type Receptacle Occupant Condition
c5 fc-private connected configured unknown
c5::202400a0b8744072,0 disk connected configured unknown
c5::202400a0b8744072,2 disk connected configured unknown
c5::202400a0b8744072,31 disk connected configured unknown

root@myserver:/root # cfgadm -al -o show_SCSI_LUN c5
Ap_Id Type Receptacle Occupant Condition
c5 fc-private connected configured unknown
c5::202400a0b8744072,0 disk connected configured unknown
c5::202400a0b8744072,2 disk connected configured unknown
c5::202400a0b8744072,31 disk connected configured unknown

Share your thoughts, if you know more ways of doing this.

Attach a Block Device

sunsolaris# lofiadm -a /export/software/iso_image.iso

/dev/lofi/1

Mount the ISO Image block device

sunsolaris# mount -F hsfs -o ro /dev/lofi/1 /mnt

Mount the mounted ISO image to the Guest zone

sunsolaris# mount -F lofs /mnt /zones/myzone/root/mnt

Where /zones/myzone/root is the root of the Zone. The files should now be visible under /mnt on the guest zone.

Let’s put it to work. tcpdrop solaris package is available for download from www.sunfreeware.com. Click one of the following links for Solaris 10 for x86 or SPARC.

tcpdrop (Solaris 10 – SPARC)

tcpdrop (Solaris 10 – x86)

Once downloaded unzip and install tcpdrop as follows:

# gunzip tcpdrop_solaris-20080516-sol10-x86-local.gz

# pkgadd -d tcpdrop_solaris-20080516-sol10-x86-local

The following packages are available:

1  SMCtcpdr     tcpdrop_solaris

(x86) 20080516

Select package(s) you wish to process (or ‘all’ to process

all packages). (default: all) [?,??,q]:

Processing package instance <SMCtcpdr> from </root/tcpdrop_solaris-20080516-sol10-x86-local>

tcpdrop_solaris(x86) 20080516

Ceri Davies

Using </usr/local> as the package base directory.

## Processing package information.

## Processing system information.

2 package pathnames are already properly installed.

## Verifying disk space requirements.

## Checking for conflicts with packages already installed.

## Checking for setuid/setgid programs.

Installing tcpdrop_solaris as <SMCtcpdr>

## Installing part 1 of 1.

/usr/local/bin/tcpdrop

/usr/local/doc/tcpdrop/OPENSOLARIS.LICENSE

/usr/local/doc/tcpdrop/README

/usr/local/man/man8/tcpdrop.8

[ verifying class <none> ]

Installation of <SMCtcpdr> was successful.

I’m testing on Solaris 10 9/10 (x86).

# cat /etc/release
Oracle Solaris 10 9/10 s10x_u9wos_14a X86
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010

I’ve established two ssh connections to the server.

# netstat -an|grep EST
192.168.1.66.22 192.168.1.65.49980 15872 51 49640 0 ESTABLISHED
192.168.1.66.22 192.168.1.65.50050 16640 0 49640 0 ESTABLISHED

Add /usr/local/bin to the PATH environment variable if it isn’t already there.

To drop a connection, the syntax for tcpdrop is

# tcpdrop <server address> <server listening port> <client address> <client source port>

In our case, to disconnect the 2nd connection,

# tcpdrop 192.168.1.66 22 192.168.1.65 50050
192.168.1.66:22 <-> 192.168.1.65:50050: dropped

where

192.168.1.66 is the local server IP

22 is the TCP port SSH server that we are listening for a connection

192.168.1.65 is the client IP

50050 is the source port of the clients ssh connection.

Let’s check netstat now:

# netstat -an|grep EST
192.168.1.66.22 192.168.1.65.49980 16896 67 49640 0 ESTABLISHED

There it is the connection is dropped!

Installing Sybase Perl Module requires the following preinstalled

1. Sybase Server/Client already installed on your server
2. GNU Make

Install GNU Make

Assuming you have already installed Sybase Server/Client on your system, lets download and install GNU Make.

GNU Make package can be downloaded from sunfreeware.com website. At the time of writing, the version available for download is Make 3.81

Download GNU Make for Solaris 10 (SPARC)

Download Make for Solaris 10 (x86)

Unzip and Install GNU Make

From the directory where the file is downloaded

# gunzip make-3.81-sol10-x86-local.gz
# pkgadd -d make-3.81-sol10-x86-local

This should install “make” under /usr/local/bin

# which make
/usr/local/bin/make

Download Perl Modules from www.cpan.org. The modules we are interested in are

1. DBI (Database Independent Interface for Perl)

2. DBD::Sybase (Sybase database Driver for the DBI module)

NOTE: I find in Solaris 10 Update 6, DBI module is already installed which means we only need to install DBD::Sybase module.

To verify either of the modules is already installed on the system,

For DBI, file DBI.pm should exist under

x86

/usr/perl5/vendor_perl/5.8.4/i86pc-solaris-64int/DBI.pm

SPARC

/usr/perl5/site_perl/5.8.4/sun4-solaris-64int/DBI.pm

For DBD::Sybase

x86

/usr/perl5/site_perl/5.8.4/i86pc-solaris-64int/DBD/Sybase.pm

SPARC

/usr/perl5/site_perl/5.8.4/sun4-solaris-64int/DBD/Sybase.pm

Now, assuming both these modules are not installed

Install Perl DBI Module

Download the DBI Module from here

From the download directory,

# gunzip DBI-1.609.tar.gz
# tar -xvf DBI-1.609.tar
# cd DBI-1.609

The above steps should (g)unzip, untar the downloaded source files.

Export $SYBASE & $LD_LIBRARY_PATH environment variables. We have the Sybase client installed under

/opt/sybase/oc15.0-EBF16084/OCS-15_0

which means the libraries exist under

/opt/sybase/oc15.0-EBF16084/OCS-15_0/lib

# export SYBASE=/opt/sybase/oc15.0-EBF16084/OCS-15_0
# export LD_LIBRARY_PATH=/opt/sybase/oc15.0-EBF16084/OCS-15_0/lib:$LD_LIBRARY_PATH

If this step is skipped, the following error comes up in the next step:

Can’t find the lib directory under /export/home/sybase! at Makefile.PL line 118, <IN> line 44.

Now comes the tricky bit, use /usr/perl5/bin/perlgcc instead of default perl as follows:

# /usr/perl5/bin/perlgcc Makefile.PL

Let’s now compile and install using GNU Make from source directory

# /usr/local/bin/make
# /usr/local/bin/make test
# /usr/local/bin/make install

To make sure the install compelted successfully. Check if the DBI.pm file exists as shown earlier.

Install Perl DBD::Sybase module

Download DBD::Sybase perl module from here.

From the downloaded directory:

# gunzip DBD-Sybase-1.09.tar.gz
# tar -xvf DBD-Sybase-1.09.tar
# cd DBD-Sybase-1.09

The above should (g)unzip, untar the downloaded file and change into the source directory.

Export the $SYBASE & $LD_LIBRARY_PATH environment variables if you have already not done so.

# export SYBASE=/opt/sybase/oc15.0-EBF16084/OCS-15_0
# export LD_LIBRARY_PATH=/opt/sybase/oc15.0-EBF16084/OCS-15_0/lib:$LD_LIBRARY_PATH

Run perlgcc against the Makefile.PL as follows:

# /usr/perl5/bin/perlgcc Makefile.PL

Select the default CHAIN mode or modify as required, then enter the Sybase Server, a Sybase Username and password and a Database as and when promted. This will allow the installation test connection. Make sure this DB connection is updated in the Sybase “interfaces” file.

NOTE: Once the installation is completed. Either the PWD file created by this procedure or the entire Source directory should be deleted as the PWD file will contain these DB access details. Also, it is advisable to provide a test DB access detail rather than a live environment.

Let’s now compile and install using GNU Make from source directory

# /usr/local/bin/make

Now, re-export $SYBASE as follows:

# export SYBASE=/opt/sybase/oc15.0-EBF16084

[NOTE: This may vary in your environment]

# /usr/local/bin/make test
# /usr/local/bin/make install

This should install the Sybase Perl module. To confirm the installation was successful, look for Sybase.pm as explained earlier.

dladm is the admin utility for Data-Link Interface which helps to display informarthe like Link Status (UP/DOWN), Speed, Duplex, MTU, VLAN Tagged and crucially statistics of network traffic on each of the interfaces historically as well as in real time. dladm can also configure and admin Link Aggregation on multiple NICs which we will not focus here.

Show Link Status/Speed/Duplex

# dladm show-dev
nxge0           link: down      speed: 0     Mbps       duplex: unknown
nxge1           link: down      speed: 0     Mbps       duplex: unknown
nxge2           link: up        speed: 1000  Mbps       duplex: full
nxge3           link: up        speed: 1000  Mbps       duplex: full

As you can see above the “show-dev” option lists all the network interfaces with Link Status (UP/DOWN), current speed in Mbps and Duplex (Half/Full)

Show Link Status

# dladm show-link
nxge0           type: non-vlan  mtu: 1500       device: nxge0
nxge1           type: non-vlan  mtu: 1500       device: nxge1
nxge2           type: non-vlan  mtu: 1500       device: nxge2
nxge3           type: non-vlan  mtu: 1500       device: nxge3

Here “show-link” option reveals the MTU and the VLAN tagging detail on each of the interfaces on the system.

Show Stats of all Interfaces for all time

# dladm show-dev -s
ipackets  rbytes         ierrors opackets        obytes      oerrors
nxge0           0         0           0       0         0           0
nxge1           0         0           0       0         0           0
nxge2           179625752557169463759657      581104982 3964684165410
nxge3           22240891  1834257868  0       5198483   395084708   0

The “-s” option along with “show-dev” or “show-link” displays network traffic statistics including Input/Output packets, input/output errors.

Stats in real-time

To show the stats of a particular interface in real-time use the “-i” option which is the interval in seconds. The first line shows again historic stats and later on the real-time information for every “n” seconds (5 seconds in our example)

# dladm show-link -s -i 5 nxge2
ipackets  rbytes         ierrors opackets        obytes      oerrors
nxge2           179637824757173944575957      581119516 3964706801670
ipackets  rbytes         ierrors opackets        obytes      oerrors
nxge2           961       319105             0       150       17874       0
ipackets  rbytes         ierrors opackets        obytes      oerrors
nxge2           887       263850             0       117       16505       0

If we fail to mention the interface then it takes the default interface (1st in the list). As you can see from the example below, we see stats for nxge0 which is not connected.

# dladm show-link -s -i 5
ipackets  rbytes         ierrors opackets        obytes      oerrors
nxge0           0         0           0       0         0           0
ipackets  rbytes         ierrors opackets        obytes      oerrors
nxge0           0         0           0       0         0           0

The PCP script works on Solaris 10/9/8 and can be downloaded from here. Simply save the pcp.txt file to your Solaris Server as a Shell scripts (say pcp.sh) and change permissions to execute.

# chmod 755 pcp.sh

PIDs for TCP Port

Run PCP with “-p” option to show the PIDs of processes having a TCP port (say Port 22)

For instance, to find PIDs opening TCP port 22.

TCP Ports open by PIDs

Run PCP with “-P” option to show the TCP ports open by specific PID

For instance, here I try to find the TCP ports open by PID 18805

All PIDs for all open TCP Ports

Use the “-a” option to list all TCP ports open with all the PIDs

In Solaris 10, IP Forwarding can be enabled or disabled using the routeadm & ifconfig commands as against the ndd commands in Solaris 9 and earlier. The advantage is the change dynamic and real-time and the change persist across reboot unlike the ndd command.

Enable/Disable IP Forwarding globally 

To globally enable IP Forwarding in Solaris 10 use the routeadm command as follows:

In IPv4

solaris10# routeadm -e ipv4-forwarding

In IPv6

solaris10# routeadm -e ipv6-forwarding

The switches “-e” enables IP Forwarding.

To disable IP Forwarding

In IPv4

solaris10# routeadm -d ipv4-forwarding

In IPv6

solaris10# routeadm -d ipv6-forwarding

The switches “-d” enables IP Forwarding.

After the change run the following command for the changes to take effect.

solaris10# routeadm -u

Enable/Disable IP Forwarding per interface 

To enable IP Forwarding on a specific interface (say ce0) using the ifconfig command

In IPv4

solaris10# ifconfig ce0 router

In IPv6

solaris10# ifconfig ce0 inet6 router

To disable IP Forwarding for an interface (say ce0)

In IPv4

solaris10# ifconfig ce0 -router

In IPv6

solaris10# ifconfig ce0 inet6 -router

The DHCP Agent by default requests for the

Subnet Mask

Default Router

Hostname

DNS Domain

Broadcast Address

Encapsulated Vendor Option (vendor specific information as described in RFC 2132)

The DHCP Agent config file is

/etc/default/dhcpagent

In the file look for the line

PARAM_REQUEST_LIST=1,3,6,12,15,28,43

This is the line that defines what is being requested for from the DHCP Server. Here, each number stands for one of the above mentioned parameters where

1 = subnet mask

3 = Default Router

6 = DNS Server

12 = hostname

15 = DNS Domain Name

28 = broadcast address

43 = Encapsulated Vendor options

If you decide to not request for any of the above parameters then all you need to do is to remove the corresponding number from the PARAM_REQUEST_LIST. Let’s say you do not want to request for the hostname then simply remove the number “3″ and the “,” that follows so it looks as follows

PARAM_REQUEST_LIST=1,6,12,15,28,43

Once done,save the file. The next time the system restarts the hostname is not requested for from the DHCP Server.

# hostname
unknown

or the /etc/hosts file has an entry as follows:

# cat /etc/hosts
#
# Internet host table
#
127.0.0.1 localhost
192.168.0.3 unknown # Added by DHCP

This happens when the DHCP server doesn’t provide a hostname for your server. Most of the DHCP Servers or routers acting as a DHCP Server doesn’t provide a hostname and Solaris DHCP agent relies on the DHCP server for its hostname.

To fix this problem, edit the dhcpagent config and set it not to request a Hostname from the DHCP server and then add a hostname to the /etc/hosts & /etc/nodename file.

Edit DHCPAGENT config

[For Solaris 10, this is not required as setting the hostname in /etc/nodename overrides the hostname provided by the DHCP Server. Move onto the next step]

Edit the dhcpagent config file /etc/default/dhcpagent with your favorite editor

# vi /etc/default/dhcpagent

and look for

PARAM_REQUEST_LIST=1,3,6,12,15,28,43

and remove the Parameter “12″ from the above to look as follows:

PARAM_REQUEST_LIST=1,3,6,15,28,43

and save the file.

Now, add the hostname to the /etc/nodename & /etc/hosts as follows

# vi /etc/nodename

and add the hostname you want it to have (solarisserver for me)

# cat /etc/nodename
solarisserver

and edit the /etc/hosts file

# vi /etc/hosts

and add the line similar to the following

192.168.0.3 solarisserver loghost

Now,reboot the server and you should be all fine with hostname set for the system.

# init 6