The following error on the Sybase ASE intalled on Sun Solaris can occur if the Maximum shared memory allocated is max’d out or if none is allocated, it takes the default value which is “Total Physical Memory /4”.
Backup Server: 18.104.22.168:  The ‘shmget’ call failed with error number 22 (Invalid argument). Refer to your operating system documentation for further details.
To confirm, if maximum shared memory has been manually set,
# projects -l
projid : 101
comment: “Sybase ASE”
users : (none)
groups : (none)
For instance, here we have set the Maximum Shared memory to 4GB to the user account which owns and runs the Sybase ASE. And, the last line “attribs: project.max-shm-memory=(privileged,4294967296,deny)” basically, sets the maximum shared memory for the user processes to 4GB (4294967296).
This is equivalent to doing
# cat /etc/projects
If something similar doesn’t exist then the ASE is using the system defaults.
In any case, we need to increase the Maximum Shared memory. In our case increasing it to 7GB fixed the problem.
To add a new Project entry,
# projadd -p 101 -U asetest -c “Sybase ASE” -K “max-shm-memory=(privileged,7GB,deny)” user.asetest
This would create a new project with ID 101, name user.asetest for user asetest and set the Maximum Shared Memory to 7GB.
To modify an existing Project entry,
# projmod -sK “project.max-shm-memory=(privileged,7GB,deny)” user.asetest
This should help!