Linux - understanding what's using my memory

[Handruin]

I have a general question that I hope someone can clarify or point me to information to research more on. I have a CentOS 6.6 box that I've been trying to account for everything consuming memory in a very detailed manner. I've used the basics of tools like "free -m", "top", "vmstat", "collectl", "/proc/meminfo" and even run a crude command "ps aux | awk '{sum+=$6} END {print sum / 1024}" to estimate the RSS memory being used by the system. Even then I've not accounted for a portion of the used memory and I'd like to know how to do this if it's even possible?

For example my box has 192GB of physical memory of which only a tiny amount is available based on what free returns.

root@system ~]# free -m
             total       used       free     shared    buffers     cached
Mem:        193661     191676       1984         68         77        344
-/+ buffers/cache:     191254       [COLOR="#FF0000"][B]2406[/B][/COLOR]
Swap:         1023         50        973

When I run "ps aux | awk '{sum+=$6} END {print sum / 1024}'" the value returned is: 165203MB (165GB) which is an estimate of all user space programs running. I have a single process configured to use 160GB of RAM for a specialty purpose. I know that I have ZFS configured to use about 16GB of RAM for ARC but aside from knowing that I configured this, what other method would itemize and show for me where this is being reserved from a kernel/system level? I know I can see the usage via running "cat /proc/spl/kstat/zfs/arcstats" which confirms I'm using all 16GB right now. Once I add in the 16GB to my 165GB I'm at 181GB of used memory. What's using the remaining 10GB? I'm assuming the kernel/OS uses some amount but how can I see this?

 

[Stereodude]

I don't know the answer to your question, but you have a PC with 192GB of RAM!?!?! :salut:

 

[Handruin]

I wasn't meaning it to show off, this is a special purpose system that I'm trying to account for memory of which I know a few user space processes that consume most of it. I realize that using "ps" is not very accurate because of many factors (shared mem, slabs, etc) but I'm trying to find a way to get close to summarizing the usage of memory in a Linux system. This process should in theory work for any Linux system regardless of memory.

 

[Mercutio]

Kernel Process data lives in /proc/slabinfo.

I guess you could individually parse the data from each entry in /proc/$pid/status, looking all the Vm* entries to break things down the way you need and add that to the output from slabinfo, but that seems like a lot of unpleasant work and I'm not 100% sure even that will give you the accuracy you need.

 

[Handruin]

Thanks for the help. When I'm looking through slabinfo and see the following info, am I parsing the sizes correctly if I multiple (num_objs * objsize)? It's unclear to me if the size is listed in bytes or some other unit. I'm making an assumption the units are in bytes.

So in the example below, I would estimate that something like "iscsi_cmnd" would be 10 objects * 384 bytes (3840 bytes)?

[B]name  <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>[/B]
iser_descriptors       0      0    128   30    1 : tunables  120   60    8 : slabdata      0      0      0
ib_mad                 0      0    512    7    1 : tunables   54   27    8 : slabdata      0      0      0
scsi_tgt_cmd           0      0     80   48    1 : tunables  120   60    8 : slabdata      0      0      0
iscsi_session          1      1   4352    1    2 : tunables    8    4    0 : slabdata      1      1      0
iscsi_conn             1      9    832    9    2 : tunables   54   27    8 : slabdata      1      1      0
iscsi_thread_pool      1     11    704   11    2 : tunables   54   27    8 : slabdata      1      1      0
[COLOR="#FF0000"][B]iscsi_cmnd[/B]             [B]1[/B]     [B]10[/B]    [B]384[/B][/COLOR]   10    1 : tunables   54   27    8 : slabdata      1      1      0
scst_blockio_work      0      0     16  202    1 : tunables  120   60    8 : slabdata      0      0      0

 

[Handruin]

I am also running collectl on this system to help me gain insight to the memory and process-level usage. It also collects slabs and I see the following in its plotable file. These values concur with slabinfo (which is where it is probably getting these details from).

#Date Time SlabName ObjInUse ObjInUseB [B]ObjAll ObjAllB[/B] SlabInUse SlabInUseB SlabAll SlabAllB SlabChg SlabPct
20150819 00:00:33 iscsi_cmnd 9 3456 [B][COLOR="#FF0000"]10 3840[/COLOR][/B] 1 4096 1 4096 0 0

 

[timwhit]

Can you try running DTrace? https://github.com/dtrace4linux/linux

 

[Chewy509]

Doug, have you seen the following blog post?

http://www.dedoimedo.com/computers/slabinfo.html

Also in addition to /proc/$pid/status, /proc/$pid/smaps will give you additional information about shared pages allocated as well. So to get the accuracy, you may need to parse status and smaps, correlate between all entries, etc...

FYI. ps will report values based on the number of pages allocated, but does not take into account (correctly) shared pages between processes (which are either shared code, or shared memory (SHM) ). SHM is used a lot with X applications for buffer/texture management as well. So use it's values with a grain of salt.

 

[Handruin]

Can you try running DTrace? https://github.com/dtrace4linux/linux

I'll give this a try, thanks for sending me a link. I haven't used this before now.

 

[Handruin]

Doug, have you seen the following blog post?

http://www.dedoimedo.com/computers/slabinfo.html

Also in addition to /proc/$pid/status, /proc/$pid/smaps will give you additional information about shared pages allocated as well. So to get the accuracy, you may need to parse status and smaps, correlate between all entries, etc...

FYI. ps will report values based on the number of pages allocated, but does not take into account (correctly) shared pages between processes (which are either shared code, or shared memory (SHM) ). SHM is used a lot with X applications for buffer/texture management as well. So use it's values with a grain of salt.

Thanks for this link I hadn't seen this one yet. This covers exactly the problem I've been facing and it clarified a bit about estimated memory. I wasn't even aware of the utility called slabtop until today.

I understand the exercise I'm going through seems tedious and I may not account for all memory usage 100% but I needed to at least get closer than I was than accounting for just user space programs. I'm relying loosely on using 'ps' and my system has no GUI components loaded such as X that I'm aware of. This deployment of CentOS 6.6 is based on my own kickstart with the bare minimum packages from the Core and Base set along with a handful of packages needed for special-purpose application.

I feel like I'm closer to an answer and I appreciate the help with this. I've learned a bunch about Linux memory management and have a better feel for identifying where my memory is possibly being used.

 

[timwhit]

I'll give this a try, thanks for sending me a link. I haven't used this before now.

I've used DTrace with Solaris in the past, you can get pretty in-depth data with it. I've never used it with Linux, so your results may vary.