While working on some issues with performance, we needed a bit more granularity than the 10 minute default that sysstat(sar) comes with. It is pretty helpful when it comes to gauging general performance over time. But what if I need to try and correlate a spike in something down to the minute, or even second?
Assuming you have it installed (yum -y install sysstat), you can adjust down to the minute easily. You simply change the value in /etc/cron.d/sysstat.
eg(default sar cron config):
[root@test-web1 ~]# cat /etc/cron.d/sysstat
# Run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# 0 * * * * root /usr/lib64/sa/sa1 600 6 &
# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A
[root@test-web1 ~]# sar -q
Linux 3.10.0-514.26.2.el7.x86_64 (test-web1) 07/30/2017 _x86_64_ (1 CPU)
01:49:37 AM LINUX RESTART
01:50:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
02:00:01 AM 1 140 0.00 0.01 0.05 0
02:10:01 AM 1 140 0.00 0.01 0.05 0
02:20:01 AM 1 141 0.00 0.01 0.05 0
Average: 1 140 0.00 0.01 0.05 0
[root@test-web1 ~]#
Above is the standard config. If you want to drop this down to get stats every one minute, simply change the first field calling sa1.
[root@test-web1 ~]# cat /etc/cron.d/sysstat
*/1 * * * * root /usr/lib64/sa/sa1 1 1
53 23 * * * root /usr/lib64/sa/sa2 -A
[root@test-web1 ~]#
Now the sa1 script gets called every one minute. So we can see stats down to the minute.
[root@test-web1 ~]# sar -q
Linux 3.10.0-514.26.2.el7.x86_64 (test-web1) 07/30/2017 _x86_64_ (1 CPU)
01:49:37 AM LINUX RESTART
01:50:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
02:00:01 AM 1 140 0.00 0.01 0.05 0
02:10:01 AM 1 140 0.00 0.01 0.05 0
02:20:01 AM 1 141 0.00 0.01 0.05 0
02:30:01 AM 3 141 0.00 0.01 0.05 0
02:31:01 AM 1 141 0.08 0.04 0.05 0
02:32:02 AM 1 140 0.03 0.04 0.05 0
02:33:01 AM 1 140 0.01 0.03 0.05 0
02:34:01 AM 1 140 0.00 0.02 0.05 0
02:35:01 AM 1 140 0.00 0.02 0.05 0
02:36:01 AM 1 140 0.00 0.01 0.05 0
02:37:01 AM 1 140 0.00 0.01 0.05 0
02:38:01 AM 1 140 0.00 0.01 0.05 0
02:39:01 AM 1 140 0.00 0.01 0.05 0
Average: 1 140 0.01 0.02 0.05 0
[root@test-web1 ~]#
As you can see it went from every 10 minutes, to every one minute.
But what if I want to go a step further? What if I want it every 30 seconds? Ah, misusing system utilities, now we’re having fun!
[root@test-web1 ~]# cat /etc/cron.d/sysstat
*/1 * * * * root /usr/lib64/sa/sa1 1 1 ; sleep 30 ; /usr/lib64/sa/sa1 1 1
53 23 * * * root /usr/lib64/sa/sa2 -A
[root@test-web1 ~]#
The cron job will run, then sleep for 30 seconds and do it again.
[root@test-web1 ~]# sar -q
Linux 3.10.0-514.26.2.el7.x86_64 (test-web1) 07/30/2017 _x86_64_ (1 CPU)
01:49:37 AM LINUX RESTART
01:50:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
02:00:01 AM 1 140 0.00 0.01 0.05 0
02:10:01 AM 1 140 0.00 0.01 0.05 0
02:20:01 AM 1 141 0.00 0.01 0.05 0
02:30:01 AM 3 141 0.00 0.01 0.05 0
02:31:01 AM 1 141 0.08 0.04 0.05 0
02:32:02 AM 1 140 0.03 0.04 0.05 0
02:33:01 AM 1 140 0.01 0.03 0.05 0
02:34:01 AM 1 140 0.00 0.02 0.05 0
02:35:01 AM 1 140 0.00 0.02 0.05 0
02:36:01 AM 1 140 0.00 0.01 0.05 0
02:37:01 AM 1 140 0.00 0.01 0.05 0
02:38:01 AM 1 140 0.00 0.01 0.05 0
02:39:01 AM 1 140 0.00 0.01 0.05 0
02:40:01 AM 1 140 0.00 0.01 0.05 0
02:41:01 AM 1 141 0.00 0.01 0.05 0
02:41:31 AM 2 141 0.00 0.01 0.05 0
02:42:01 AM 1 141 0.00 0.01 0.05 0
02:42:31 AM 1 141 0.00 0.01 0.05 0
02:43:01 AM 1 141 0.00 0.01 0.05 0
Average: 1 140 0.01 0.02 0.05 0
[root@test-web1 ~]#
So, now we see it running every 10 minutes, then every 1 minute, and now every 30 seconds.
What about… every one second?(just because we can, does not mean we should)
[root@test-web1 ~]# cat /etc/cron.d/sysstat
*/1 * * * * root for i in `seq 1 59` ; do /usr/lib64/sa/sa1 1 1 ; sleep 1 ; done
53 23 * * * root /usr/lib64/sa/sa2 -A
[root@test-web1 ~]#
With a little bit of shell scripting, we have stats every one second.
[root@test-web1 ~]# sar -q
Linux 3.10.0-514.26.2.el7.x86_64 (test-web1) 07/30/2017 _x86_64_ (1 CPU)
01:49:37 AM LINUX RESTART
01:50:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
02:00:01 AM 1 140 0.00 0.01 0.05 0
02:10:01 AM 1 140 0.00 0.01 0.05 0
02:20:01 AM 1 141 0.00 0.01 0.05 0
02:30:01 AM 3 141 0.00 0.01 0.05 0
02:31:01 AM 1 141 0.08 0.04 0.05 0
02:32:02 AM 1 140 0.03 0.04 0.05 0
02:33:01 AM 1 140 0.01 0.03 0.05 0
02:34:01 AM 1 140 0.00 0.02 0.05 0
02:35:01 AM 1 140 0.00 0.02 0.05 0
02:36:01 AM 1 140 0.00 0.01 0.05 0
02:37:01 AM 1 140 0.00 0.01 0.05 0
02:38:01 AM 1 140 0.00 0.01 0.05 0
02:39:01 AM 1 140 0.00 0.01 0.05 0
02:40:01 AM 1 140 0.00 0.01 0.05 0
02:41:01 AM 1 141 0.00 0.01 0.05 0
02:41:31 AM 2 141 0.00 0.01 0.05 0
02:42:01 AM 1 141 0.00 0.01 0.05 0
02:42:31 AM 1 141 0.00 0.01 0.05 0
02:43:01 AM 1 141 0.00 0.01 0.05 0
02:43:31 AM 1 141 0.00 0.01 0.05 0
02:44:01 AM 1 141 0.00 0.01 0.05 0
02:44:31 AM 1 141 0.00 0.01 0.05 0
02:45:01 AM 1 141 0.00 0.01 0.05 0
02:45:31 AM 1 141 0.00 0.01 0.05 0
02:46:01 AM 2 141 0.00 0.01 0.05 0
02:46:31 AM 1 141 0.00 0.01 0.05 0
02:47:01 AM 1 141 0.00 0.01 0.05 0
02:47:31 AM 1 142 0.00 0.01 0.05 0
02:48:01 AM 1 141 0.00 0.01 0.05 0
02:48:02 AM 1 141 0.00 0.01 0.05 0
02:48:03 AM 1 141 0.00 0.01 0.05 0
02:48:04 AM 1 141 0.00 0.01 0.05 0
02:48:05 AM 1 141 0.00 0.01 0.05 0
02:48:06 AM 1 141 0.00 0.01 0.05 0
02:48:07 AM 1 141 0.00 0.01 0.05 0
02:48:08 AM 1 141 0.00 0.01 0.05 0
02:48:09 AM 1 142 0.00 0.01 0.05 0
02:48:10 AM 1 142 0.00 0.01 0.05 0
02:48:11 AM 2 142 0.00 0.01 0.05 0
02:48:12 AM 1 142 0.00 0.01 0.05 0
02:48:13 AM 1 142 0.00 0.01 0.05 0
02:48:14 AM 1 142 0.00 0.01 0.05 0
02:48:15 AM 1 142 0.00 0.01 0.05 0
02:48:16 AM 1 142 0.00 0.01 0.05 0
02:48:17 AM 1 142 0.00 0.01 0.05 0
02:48:18 AM 1 142 0.00 0.01 0.05 0
02:48:19 AM 1 142 0.00 0.01 0.05 0