Finding overall and per core CPU utilization

Today I will post about how to monitor CPU usage by processor in Linux. As you might have expected this will simply access the CPU time information from /proc pseudo-filesystem and report the results.

The proc filesystem or the Procfs is a special filesystem which gives you a view into the kernel data. No files in procfs exists actually in the disk. There is no disk inodes and thus storage related to the files. Instead of going into procfs I will redirect you to wikipedia: http://en.wikipedia.org/wiki/Procfs. Let’s get in.

What’s the stuff

The file which is required to get the CPU usage is /proc/stat. This file contains some stats about the kernel, like the number of processes created since boot, number or processes in runnable/blocked states, number of context switches in the system till now, number of swaps or page in page outs, and information about usage of CPU cycles. What fields the file will contain varies with the architecture and the kernel version. For details about the /proc file system check man 5 proc.

We are interested in the file /proc/stat and within it we are interested in the field cpu. If you open the file you will see that some integers follows the cpu field. They have specific interpretations as described in the manual. The kernel I am running has 10 fields which have the following interpretations.

+-------+------------+-------------------------------------------------------+
| field |    name    |   description                                         |
+-------+------------+-------------------------------------------------------+
|   1   |    user    |   Time spent in user mode                             |
+-------+------------+-------------------------------------------------------+
|   2   |    nice    |   Time spent in user mode with low priority           |
+-------+------------+-------------------------------------------------------+
|   3   |   system   |   Time spent in system mode                           |
+-------+------------+-------------------------------------------------------+
|   4   |    idle    |   Time spent in idle task                             |
+-------+------------+-------------------------------------------------------+
|   5   |   iowait   |   Time waiting for I/O to complete                    |
+-------+------------+-------------------------------------------------------+
|   6   |     irq    |   Time servicing interrupts                           |
+-------+------------+-------------------------------------------------------+
|   7   |   softirq  |   Time servicing softirqs                             |
+-------+------------+-------------------------------------------------------+
|   8   |    steal   |   Time spent in other OSes when in virtualized env    |
+-------+------------+-------------------------------------------------------+
|   9   |    quest   |   Time spent running a virtual CPU for guest OS       |
+-------+------------+-------------------------------------------------------+
|  10   | quest_nice |   Time spent running niced guest                      |
+-------+------------+-------------------------------------------------------+

First we will be interpreting only fields 1, 2, 3, and 4. These should be present in all Linux kernels. For example

cpu  434566 7188 225392 23354859 217973 28 4668 0 0 0

says the CPU of the system (all the CPUs accumulated) has spent 434566 unit time in user mode, 7188 unit of time in low priority user mode, 225392 units of time in system mode, 23354859 units of time as idle. These values indicates the measure of time since the system was booted. We are not interested about the unit of time, because we are interested to find ratio. Although for information, these are in number of cycles spent. Number of cycles in a second in your system can be found by executing

getconf CLK_TCK

For my system the result is 100. That means there are 100 clock ticks per second, which makes one cycle of duration 1/100 second. Therefore in the above example the CPU has ran for 434566/100 = 4345.66 seconds in user mode.

Overall CPU utilization

Coming back to the point. To find what is the load on the processor (all CPUs accumulated) we need to find what fraction of the total time did the CPU spend in user and system mode in a specific time interval. Note that the values in the fields are total time since boot (mod), therefore we need to find the change of the field values in some interval. The process is as follows. We will read the first 4 fields first at time 0. Say the values are user(0), nice(0), system(0) and idle(0). Then keep on reading then in specific time intervals t_1, t_2, t_3, ... ,t_{i - 1}, t_i, ... . Here each t_i are the time at which we read the fields, and t_i - t_{i - 1} = delta_t where delta_t denotes the actual intervals between the readings (say 1 second). After each reading, we will find the change (delta) in the value compared to the previous reading by delta_user(t) = user(t) - user(t-1), delta_nice(t) = nice(t) - nice(t-1), delta_system(t) = system(t) - system(t-1) and delta_idle(t) = idle(t) - idle(t-1), where delta_t = 1 in this example. This tells us what was the usage of time in the corresponding fields in the last delta_t second. Once we find these deltas, we have the time spent for each 4 fields in the last interval. Now we need to just find the percentage of time spent in user + nice + system fields current delta to the total amount of time total spend in this delta.

                             user_delta(t)  +  nice_delta(t)  +  system_delta (t)
Percent CPU Usage = ---------------------------------------------------------------------- X 100
                       user_delta(t) + nice_delta(t) + system_delta (t) + idle_delta(t)

Here delta_t1 = t1 - t0, this is because, when we read the values first, we have no base values with which we can find the deltas. Therefore for the first computation we need two reads to generate the first delta, after that point each read will get us a delta compared to the last read.

Therefore the process is easy. Read the “cpu” fields after every second (or any time interval) and find the change in each field compared to the last read to get the time spent since the last second (or any time interval), then find the percentage of time spent in the last second (or any time interval) compared to the total time spent since the last change.

The above discussion was done only with the four fields, but the CPU also does spend time to do other works like servicing interrupt requests, do iowait etc, which are listed in the table. Therefore to compute the percent of processor used I will also include the other fields too. This is exactly similar to the above discussion. Instead of the 4 fields, now we find the percentage of the delta since last second (or any time interval) which is the summation of the delta for the 9 fields except the idle field, compared to the total time spent, which is the summation of the deltas of all the 10 fields.

An issue is how to interpret iowait. This is because when the cpu is idle and is doing an I/O the iowait is incremented and if not doing an I/O then idle is incremented. But when a process is waiting for an I/O to complete the cpu will schedule another process on the processor, therefore on the context of the process it is not progressing but the CPU does continue with another process. I will update the post whenever I find an accurate answer, maybe after reading codes of existing tools (can anyone help?). For now I am considering iowait as a task as a time “doing waiting”.

Sourcecode

Here is the perl code to find the total CPU usage:

#!/usr/bin/perl -w
#Show overall CPU utilization of the system
#This is a part of the post https://phoxis.org/2013/09/05/finding-overall-and-per-core-cpu-utilization

use strict;
use warnings;
use List::Util qw(sum);
use Fcntl;

my $idle_old = 0;
my $total_old = 0;

open (my $STAT, "/proc/stat") or die ("Cannot open /proc/stat\n");
while (1)
{
  seek ($STAT, Fcntl::SEEK_SET, 0);

  while (<$STAT>)
  {
    next unless ("$_" =~ m/^cpu\s+/);

    my @cpu_time_info = split (/\s+/, "$_");
    shift @cpu_time_info;
    my $total = sum(@cpu_time_info);
    my $idle = $cpu_time_info[3];

    my $del_idle = $idle - $idle_old;
    my $del_total = $total - $total_old;

    my $usage = 100 * (($del_total - $del_idle)/$del_total);

    printf ("Total CPU Usage: %0.2f%%\n", $usage);

    $idle_old = $idle;
    $total_old = $total;
  }
  sleep 1;
}
close ($STAT);

Same with C language

/* Show overall CPU utilization of the system 
 * This is a part of the post https://phoxis.org/2013/09/05/finding-overall-and-per-core-cpu-utilization
 */
#include <stdio.h>
#include <unistd.h>

#define BUF_MAX 1024

int read_fields (FILE *fp, unsigned long long int *fields)
{
  int retval;
  char buffer[BUF_MAX];


  if (!fgets (buffer, BUF_MAX, fp))
  { perror ("Error"); }
  retval = sscanf (buffer, "cpu %Lu %Lu %Lu %Lu %Lu %Lu %Lu %Lu %Lu %Lu", 
                            &fields[0], 
                            &fields[1], 
                            &fields[2], 
                            &fields[3], 
                            &fields[4], 
                            &fields[5], 
                            &fields[6], 
                            &fields[7], 
                            &fields[8], 
                            &fields[9]); 
  if (retval < 4) /* Atleast 4 fields is to be read */
  {
    fprintf (stderr, "Error reading /proc/stat cpu field\n");
    return 0;
  }
  return 1;
}

int main (void)
{
  FILE *fp;
  unsigned long long int fields[10], total_tick, total_tick_old, idle, idle_old, del_total_tick, del_idle;
  int update_cycle = 0, i, flag = 1;
  double percent_usage;

  fp = fopen ("/proc/stat", "r");
  if (fp == NULL)
  {
    perror ("Error");
  }


  if (!read_fields (fp, fields)) 
  { return 0; }

  for (i=0, total_tick = 0; i<10; i++)
  { total_tick += fields[i]; }
  idle = fields[3]; /* idle ticks index */

  while (flag)
  {
    sleep (1);
    total_tick_old = total_tick;
    idle_old = idle;
    
    fseek (fp, 0, SEEK_SET);
    fflush (fp);
    if (!read_fields (fp, fields))
    { return 0; }

    for (i=0, total_tick = 0; i<10; i++)
    { total_tick += fields[i]; }
    idle = fields[3];

    del_total_tick = total_tick - total_tick_old;
    del_idle = idle - idle_old;

    percent_usage = ((del_total_tick - del_idle) / (double) del_total_tick) * 100; /* 3 is index of idle time */
    printf ("Total CPU Usage: %3.2lf%%\n", percent_usage);
    update_cycle++;
  }


  fclose (fp); /* Ctrl + C quit, therefore this will not be reached. We rely on the kernel to close this file */

  return 0;
}

Note that in both the case the we need to end the process using Ctrl + C. In that case we won’t be able to close the file, and do other stuffs (if we had anything to do). We can solve this we can do something like as follows. Declare flag = 1 a global variable. Use while (flag) and set a signal handler which will set flag = 0 on SIGINT. This is not shown in the code.

CPU utilization per CPU core

The /proc/stat holds the CPU time usage for each core of the CPU also. Therefore using those fields you can find per core usage percentage. The process is identical. Just find the per core usage as above. Here are the codes. The change here is just reading each of the lines for each core and calculating the percent CPU usage per core. In the case of Perl I have counted the number of cores available in the system using the /proc/cpuinfo file. In the C implementation the number of lines which start with “cpu” or “cpu[0-9]” is counted initially outside the while loop which determines the number of cores in the system. A set of variables for total time, idle time are stored for each core.

Sourcecode

Almost identical to the above code:

#!/usr/bin/perl -w

# Show per core CPU utilization of the system 
# This is a part of the post https://phoxis.org/2013/09/05/finding-overall-and-per-core-cpu-utilization

use strict;
use warnings;
use List::Util qw(sum);
use Fcntl;

#Open /proc/cpuinfo and count the number of cpus in the system
#For each cpu this file will have a section of description of that cpu
#And each such file will contain a field "processor" which lists the
#processor number of that processor.
open ($CPUINFO, "/proc/cpuinfo");
my $cpu_count = 0;
while (<$CPUINFO>)
{
  next unless "$_" =~ /^processor/;
  $cpu_count++;
}
close ($CPUINFO);

#Initialize old states for each cpu.
my @idle_old = split ('', 0 x $cpu_count);
my @total_old = split ('', 0 x $cpu_count);
my $update_cycle = 0;

#Infinitely update cpu usage, press Ctrl + C to quit
while (1)
{
  #Open /proc/stat file
  open (my $STAT, "/proc/stat") or die "Cannot open /proc/stat\n";

  my $count = 0;
  my $total_usage = 0;
  print "[Update cycle: $update_cycle]\n";
  while (<$STAT>)
  {
    #Find the lines starting with cpu followed by a number and whitespace and parse it
    #Formula is to find the delta of total time and idle time of the cpu from the last update
    #and calcute the ratio of time in processing (not idle) to the total time this cycle.
    #check manual for /proc/status field description (man 5 proc). Also average the n number of cpus and
    #show the overall cpu usage.
    
    next unless ("$_" =~ m/^cpu[0-9]+\s+/);

    my @cpu_time_info = split (/\s+/, "$_");
    shift @cpu_time_info;
    my $total = sum(@cpu_time_info);
    my $idle = $cpu_time_info[3];

    my $del_idle = $idle - $idle_old[$count];
    my $del_total = $total - $total_old[$count];

    my $this_cpu_usage = 100 * (($del_total - $del_idle)/$del_total);
    $total_usage = $total_usage + $this_cpu_usage;

    printf ("CPU%d Usage: %0.2f%%\n", $count, $this_cpu_usage);

    $idle_old[$count] = $idle;
    $total_old[$count] = $total;

    $count++;
  }
  $update_cycle++;
  printf ("Total CPU Usage: %0.2f%%\n", $total_usage/$cpu_count); 
  print "\n";
  close ($STAT);
  sleep 1;
}

Same with C language

/* Show per core CPU utilization of the system 
 * This is a part of the post https://phoxis.org/2013/09/05/finding-overall-and-per-core-cpu-utilization
 */
#include <stdio.h>
#include <unistd.h>

#define BUF_MAX 1024
#define MAX_CPU 128

int read_fields (FILE *fp, unsigned long long int *fields)
{
  int retval;
  char buffer[BUF_MAX];


  if (!fgets (buffer, BUF_MAX, fp))
  { perror ("Error"); }
  /* line starts with c and a string. This is to handle cpu, cpu[0-9]+ */
  retval = sscanf (buffer, "c%*s %Lu %Lu %Lu %Lu %Lu %Lu %Lu %Lu %Lu %Lu",
                            &fields[0], 
                            &fields[1], 
                            &fields[2], 
                            &fields[3], 
                            &fields[4], 
                            &fields[5], 
                            &fields[6], 
                            &fields[7], 
                            &fields[8], 
                            &fields[9]); 
  if (retval == 0)
  { return -1; }
  if (retval < 4) /* Atleast 4 fields is to be read */
  {
    fprintf (stderr, "Error reading /proc/stat cpu field\n");
    return 0;
  }
  return 1;
}

int main (void)
{
  FILE *fp;
  unsigned long long int fields[10], total_tick[MAX_CPU], total_tick_old[MAX_CPU], idle[MAX_CPU], idle_old[MAX_CPU], del_total_tick[MAX_CPU], del_idle[MAX_CPU];
  int update_cycle = 0, i, cpus = 0, count;
  double percent_usage;


  fp = fopen ("/proc/stat", "r");
  if (fp == NULL)
  {
    perror ("Error");
  }


  while (read_fields (fp, fields) != -1)
  {
    for (i=0, total_tick[cpus] = 0; i<10; i++)
    { total_tick[cpus] += fields[i]; }
    idle[cpus] = fields[3]; /* idle ticks index */
    cpus++;
  }

  while (1)
  {
    sleep (1);
    fseek (fp, 0, SEEK_SET);
    fflush (fp);
    printf ("[Update cycle %d]\n", update_cycle); 
    for (count = 0; count < cpus; count++)
    {
      total_tick_old[count] = total_tick[count];
      idle_old[count] = idle[count];
    
      if (!read_fields (fp, fields))
      { return 0; }

      for (i=0, total_tick[count] = 0; i<10; i++)
      { total_tick[count] += fields[i]; }
      idle[count] = fields[3];

      del_total_tick[count] = total_tick[count] - total_tick_old[count];
      del_idle[count] = idle[count] - idle_old[count];

      percent_usage = ((del_total_tick[count] - del_idle[count]) / (double) del_total_tick[count]) * 100;
      if (count == 0)
      { printf ("Total CPU Usage: %3.2lf%%\n", percent_usage); }
      else 
      { printf ("\tCPU%d Usage: %3.2lf%%\n", count - 1, percent_usage); }
    }
    update_cycle++;
    printf ("\n");
  }

  /* Ctrl + C quit, therefore this will not be reached. We rely on the kernel to close this file */
  fclose (fp);

  return 0;
}

You can also display the percent of usage of user, system etc separately, like it is shown in top.

End of story.

Links and References

• What is exactly “iowait”
• The /proc Filesystem
• man 5 proc
• http://procps.sourceforge.net/