Sysinfo Logging

This article shows my approach to periodically record my server load with minimum overhead. A system timer executes a C program writes the system information into a page and cluster aligned file. The file is then downloaded from server to client and rendered using Javascript in HTML-SVG.

--by Captdam @ Oct 28, 2026 Oct 28, 2026

Index

I had my old server running Longview for several years. Recently, I installed my website on a new Linode server with a new Ubuntu distribution version, but I am having bad luck installing Longview. After some troubleshooting and headache, I decided to stop. After all, it is just a system performance logging program, why can't I write my own?

The free plan of Longview records system loads every 5 minutes, for up to 12 hours of data retention, paid plans records every minute and provide unlimited data retention. However, If I write my own, I can have higher resolution (up to every second) and unlimited data retention with no cost, except some CPU time and disk storage on my server.

Record Information

The first question is, what should I record?

Definitely, I need to record CPU usage, RAM usage and swap usage. The CPU usage tells us how much request we can serve at what rate; the RAM and swap usage tells us whether we are facing a potential out-of-memory problem. (There was once my server running out of memory and my websites were down for a day; therefore I am concerned about this issue.)

Since my server is dedicated to serve websites, the main load on this server is PHP-FPM. Therefore, I think there is no need to record the task tree. In terms of FPM status (fpm_get_status()), I will put it in another tool.

Record File Structure

I will save the system information record in a file. This file contains a header, which includes:

, followed by an array of data entries:


#define DATA_COUNT 255
typedef struct {
	header_t header;
	data_t data[DATA_COUNT];
} buffer_t;
	

The data entries array acts like a ring buffer. We write the first entry at time 1, then write the second entry at time 2. Once all data entries are written, we warp back to the first entry. An index specifies the next entry to write data_wp is stored in the header. The following table illustrates the ring buffer architecture:

Ring Buffer File Structure
OffsetContent
Headerdata_wp = 3
Entry 0Record [t-3]
Entry 1Record [t-2]
Entry 2Record [t-1]
Entry 3Record [t]
Entry 4Record [t+1-DATA_COUNT]
......
Entry DATA_COUNT-1Record [t-4]

With the ring buffer design, we only write the header and the specific entry. Compared to regular FIFO where we must shift every entry in the buffer, the ring buffer needs minimum number of writes, hence provides the best performance.

We will manually implement the ring buffer in C language:


buffer_t* buffer = getRecordFile();
header_t* header = &(buffer->header);

data_t* data = &(buffer->data[header->data_wp]);
data->something = getSomething();
header->data_wp = header->data_wp == DATA_COUNT - 1 ? 0 : header->data_wp + 1;
	

Note, the number of data entries is NOT a power of 2. We use header->data_wp = header->data_wp == DATA_COUNT - 1 ? 0 : header->data_wp + 1 instead of header->data_wp = (header->data_wp + 1) & (DATA_COUNT - 1) to wrap back the index.

In our record file implementation, the record file is aligned with the system memory page size and disk cluster size. In another word, what is a power of 2 is the size of the record file, not the number of data entries.

Sysinfo

On Linux, we can get the memory information by reading the system "files" /proc/meminfo:


cat /proc/meminfo

MemTotal:         979328 kB
MemFree:          119688 kB
MemAvailable:     623512 kB
Buffers:           70664 kB
Cached:           544036 kB
SwapCached:        19616 kB
...
	

and get the CPU load by reading /proc/stat:


cat /proc/stat

cpu  742019 33020 1270627 216150208 151630 0 12232 156155 0 0
cpu0 742019 33020 1270627 216150208 151630 0 12232 156155 0 0
intr 306200231 94 9 0 0 1624 0 0 0 1 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15204220 0 7117751 8178783 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
...
	

For example, the following PHP script parses memory information:


function getSizeRaw(string $value): int {
	$t = trim($value);
	if (substr($t, -2) == 'kB') {
		return 1024 * substr($t, 0, -2);
	} else if (substr($t, -2) == 'MB') {
		return 1024 * 1024 * substr($t, 0, -2);
	} else if (substr($t, -2) == 'GB') {
		return 1024 * 1024 * 1024 * substr($t, 0, -2);
	}
	return (int)$t;
}

$data = [];
foreach(file('/proc/meminfo', FILE_SKIP_EMPTY_LINES) as $x) {
	[$name, $value] = explode(':', $x);
	switch (trim($name)) {
		case 'MemTotal':	$data['total'] = getSizeRaw($value);		break;
		case 'MemFree':		$data['free'] = getSizeRaw($value);		break;
		case 'MemAvailable':	$data['avail'] = getSizeRaw($value);		break;
		case 'SwapTotal':	$data['swaptotal'] = getSizeRaw($value);	break;
		case 'SwapFree':	$data['swapfree'] = getSizeRaw($value);		break;
	}
}
	

The C standard library (libc) provides a system call sysinfo() that allows us to get the current system information without manually parse the system file, including:

In addition, 64-bit platform includes:

Size Optimization

The sysinfo() system call returns a structure:


// 32-bit
struct sysinfo {
	long uptime;             /* Seconds since boot */
	unsigned long loads[3];  /* 1, 5, and 15 minute load averages */
	unsigned long totalram;  /* Total usable main memory size */
	unsigned long freeram;   /* Available memory size */
	unsigned long sharedram; /* Amount of shared memory */
	unsigned long bufferram; /* Memory used by buffers */
	unsigned long totalswap; /* Total swap space size */
	unsigned long freeswap;  /* Swap space still available */
	unsigned short procs;    /* Number of current processes */
	char _f[22];             /* Pads structure to 64 bytes */
};

// 64-bit
struct sysinfo {
	long uptime;             /* Seconds since boot */
	unsigned long loads[3];  /* 1, 5, and 15 minute load averages */
	unsigned long totalram;  /* Total usable main memory size */
	unsigned long freeram;   /* Available memory size */
	unsigned long sharedram; /* Amount of shared memory */
	unsigned long bufferram; /* Memory used by buffers */
	unsigned long totalswap; /* Total swap space size */
	unsigned long freeswap;  /* Swap space still available */
	unsigned short procs;    /* Number of current processes */
	unsigned long totalhigh; /* Total high memory size */
	unsigned long freehigh;  /* Available high memory size */
	unsigned int mem_unit;   /* Memory unit size in bytes */
	char _f[20-2*sizeof(long)-sizeof(int)];
};
	

On a 32-bit platform, this structure is 64 bytes long; on a 64-bit platform, this structure is 112 bytes long.

For most computers, long is 32-bit wide on 32-bit platform and 64-bit wide on 64-bit platform; int is 32-bit wide on both platform; short is 16-bit wide on both platform.

On 64-bit platform, padding may be added after procs to ensure long totalhigh is aligned to be readable.

In some case, the structure size may differ on different platforms, even both platforms are 64-bit, because there is no universal agreement for width of int and long. Personally, I would perfer use data type with defined width, such as int32_t and uint64_t.

To save storage space, and to increase the performance of this record program, I decided to minimize the data structure size:


typedef struct {
	uint16_t data_wp, data_total;
	uint16_t ram_total, swap_total;
	uint16_t data_blocksize;
	uint16_t _pad[3];
} header_t;

typedef struct {
	uint32_t uptime;
	uint16_t process;
	uint8_t load1, load5;
	uint16_t ram_free, ram_shared, ram_buffer, swap_free;
} data_t;
	

For data related to the system information:

For data related to the record file:

In this way, the header is 16 bytes long, and each data entry is 16 bytes long. Up to 255 data entries can be fit into a 4kB space; up to 4095 data entries can be fit into a 64kB space.

We want to use as few disk clusters and memory pages as possible. This not only saves RAM and disk usage, but also speeds up file open, memory write and file save when we record the system information. We do not want the system information logging program to eat too much system resources.

4kB is the magic number. It is the memory page size (without using a huge page), meaning it consumes the minimum amount of RAM. Even if we use a smaller number, the system must allocate at least this much RAM to our program because the virtual memory is allocated by memory page. Furthermore, it is the disk cluster size, meaning it consumes a single disk IO operation to read or write the record file. Using a smaller number doesn't save IO (instead it may increase IO, because we need to read-modify-write the entire cluster), using a larger number requires extra IOs.

Record Frequency and Length

Following is a worksheet of record file size (number of entries), record frequency vs record duration:

Record File Worksheet
File sizeEntriesFrequencyDuration
4kB2551 second4 minutes 15 seconds
4kB2556 minutes25 hours 30 minutes
64kB40956 minutes17 days
64kB2550.5 hours85 days
1MB655351 hours7.5 years

Record Program

We will write a program that records the current system information into the record file in C language.

This program should be periodically executed by a system timer, such as systemd-timer or cronjob. This simplifies the program and provides more flexibility compared to writing a long-running background daemon that periodically records the system information by itself.

Data Structure


#define FILE_SIZE 4096 // 65536 // Better align with memory page size and disk cluster size
#define DATA_COUNT 255 // 4095 // Cross check with FILE_SIZE, FILE_SIZE = 16 + 16 * DATA_UNITSIZE
#define DATA_UNITSIZE ((uint64_t)(64 * 4096))

typedef struct {
	uint16_t data_wp, data_total;
	uint16_t ram_total, swap_total;
	uint16_t data_blocksize;
	uint16_t _pad[3];
} header_t;

typedef struct {
	uint32_t uptime;
	uint16_t process;
	uint8_t load1, load5;
	uint16_t ram_free, ram_shared, ram_buffer, swap_free;
} data_t;

typedef struct {
	header_t header;
	data_t data[DATA_COUNT];
} buffer_t;

int main(int argc, char** argv) {
	static_assert(sizeof(data_t) == 16, "Data size must be 16");
	static_assert(sizeof(header_t) == 16, "Header size must be 16");
	static_assert(sizeof(buffer_t) == FILE_SIZE, "Buffer size mismatch with data count");
	static_assert(DATA_UNITSIZE / 4096 <= 65535, "Data unit cannot fit into uint16_t");
		

Define the record file data structure. The record file can be represented by buffer_t, it includes one header header_t header, followed by an array of data entries data_t data[DATA_COUNT].

The header includes:

  • Number of data entries.
  • Next data entries to write (index).
  • Total RAM size (unit: block).
  • Total swap size (unit: block).
  • RAM and swap block size (unit: 4kB).
  • Padding to 16 bytes.

Each data entry contains:

  • Uptime (seconds since boot).
  • Process count.
  • 1-minute and 5-minute average CPU loads (out of 250).
  • Free, shared and buffer RAM, free swap (unit: block).

At compile time, we will ensure that:

  • Both the header and the data entry size is 16 bytes long.

  • Record file size FILE_SIZE and number of data entries DATA_COUNT matches (FILE_SIZE = 16 + 16 * FILE_SIZE). This is to cross check when I make any change to the size. It ensures I know I am making the size aligned.

  • RAM and swap block size DATA_UNITSIZE (in the unit of 4kB) can be fit into uint16_t. Max allowed block size is 262140MB (65535 * 4kB).

Measure the Current System Information


	struct sysinfo sys_info;
	if (sysinfo(&sys_info)) {
		fprintf(stderr, "Cannot get sysinfo err: %d.\n", errno);
		return -1;
	}
	uint64_t ram_total = (uint64_t)(sys_info.totalram) * sys_info.mem_unit, ram_write = ram_total / DATA_UNITSIZE;
	uint64_t swap_total = (uint64_t)(sys_info.totalswap) * sys_info.mem_unit, swap_write = swap_total / DATA_UNITSIZE;
	if ( ram_write > (uint16_t)-1 || swap_write > (uint16_t)-1 ) {
		fprintf(stderr, "DATA_UNITSIZE (%"PRIu64") too small, allow max %"PRIu64" - RAM needs %"PRIu64" Swap needs %"PRIu64"\n", DATA_UNITSIZE, (uint64_t)(DATA_UNITSIZE * (uint16_t)-1), ram_total, swap_total);
		return -1;
	}
		

Use system call sysinfo() to get the current system information.

Perform a run-time check for the RAM and swap size space requirement. If the total RAM size or the total swap size (in the unit of block) cannot be fit into uint16_t, this program will halt.

Because the total RAM will always be greater than the free, shared and buffer RAM, and the total swap will always be greater than the free swap, we only need to check the total RAM and total swap. Furthermore, the total RAM and total swap size will never change, if the total RAM size or total swap size cannot be fit into uint16_t, this issue will be exposed in the first time we run this program. We can tweek the block size then.

Open Record File


	if (argc != 2) {
		fprintf(stderr, "Buffer filename not provided.\n");
		return -1;
	}
	int file = open(argv[1],
		O_RDWR | O_CREAT,
		S_IRUSR|S_IWUSR | S_IRGRP | S_IROTH
	);
	if (file < 0) {
		fprintf(stderr, "Cannot open file err: %d.\n", errno);
		return -1;
	}
	if (flock(file, LOCK_EX) < 0) {
		fprintf(stderr, "Cannot lock file err: %d.\n", errno);
		return -1;
	}
	if (ftruncate(file, sizeof(buffer_t)) < 0) {
		fprintf(stderr, "Cannot allocate buffer file err: %d.\n", errno);
		return -1;
	}

	buffer_t* buffer = mmap(NULL, sizeof(buffer_t), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, file, 0);
		

For better performance, we will use file descriptor (int file = open()) instead of file pointer (FILE* fp = fopen()) to access the record file.

We open this file in read-write mode. If this file does not exist (first run), create it. Allow read and write for self, allow read only for group and others.

We must give read access to others if we want to allow another program to read the system information record.

Obtain an advisory write file lock on the record file. This prevents us from writing the file while others are reading, and others from reading this file while we are writing.

Truncate (also means expand) this file to the desired length. This is mainly used for newly created record files. Because our record file has a fixed length FILE_SIZE, it is better to allocate this much size at the beginning, in one shot. Therefore, we do not need to dynamically allocate space as we write, which is slow and has a higher risk of failure.

Map the record file into memory in read-write mode (same as when we open the record file). Make it shared, so any change made by our program can be reflected outside. Populate the buffer to prevent future page fault.

A defined and fixed buffer in memory is always the safest place to work.

Generate File Header (First Run)


	header_t* header = &(buffer->header);

	if (header->data_total == 0) {
		fprintf(stderr, "Create file.\n");
		header->data_wp		= 0;
		header->data_total	= DATA_COUNT;
		header->data_blocksize	= DATA_UNITSIZE / 4096; // of 4k page
		header->ram_total	= ram_total / DATA_UNITSIZE;
		header->swap_total	= swap_total / DATA_UNITSIZE;
	}
		

The record file does not exist if it is the first time running this program. In this case, we create the record file and populate it to desired size in the previous step. When populating the newly created record file, zero if filled. We can check whether the record file is newly created by reading the header portion. I decided to read the number of entries header->data_total in the header. If it is zero, the record file is newly created.

For a newly created record file, we need to populate the following information:

  • The write index. At initialization, we will use 0, which means the next data entry to write is in slot 0.

  • Number of data entries.

  • RAM and swap block size, in the unit of 4kB.

  • Total RAM and swap size, in the unit of block.

We use uint16_t to record this information to save space. We must cast the data into larger data types, such as uint64_t while processing them; otherwise, the C compiler will use the original data type, which is uint16_t while processing them, risking overflow. We only truncate them to uint16_t when writing them to memory.

Write the System Information


	float load1 = 250.0f * sys_info.loads[0] / get_nprocs() / (1<<SI_LOAD_SHIFT); // out of 250
	float load5 = 250.0f * sys_info.loads[1] / get_nprocs() / (1<<SI_LOAD_SHIFT); // out of 250

	data_t* data = &(buffer->data[header->data_wp]);
	data->uptime		= sys_info.uptime; // Need more than tens of years uptime?
	data->process		= sys_info.procs > (uint16_t)-1 ? (uint16_t)-1 : sys_info.procs;
	data->load1		= load1 > 255 ? 255 : load1;
	data->load5		= load5 > 255 ? 255 : load5;
	data->ram_free		= (uint64_t)(sys_info.freeram) * sys_info.mem_unit / DATA_UNITSIZE;
	data->ram_shared	= (uint64_t)(sys_info.sharedram) * sys_info.mem_unit / DATA_UNITSIZE;
	data->ram_buffer	= (uint64_t)(sys_info.bufferram) * sys_info.mem_unit / DATA_UNITSIZE;
	data->swap_free		= (uint64_t)(sys_info.freeswap) * sys_info.mem_unit / DATA_UNITSIZE;

	header->data_wp = header->data_wp == DATA_COUNT - 1 ? 0 : header->data_wp + 1;
		

Use the write index &(buffer->data[header->data_wp]) to create a pointer to the entry to write.

  • CPU Load

    The sysinfo() system call returns the system load in unsigned long.

    This number contains the sum of load of all cores; hence, it may be greater than 100% on a multi-core platform. To find the average per core load, divide this number by number of cores, by using get_nprocs().

    The range of the CPU load can be represented by 1<<SI_LOAD_SHIFT. We will divide the per core CPU load by this number. This results in a unit per core load, its range between 0 to 1.

    Since we are using uint8_t to record the CPU load, which has a range of 0 to 255. We multiply the unit per core load by 250, which gives us a number between 0 to 250, that is a resolution of 0.4%. To prevent any overflow and to reduce rounding issue during the process, we will need to use floating point numbers float, single precision is sufficient.

    When write the CPU load to memory in uint8_t, clamp it to 0 to 255, just in case CPU load goes above 100%.

  • Uptime

    Truncate the uptime to 32-bit unsigned integer. This allows up to 136 years of non-stoping running, which is sufficient. Nobody runs a server this long without a reboot.

  • Process Count

    If sysinfo() system call reports more than 65535 processes, clamp it to 65535. This is not likely to happen, because the struct sysinfo also uses unsigned short for this information, which is 16-bit wide.

  • RAM and Swap

    Record the free RAM, shared RAM, buffer RAM and free swap, in units of block.

At the end, increment the write index. Wrap the index back to 0 if it reaches the end of the ring buffer.

Following code is added during debugging to inspect the recorded data, note the first run creates the record file:


	fprintf(stdout, "data_total %"PRIu32" Write pointer to %"PRIu32" %p %p.\n", header->data_total, header->data_wp, buffer, data);
	fprintf(stdout, "uptime %"PRIu32"\n", data->uptime);
	fprintf(stdout, "- process %"PRIu16"\n", data->process);
	fprintf(stdout, "- load %"PRIu8" %"PRIu8"\n", data->load1, data->load5);
	fprintf(stdout, "ram_total %"PRIu16"\n", header->ram_total);
	fprintf(stdout, "- ram_free %"PRIu16"\n", data->ram_free);
	fprintf(stdout, "- ram_shared %"PRIu16"\n", data->ram_shared);
	fprintf(stdout, "- ram_buffer %"PRIu16"\n", data->ram_buffer);
	fprintf(stdout, "swap_total %"PRIu16"\n", header->swap_total);
	fprintf(stdout, "- swap_free %"PRIu16"\n", data->swap_free);
	fprintf(stdout, "x %"PRIu16" x 4kB\n", header->data_blocksize);
		

./sysinfo_log4k.elf 4k.bin
Create file.
data_total 255 Write pointer to 0 0x74d6e5c95000 0x74d6e5c95010.
uptime 2205178
- process 189
- load 11 24
ram_total 3825
- ram_free 406
- ram_shared 23
- ram_buffer 285
swap_total 40895
- swap_free 40446
x 64 x 4kB

./sysinfo_log4k.elf 4k.bin
data_total 255 Write pointer to 1 0x7177825c7000 0x7177825c7020.
uptime 2205190
- process 189
- load 48 31
ram_total 3825
- ram_free 406
- ram_shared 23
- ram_buffer 285
swap_total 40895
- swap_free 40446
x 64 x 4kB
		

Sync and Close the Record File


	int pagesize = getpagesize();
	void* data_page = (void*)((uint64_t)data & ~(uint64_t)(pagesize - 1));
	if (data_page == buffer) {
		msync(data_page, pagesize, MS_ASYNC);
	}
	msync(buffer, pagesize, MS_SYNC);
	
	munmap(buffer, sizeof(buffer_t));
	flock(file, LOCK_UN);
	close(file);
	return 0;
}
		

Sync the change in memory to file. Unmap the file from memory, release the lock and close the file descriptor.

Multi-page Sync

Before we discuss the memory map sync, we should be clear that:

  • We can safely assume that the memory page size is power of 2 and at least 4096 bytes long. In nearly all cases, this will be 4096.

  • When we sync the record file from the memory map to the underlying file system, we must sync it as whole pages. If we only change a portion of a page, we must sync the entire page.

  • We define the record file size to be a multiple of pages. For example, 4kB, 64kB.

  • Both the header and the data entries are 16 bytes. In other words, page size is multiple of header / data size. This means, a header or a data can only be located in a specific page; it is impossible to have a portion of a header / data in one page and the remaining in another page.

After we record the system information, there are two changes:

  • The write index in the header: buffer->header.data_wp.

  • The modified data entry: buffer->data[buffer->header.data_wp].

If the header and the modified data entry fills in the same page, we only need to sync that page. Since the header is at the beginning of the buffer, header is same as buffer, and header must be page aligned (buffer returned by mmap()).

If the header and the modified data entry are located in different pages, we will need to sync both pages. We will sync the data page asynchronously (no wait), followed by the header page synchronously (wait until file write finished).

This order is important, because if the file lock fails (note the file lock is an advisory lock, another process may ignore this lock when reading, or in some cases, such as remote file access, it may lack file lock support), the reading process can treat the data entry pointed by the write index as dirty, see the following table:

Lock Fail-safe Ring Buffer
OffsetContent
Headerdata_wp = 3
Entry 0Record [t-3]
Entry 1Record [t-2]
Entry 2Most recent record
Entry 3Dirty (write in progress / to be write soon / just finish writing)
Entry 4Oldest record (danger if reading is slower than sampling speed)
......
Entry DATA_COUNT-1Record [t-4]
  • If a read happens before the data page and header page write - Dirty entry is the oldest data.

  • If a read happens after the data page write but before the header page write - Dirty entry is the most recent data. We will need to wait to the next read to consider this data valid.

  • If a read happens after the data page and header page write - Dirty entry moves to next entry, to the now-oldest data. The before-dirty entry with the most recent data can be consider valid now.

It is possible to sync the entire record file in one shot even if it is greater than 4kB. Functionality wise, it is fine; performance wise, this causes pages that are not modified being synced redundantly. The impact becomes significant when the record file is very large. However, it is worth mentioning that two separate msynch() system calls for two pages involve two context switches, and context switches are expensive.

Compile and Test Run


#define FILE_SIZE 4096
#define DATA_COUNT 255
gcc a.c -O2 -o sysinfo_log4k.elf

#define FILE_SIZE 65536
#define DATA_COUNT 4095
gcc a.c -O2 -o sysinfo_log64k.elf
	

Compile this C language program using GCC. I decided to make two versions, one creates a 4kB record file with 255 data entries, another creates a 64kB record file with 4095 data entries.

Following figure shows the record file's content after running this program for several times:

Record File Content
Record File Content
Record File Description
Address FEDC BA98 7654 3210
Header Pad Block sizeTotal SwapTotal RAM Number of entriesWrite index
0x000 Pad0x0040
64 pages
256kB
0x9FBF
40895 blocks
10,469,120kB
0x0EF1
3825 blocks
979,200kB
0x00FF
255
0x000F
15
Data entries Free swapBuffer RAMShared RAMFree RAM 5-minute load avg1-minute load avg Process countUptime
0x010 0x9DFE
40446 blocks
10,354,176kB
0x011D
285 blocks
72,960kB
0x0017
23 blocks
4,352kB
0x0E96
406 blocks
103,936kB
0x18
24
9.6%
0x0B
11
4.4%
0x00BD
189
0x0021A5FA
2205178s
25.5 days
0x100 0x00000000000000000000000000000000
Empty, to be write
0xFF0 0x00000000000000000000000000000000
Last entry in the ring buffer, index = 255

System Timer

I decided to use systemd-timer instead of cronjob because systemd-timer allows up to second-level frequency, whereas cronjob only allows minute-level frequency.

I created 3 timer jobs:

Short Period Real-time Record

This job records the real-time system information. It helps discover a real-time load spike.

For example, I can use this job to monitor the load while stress testing the server. Any system load increase reflects immediately (in fact, in the next sampling point, that is in the next few seconds).

My first trial is to record every second, for 255 entries in the record file, which gives around 4 minutes of real-time system information. However, the numbers returned by sysinfo() seem not changed between some records. Maybe the underlying implementation refreshes at a frequency lower than 1Hz. Therefore, I decided to slow down the sampling rate to 2Hz.

The following timer and service files are used to define this job:


// sysinfo_log_2s.timer
[Unit]
Description=Record sysinfo every 2 seconds
StartLimitBurst=3
StartLimitIntervalSec=1
[Timer]
OnBootSec=2s
OnUnitActiveSec=2s
AccuracySec=1s
[Install]
WantedBy=timers.target

// sysinfo_log_2s.service
[Unit]
Description=Record sysinfo every 2 seconds
StartLimitBurst=3
StartLimitIntervalSec=1
[Service]
ExecStart=/path/to/executable/sysinfo_log4k.elf /path/to/recordfile/sysinfo_log_2s.bin
	

Day-length Record

This job records the system information over a 24-hour period. It helps discover the trend over a day, for example, day time vs nigh time.

This job will record every 6 minutes, for 255 entries in the record file, which gives 25.5 hours of system information.

The following timer and service files are used to define this job:


// sysinfo_log_6m.timer
[Unit]
Description=Record sysinfo every 6 minutes
StartLimitBurst=3
StartLimitIntervalSec=1
[Timer]
OnBootSec=2s
OnUnitActiveSec=6m
AccuracySec=1m
[Install]
WantedBy=timers.target

// sysinfo_log_6m.service
[Unit]
Description=Record sysinfo every 6 minutes
StartLimitBurst=3
StartLimitIntervalSec=1
[Service]
ExecStart=/path/to/executable/sysinfo_log4k.elf /path/to/recordfile/sysinfo_log_6m.bin
	

Month-length Record

This job records the system information over a 1-month period. It helps discover the long-term trend.

This job will record every 10 minutes, for 4095 entries in the record file, which gives 28.4 days of system information.

The following timer and service files are used to define this job:


// sysinfo_log_6m.timer
[Unit]
Description=Record sysinfo every 10 minutes
StartLimitBurst=3
StartLimitIntervalSec=1
[Timer]
OnBootSec=2s
OnUnitActiveSec=10m
AccuracySec=1m
[Install]
WantedBy=timers.target

// sysinfo_log_6m.service
[Unit]
Description=Record sysinfo every 10 minutes
StartLimitBurst=3
StartLimitIntervalSec=1
[Service]
ExecStart=/path/to/executable/sysinfo_log64k.elf /path/to/recordfile/sysinfo_log_10m.bin
	

Enable Systemd Timer

The following command is used to enable the timers:


systemctl enable --now sysinfo_log_2s.timer
systemctl enable --now sysinfo_log_6m.timer
systemctl enable --now sysinfo_log_10m.timer
	

To list all timers:


systemctl list-timers
NEXT                            LEFT LAST                               PASSED UNIT                           ACTIVATES
Tue 2026-10-27 18:50:24 UTC    799ms Tue 2026-10-27 18:50:22 UTC        1s ago sysinfo_log_2s.timer           sysinfo_log_2s.service
Tue 2026-10-27 18:52:30 UTC  2min 5s Tue 2026-10-27 18:46:30 UTC  3min 54s ago sysinfo_log_6m.timer           sysinfo_log_6m.service
Tue 2026-10-27 18:55:58 UTC     5min Tue 2026-10-27 18:45:58 UTC  4min 25s ago sysinfo_log_10m.timer          sysinfo_log_10m.service
	

Read (and Render) the Record

The record program executables and the record files are saved in a web inaccessible directory (for example: /home/myhome), but with mod xx4 (read-only for anyone). Soft symlinks are created in a web accessible directory (for example: /var/www/html). These symlinks link to the record files. Therefore, the web interface (run by user www-data) cannot read or modify the record program executables and the orginal record files; instead, only the copies of the record files is accessible. If the web interface doomed, it cannot harm the server by using our record program. (For example, replace the record program by a malicious program.)

The web server does not perform any data manipulation over the record file. Instead, the record file is send to client as a raw binary blob.

It is worth mentioning that, there should some kind of access control on the web server. For example, download is allowed to admin users only. The access control implementation at the web server is out of this article's scope. Furthermore, the web server should respect the file lock.

After the record file is downloaded to the client side, it can be decoded and rendered using Javascript.

Render the record on webpage
Render the record on webpage

Helper Functions


const _ = (s, p = document) => p.querySelector(s);
const __ = (s, p = document) => p.querySelectorAll(s);
const dom = j => {
	if (typeof j == 'string') {
		const x = (new DOMParser()).parseFromString(j, 'text/html').body;
		if (_('parsererror', x))
			throw new Error(_('parsererror', x).textContent);
		if (_('sourcetext', x))
			throw new Error(_('sourcetext', x).textContent);
		return x;
	} else {
		let tag = 'div', children = [];
		if ('_' in j) { tag = j._; delete j._; }
		if ('children' in j) { children = j.children; delete j.children; }
		const x =  Object.assign(document.createElement(tag), j);
		children.map(y => {if (y) x.appendChild(dom(y))});
		return x;
	}
};

const svg_new = (width, height) => {
	const svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
	svg.setAttribute('preserveAspectRatio', 'none');
	svg.setAttribute('viewBox', '0 0 '+width+ ' '+height);
	return svg;
};
const poly_new = (type, points, color='transparent', fill='transparent') => {
	const poly = document.createElementNS('http://www.w3.org/2000/svg', type);
	poly.setAttribute('points', points);
	poly.setAttribute('stroke', color);
	poly.setAttribute('fill', fill);
	return poly;
};

class MaxMin {
	constructor(min, max) {
		this.min = min;
		this.max = max;
	}
	add(value) {
		this.min = value < this.min ? value : this.min;
		this.max = value > this.max ? value : this.max;
	}
}

const uptimeHuman = second => {
	const day = Math.floor(second / 3600 / 24);
	second -= day * 3600 * 24;
	const hour = Math.floor(second / 3600);
	second -= hour * 3600;
	const minute = Math.floor(second / 60);
	second -= minute * 60;
	return [day+'d'+hour+'h'+minute+'m'+second+'s', day, hour, minute, second];
};
		

Helper functions:

  • _() and __() - Query selector.

  • dom() - Create DOM tree.

  • svg_new() and poly_new() - Create SVG and SVG polygon / polyline.

  • MaxMin - Record the min and max of a set of numbers.

  • uptimeHuman() - Convert seconds to human-readable day-hour-minute-second format.

DOM


// HTML
<div id="waveform_text"><div id="waveform_text_overview"></div><div id="waveform_text_pointer"></div></div>
<svg id="waveform" viewBox="0 0 100 100" preserveAspectRatio="none" stroke-width="0.15"></svg>
<style>#waveform_text { position: absolute; padding: 0 1ch; background: rgba(255,255,255,0.25); } #waveform_text:hover { background: rgba(255,255,255,0.85); } #waveform { width: 100%; height: 400px; fill: transparent; } #waveform>* { width: 100%; height: 100%; }</style>

// JS
let sysinfo_data = [];
let sysinfo_pointer_svg = svg_new(2500, 100);
		

Create HTML elements:

  • div #waveform_text - Print the system information. The first child is used to print the trend information, the second child is used to print the interactive dashboard (information at selected time).

  • #waveform - Draw the trend.

Create two global variables:

  • sysinfo_data - Buffer to store the decoded record file received from the server.

  • #waveform - DOM to draw the interactive dashboard (a cursor).

Trend

Fetch the Record File


const sysinfo_log = async (filename) => { try {
	const file = new Uint16Array(( await (( await fetch('/sysinfo_log_' + filename + '.bin') ).bytes()) ).buffer);
		

Fetch the record file from the server. Since most information is saved using uint16_t in the record file, we will use Uint16Array to store them in Javascript.

Process the Header


	const data_blocksize = file[4] * 4096;
	const svg_width = file[1] - 1;
	const header = {
		data_wp:	file[0],
		data_total:	file[1],
		ram_total:	file[2] * data_blocksize,
		swap_total:	file[3] * data_blocksize
	};

	let path_cpu = '', maxmin_cpu = new MaxMin(100, 0);
	let path_cpu5 = '', maxmin_cpu5 = new MaxMin(100, 0);
	let path_ramFree = '', maxmin_ramFree = new MaxMin(header.ram_total, 0);
	let path_ramShared = '', maxmin_ramShared = new MaxMin(header.ram_total, 0);
	let path_ramBuffer = '', maxmin_ramBuffer = new MaxMin(header.ram_total, 0);
	let path_swapFree = '', maxmin_swapFree = new MaxMin(header.swap_total, 0);
	let maxmin_ramUsed = new MaxMin(header.ram_total, 0);
	let maxmin_swapUsed = new MaxMin(header.swap_total, 0);
		

Extract the header information:

  • data_wp - Write index of the ring buffer.

  • data_total - Total number of data entries in the ring buffer.

  • ram_total - Total RAM size, convert it from block size to byte size.

  • swap_total - Total swap size, convert it from block size to byte size.

  • data_blocksize - RAM and swap block size, convert it from page size to byte size.

Storage size is no longer a concern. Use raw size at client-side.

Process the Data Entries


	sysinfo_data = [];
	for (let i = 0; i < header.block_total; i++) {
		let ptr = (i + header.block_wp) % header.block_total;
		const ram_free = file[8 + 8 * ptr + 4] * data_blocksize;
		const ram_shared = file[8 + 8 * ptr + 5] * data_blocksize;
		const ram_buffer = file[8 + 8 * ptr + 6] * data_blocksize;
		const swap_free = file[8 + 8 * ptr + 7] * data_blocksize;
		const d = {
			uptime:		(file[8 + 8 * ptr + 1] << 16) | file[8 + 8 * ptr + 0],
			process:	file[8 + 8 * ptr + 2],
			load:		(file[8 + 8 * ptr + 3] & 0xFF) / 2.5, // Server gives uint8_t[2], range 0 - 250
			load5:		(file[8 + 8 * ptr + 3] & 0xFF00) / (2.5 * 256),
			ram_free:	ram_free,
			ram_shared:	ram_shared,
			ram_buffer:	ram_buffer,
			swap_free:	swap_free,
			ram_used:	header.ram_total - ram_free - ram_shared - ram_buffer,
			swap_used:	header.swap_total - swap_free
		};
		sysinfo_data.push(d);

		//max_cpu = d.load > max_cpu ? d.load : max_cpu;
		maxmin_cpu.add(d.load);
		maxmin_cpu5.add(d.load5);
		maxmin_ramFree.add(d.ram_free);
		maxmin_ramShared.add(d.ram_shared);
		maxmin_ramBuffer.add(d.ram_buffer);
		maxmin_swapFree.add(d.swap_free);
		maxmin_ramUsed.add(d.ram_used);
		maxmin_swapUsed.add(d.swap_used);

		path_cpu += ' '+i+','+ (100-d.load);
		path_cpu5 += ' '+i+','+ (100-d.load5);
		path_ramFree += ' '+i+','+ (d.ram_free/data_blocksize); // Reduce svg size, don't use byte number (too large), use block size
		path_ramBuffer += ' '+i+','+ (d.ram_buffer/data_blocksize + d.ram_free/data_blocksize) ;
		path_ramShared += ' '+i+','+ (d.ram_shared/data_blocksize + d.ram_buffer/data_blocksize + d.ram_free/data_blocksize); // Stack display
		path_swapFree += ' '+i+','+ d.swap_free/data_blocksize;
	}
		

Reset the buffer sysinfo_data, we are going to refill it with the record file just downloaded.

Because we are using a ring buffer design, and the write index data_wp is pointed to the next to write entry, which is the oldest (or the dirty one if the lock failed), data_wp+0 will be the oldest data entry, data_wp+1 will be the second oldest data entry, data_wp-1 will be the most recent data entry.

Each data entry includes following information:

  • uptime - 32-bit, concat the lower and higher Uint16Array.

  • process - Process count.

  • load - 1-minute load average, stored in the lower byte, convert it from 0 - 250 range to 0.0 - 100.0 range. Note, Javascript uses float by default.

  • load5 - 5-minute load average, stored in the higher byte, convert it from 0 - 250 range to 0.0 - 100.0 range.

  • ram_free - Free RAM, convert it from block size to byte size.

  • ram_shared - Shared RAM, convert it from block size to byte size.

  • ram_buffer - Buffer RAM, convert it from block size to byte size.

  • swap_free - Free swap, convert it from block size to byte size.

  • ram_used - Used RAM, which is not saved in the record file, but can be calculated by using total RAM size and other RAM size.

  • swap_used - Used swap, which is not saved in the record file, but can be calculated by using total swap size and free swap size.

For each information:

  • Use MaxMin to track its minimum and maximum across the entire record file.

  • Place a way point in the polygon / polyline path. To reduce the SVG size, use block size for RAM and swap, this limits the SVG height to 65k pixels (UINT16_MAX); otherwise, the SVG height can be up to gigapixels.

Print Information


	const data_last = sysinfo_data[sysinfo_data.length-1];
	_('#waveform_text_overview').replaceChildren(
		dom({_: 'p', textContent: 'CPU: Current '+data_last.load+'% ('+maxmin_cpu.min+'% min '+maxmin_cpu.max+'% max) - Avg5 '+data_last.load5+'% ('+maxmin_cpu5.min+'% min '+maxmin_cpu5.max+'% max) - Process '+data_last.process, style: 'color: red'}),
		dom({_: 'p', textContent: 'RAM: Free '+data_last.ram_free/1024/1024+'MB ('+maxmin_ramFree.min/1024/1024+'MB min '+maxmin_ramFree.max/1024/1024+'MB max) - Buffer '+data_last.ram_buffer/1024/1024+'MB ('+maxmin_ramBuffer.min/1024/1024+'MB min '+maxmin_ramBuffer.max/1024/1024+'MB max) - Shared '+data_last.ram_shared/1024/1024+'MB ('+maxmin_ramShared.min/1024/1024+'MB min '+maxmin_ramShared.max/1024/1024+'MB max) - Used '+data_last.ram_used/1024/1024+'MB ('+maxmin_ramUsed.min/1024/1024+'MB min '+maxmin_ramUsed.max/1024/1024+'MB max)', style: 'color: darkgreen'}),
		dom({_: 'p', textContent: 'Swap: Free '+data_last.swap_free/1024/1024+'MB ('+maxmin_swapFree.min/1024/1024+'MB min '+maxmin_swapFree.max/1024/1024+'MB max) - Used '+data_last.swap_used/1024/1024+'MB ('+maxmin_swapUsed.min/1024/1024+'MB min '+maxmin_swapUsed.max/1024/1024+'MB max)', style: 'color: blue'}),
		dom({_: 'p', textContent: 'Uptime '+uptimeHuman(data_last.uptime)[0]+' - '+header.block_total+' samples'}),
	);
		

Print the current (most recent), minimum and maximum of the information.

Draw SVG


	const svg_cpu = svg_new(svg_width, 100);
	svg_cpu.appendChild(poly_new('polyline', path_cpu, 'rgb(255, 0, 0)'));
	svg_cpu.appendChild(poly_new('polyline', path_cpu5, 'rgb(127, 63, 63)'));

	const svg_ram = svg_new(svg_width, header.ram_total/data_blocksize);
	svg_ram.appendChild(poly_new('polygon', svg_width+',0 0,0 0,'+header.ram_total/data_blocksize+' '+svg_width+','+header.ram_total/data_blocksize, '#030', '#030'));
	svg_ram.appendChild(poly_new('polygon', svg_width+',0 0,0' + path_ramShared, 'transparent', 'darkgreen'));
	svg_ram.appendChild(poly_new('polygon', svg_width+',0 0,0' + path_ramBuffer, 'transparent', 'green'));
	svg_ram.appendChild(poly_new('polygon', svg_width+',0 0,0' + path_ramFree, 'transparent', 'lightgreen'));

	const svg_swap = svg_new(svg_width, header.swap_total/data_blocksize);
	svg_swap.appendChild(poly_new('polyline', path_swapFree, 'rgb(0, 0, 255)'));

	const svg_grid = svg_new(svg_width, 100);
	for (let i = 1; i < 10; i++) {
		const y = 10 * i;
		svg_grid.appendChild(poly_new('polyline', svg_width+','+y+' 0,'+y, i == 5 ? 'rgba(128,128,128,0.75)' : 'rgba(128,128,128,0.45)'));
	}

	sysinfo_pointer_svg = svg_new(svg_width, 100);

	_('#waveform').replaceChildren(svg_ram, svg_grid, svg_cpu, svg_swap, sysinfo_pointer_svg);
	_('#waveform').dispatchEvent(new Event('mousemove'));
} catch (e) { dialog_error(0, 'System info log failed: ' + e); throw e; } };
		

Use polyline to draw the CPU load (red) and swap usage (blue) trend.

Use polygon to draw the RAM usage (green) trend.

At the end, draw a grid (grey) to help reading the trend. Each grid is 10% high.

Since the SVG data changed, fire a event to the interactive dashboard to force it to refresh.

This task can be fired periodically using setInterval() to refresh the trend SVG.

Interactive Dashboard


_('#waveform').onmousemove = event => {
	const cursor = event.layerX ?? _('#waveform').dataset.cursor ?? 0;
	_('#waveform').dataset.cursor = cursor;
	const x = cursor / _('#waveform').getBoundingClientRect().width;

	const data_ptr = sysinfo_data[Math.floor(x * sysinfo_data.length)];
	_('#waveform_text_pointer').replaceChildren(
		dom({_: 'span', textContent: 'CPU: '+data_ptr.load+'% - ('+data_ptr.load5+'%) - Process '+data_ptr.process, style: 'color: red'}),
		dom({_: 'span', textContent: ' | RAM: Free '+data_ptr.ram_free/1024/1024+'MB - Buffer '+data_ptr.ram_buffer/1024/1024+'MB - Shared '+data_ptr.ram_shared/1024/1024+'MB - Used '+data_ptr.ram_used/1024/1024+'MB', style: 'color: darkgreen'}),
		dom({_: 'span', textContent: ' | Swap: Free '+data_ptr.swap_free/1024/1024+'MB - Used '+data_ptr.swap_used/1024/1024+'MB', style: 'color: blue'}),
		dom({_: 'span', textContent: ' | Uptime '+uptimeHuman(data_ptr.uptime)[0]+' ('+uptimeHuman(sysinfo_data[sysinfo_data.length-1].uptime-data_ptr.uptime)[0]+' ago)'}),
	);

	const svg_width = sysinfo_pointer_svg.viewBox.baseVal.width, svg_height = sysinfo_pointer_svg.viewBox.baseVal.height;
	const poly_x = x * svg_width, poly_y = svg_height;
	sysinfo_pointer_svg.replaceChildren(poly_new('polyline', poly_x+',0 '+poly_x+','+poly_y, '#FFF'));
};
		

When the cursor moves on the trend SVG, read the information from buffer sysinfo_data for that specific position (time), base on the location of the cursor.

Print that information and draw the cursor.