Motivation

When using physical SSDs in a ZFS configuration, write performance degrades over time when the storage
system is used for a long time. This problem occurs due to the nature of the SSD. Depending on the type
of SSD and how long that drive is loaded, the negative impact on write performance can be significant.
To solve this problem, ZFS supports the TRIM feature, which helps prevent performance degradation of SSDs.

There is also a need to periodically check the integrity of the data in EVE in the ZFS configuration using
the Scrub function. This verification is necessary in order to detect and prevent errors before they cause
hardware failure or data corruption in EVE, which is not acceptable to us.

The requirement to be able to maintain the storage system and periodically check it for errors comes from
EVE customers and also is in line with the requirements for improving the storage system in EVE.
And this proposal is to add the ability to instantly launch or schedule these commands on the controller side.
Since commands can negatively affect performance in the course of their work, it is proposed to add
the ability for users to set the time for their launch themselves through the controller.

TRIM

TRIM is a command which allows the filesystem to notify the storage device which blocks are no longer in use.
Сan read more about what TRIM is on the wiki. 

...

An example of the positive impact of the TRIM team was presented at the OpenZfs conference in 2019 in the
presentation of this functionality:

SCRUB

ZFS checksums every block of data that is written to disk, and compares this checksum when the data is read back
into memory. If the checksums don’t match we know the data was changed by something other than ZFS
(assuming a ZFS bug isn’t the culprit), and assuming we are using ZFS to RAID protect the storage the issue
will be automatically fixed for us.

...

In this proposal, it is proposed to add functionality that allows you to run this function from the controller,
including according to the schedule.

Weak sides

The weak side is the fact that performance decreases when executing these commands since for their execution they,
like everyone else, require resources.

A study was also conducted to determine the negative impact on performance when reading and writing at the time
of executing these commands. And it turned out that both commands in the process of their execution show a decrease
in performance on the device by at least half when reading from disk or zpool. And only the Scrub command has
a negative impact on system performance when writing. The scrub operation consumes all I/O resources on the system.

There are levers in ZFS to limit these commands in system resources, but another big new study is needed to test them. 
After which it will be possible to think about improving this functionality in EVE.

EVE API Additions

The StorageCmdConfig message is expected to be received from the controller:

...

// StorageServiceCmd - information about servicing storage systems (Trim/Scrub)
message StorageServiceCmd {
	org.lfedge.eve.config.StorageCmdType cmd_type = 1; // Scrub or Trim
	org.lfedge.eve.config.StorageCmdRunType run_type = 2; // The run instruction (time period)
	uint64 last_update_time = 3; // Time of the last scheduled start check
	uint64 next_run_time = 4; // The time when the command will be run next
	uint64 last_change_cmd_run_type_time = 5; // The time the command was last modified
	repeated uint64 launch_times_history_list = 6; // Stores history of previous runs
}

Reducing performance impact when running TRIM and Scrub

At the moment, there are not many options for these commands to regulate the resources used.
Only those options with default value that affect performance will be described here.

For Scrub:

zfs_vdev_nia_delay=5 (int)

...

Minimum scrub I/O operations active to each device.

For TRIM:

zfs_trim_queue_limit=10 (uint)

...

Minimum trim/discard I/O operations active to each device.

Investigating performance when changing options

TRIM command

The problem with this command is that a significant performance drop (more than 50%) occurs only for
the read performance when it is executed. For this command, after testing and checking, it was determined that changing
only one zfs_trim_queue_limit option to a value of 5 reduces the drawdown to 30-35%.  Changing the rest of
the trim_min_active and trim_max_active commands does not produce a positive effect, since they are already
at their minimum values. The Trim command has almost no effect on write performance (Permissible performance degradation within 5 - 7%).

Recommended value for set in EVE:
zfs_trim_queue_limit=5

Scrub command


The scrub command has a strong impact on both read and write operations. Moreover, if the zpool is large,
it can take a long time, which is a very critical factor for us (performance drop for a long time).

...

zfs_vdev_scrub_max_active=1
zfs_vdev_nia_credit=1


Discussion

• Do we want to be able to pause these commands through the controller?
• The EVE API Additions topic suggested options for time periods to automatically run these commands on a schedule.
  If you have any objections or suggestions, let's discuss them in the comments.

...