Monday, October 10, 2011

Changing linux I/O Scheduler on-line



When performance tuning any modern Linux system, selecting a proper I/O scheduler is an important but often under looked optimization. Linux comes configured with the cfq (Completely Fair Queuing) scheduler which has been optimized for best performance over a wide variety of tasks. If you are running specific applications such as mysql or are running on more advanced hardware such as powerful raid controllers or solid state drives, the cfq scheduler may not be providing you the best performance possible.
Linux currently ships with four different I/O schedulers. They are: deadline, noop, anticipatory, and cfq. There are many differences between these scheduling algorithms, here is a brief outline of each:
CFQ: This is the default algorithm in most Linux distributions. It attempts to distribute all I/O bandwidth evenly among all processes requesting I/O. It is ideal for most purposes.
NOOP: The noop algorithm attempts to use as little cpu as possible. It acts as a basic FIFO queue expecting the hardware controller to handle the performance operations of the requests.
Anticipatory: This algorithm attempts to reorder all disk I/O operations to optimize disk seeks. It is designed to increase performance on systems that have slow disks.
Deadline: This scheduling algorithm places I/O requests in a priority queue so each is guaranteed to be ran within a certain time. It is often used in real-time operating systems.
Since the 2.6.10 kernel you were given the option to easily adjust the disk scheduler on the fly without a reboot. You can see which scheduler you are currently running by typing:
1
2
# cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
In this case sda is the block device in question., here we can see we are currently running the cfq scheduler. You can easily change the scheduler you are using by echoing the scheduler you wish to use into it:
1
# echo noop > /sys/block/sda/queue/scheduler
We now are using the noop scheduler indicated by brackets:
1
2
# cat /sys/block/sda/queue/scheduler
[noop] anticipatory deadline cfq
Changing schedulers on the fly allows you to test and benchmark the algorithms for your specific application .Once the change is issued, any current I/O operations will be executed before the new scheduler goes into effect, so the change will not be instantaneous. Also remember that once one is set and performs to your liking, be sure to set the change to be applied on subsequent reboots.
There are cases where cfq may not be the best scheduler for your system or application. An example is if you are running a raid disk array with a caching raid controller. In this instance many reads and writes will be cached by the disk controller to be executed based on its own scheduling algorithm. Scheduling algorithms such as noop may offer you better performance by not wasting host os cpu cycles reordering I/O operations that do not offer the disk subsystem any benefit. Solid state drives are another case where any algorithm that reorders disk operations may not offer any benefit since random access times are identical to sequential access times. It is often recommend to use noop or deadline on any SSD drive.
There is usually no definitive answer to which algorithm to use, so benchmarking each one will be your best option. Performance results will depend greatly on the applications that are running, the type and speed of hard drives being used, the disk controllers or raid cards / level in use, along with the overall system load and amount of disk I/O. Please also keep in mind that each of these algorithms has many additional configurable options to further tweak performance to your needs depending on how much time you wish to spend on them.


No comments: