Difference between revisions of "StableBit DrivePool - Develop Balancing Plugins"

From Covecube - Wiki
Line 7: Line 7:
 
== Getting Started ==
 
== Getting Started ==
 
* Download the sample balancing plug-in solution below.
 
* Download the sample balancing plug-in solution below.
−
* Extract it and open it in Visual Studio
+
* Extract it.
−
* See the '''README.txt''' for detailed instructions on how to customize and build your own balancing plug-in.
+
* Run '''RunMe.cmd'''.<br />This will initialize the source files for your plug-in and prepare it for being built.
 +
* Open the solution in Visual Studio and Build it.
  
−
'''Download:''' [http://dl.covecube.com/DrivePoolBalancingPlugins/BalancingPlugin/BalancingPlugin_src.1.zip BalancingPlugin_src.1.zip]
+
'''Download:''' [http://dl.covecube.com/DrivePoolBalancingPlugins/BalancingPlugin/BalancingPlugin_src.2.zip BalancingPlugin_src.2.zip]
  
−
'''Revision: 1'''
+
'''Revision: 2'''
  
−
== Distribution ==
+
==Coding the Balancer==
 +
The balancing plug-in itself consists of 3 parts:
 +
 
 +
* '''BalanceSettingsState''' - This holds properties which represent any settings that you would like to expose to the user in the UI.
 +
* '''SettingsUI''' - The WPF user control that exposes the settings as user interface elements.
 +
* '''Balancer''' - The balancing algorithm.
 +
 
 +
You may have noticed that this is the Model-View-Controller pattern, but I won't use this terminology here.
 +
 
 +
==Settings==
 +
First, define the settings that your balancer wants to expose to the user as properties of the '''BalancerSettingsState'''.
 +
Put them all in the 'Settings properties' region.
 +
 
 +
See '''SampleProperty1''' which is already there.
 +
 
 +
Don't forget to call ''OnChanged()'' after every set.
 +
 
 +
The system will Serialize and Deserialize your '''BalancerSettingsState''' for you automatically between restarts
 +
so you don't have to worry about saving anything.
 +
 
 +
==Settings UI==
 +
The settings UI is a WPF control which will be data bound to an instance of '''BalancerSettingsState''' object.
 +
 
 +
Set up any controls that you would like to expose to the user and data bind them to the properties that you've set
 +
up in '''BalancerSettingsState'''.
 +
 
 +
The '''DataContext''' of '''SettingsUI''' will be set to an instance of '''BalancerSettingsState'''.
 +
 
 +
Make use of the '''SettingsSection''' control and the general outline in the sample to provide a consistent look for
 +
your plug-in.
 +
 
 +
===No Settings===
 +
If you will not be exposing any settings to the user, simply delete the '''SettingsUI.xaml''' file.
 +
 
 +
==Balancing Algorithm==
 +
Open '''Balancer.cs''' and write your balancing algorithm in the ''Balance()'' function.
 +
 
 +
You will be passed in an enumeration of '''BalanceStateInfos'''. Each '''BalanceStateInfo''' represents one or more disks part
 +
of the pool that are on the same physical disk.
 +
 
 +
DO NOT do any lengthy work in this function. DrivePool expects this operation to be very quick.
 +
 
 +
If you need to do some external work, such as query WMI or some other time consuming task, you must spawn a new
 +
thread in the constructor and do all the work there.
 +
 
 +
The rest of this document will describe the various operations that your balancing plug-in can perform.
 +
 
 +
===Moving Files Around===
 +
The '''BalanceStateInfo.PoolParts''' property will enumerate each individual pool part that is on the same physical disk.
 +
 
 +
You can examine the various properties of each '''BalanceStateInfo''' in order to discover current file distributions and
 +
call ''MoveFiles()'' to move files around.
 +
 
 +
When requesting to move files you always pass a target ratio. This ratio is always calculated as a ratio of
 +
total disk capacity.
 +
 
 +
<pre>
 +
// Example:
 +
var moveResult = balanceState.MoveFiles(FileTypes.UnprotectedFiles, 0.25);
 +
</pre>
 +
 
 +
This means that you would like un-protected (or un-duplicated) files to use up 25% of this storage-unit.
 +
 
 +
DrivePool will go out and see whether it's possible to satisfy your request given the current file distribution
 +
across all the disks, it will try to satisfy as much of your request as possible and then give you back a moveResult
 +
which will contain information about how much data was actually scheduled to move.
 +
 
 +
You can also pass a MoveOptions instance as the third parameter for more control over the move operation.
 +
 
 +
===Real-time File Placement Limit===
 +
In addition to moving files around you can also set real-time file placement limits like this:
 +
 
 +
<pre>
 +
balanceState.SetLimit(LimitTypes.NoProtectedFiles);
 +
</pre>
 +
 
 +
===Equalizing Pool Parts===
 +
Since you can only move files among the different balance states and not the individual pool parts, it may be
 +
advantageous to be able to equalize the disk space usage on all the pool parts that comprise a balance state.
 +
 
 +
<pre>
 +
// Example:
 +
balanceState.EqualizePoolParts();
 +
</pre>
 +
 
 +
===Forcing an Immediate Balancing Pass===
 +
If your balancer requires the immediate moving of files, regardless of the trigger conditions or the balancing
 +
schedule, you can tell DrivePool that you want to start balancing immediately.
 +
 
 +
<pre>
 +
// Example:
 +
this.Actions.BalanceNow();
 +
</pre>
 +
 
 +
You CANNOT call this from the constructor of the balancer.
 +
 
 +
The user can disable this ability in balancing settings.
 +
 
 +
==How it All Works==
 +
DrivePool keeps track of all file statistics in real-time and will periodically call your ''Balance()'' function
 +
in order to ask your balancer if it wants to modify the current file distribution.
 +
 
 +
DrivePool will do this for all the balancers in the stack, starting with the lowest priority balancer first.
 +
 
 +
At the end of this 'Calculation' pass DrivePool will determine how much data needs to be moved in terms of
 +
bytes and percent of total pool usage. It presents this to the user in the 'Pool condition" bar allowing
 +
the user to start a manual balancing pass at any time.
 +
 
 +
It will also evaluate any automatic balancing triggers and start an automatic balancing pass if required
 +
and possible.
 +
 
 +
The balancing pass will once again ask your plug-in to update the balancing model. It will then start to
 +
satisfy this model using the built-in 'FileMover' which is responsible for cataloging the (possibly)
 +
millions of files and building an optimal move set in order to satisfy the balancing model. This is no
 +
small feat, but nothing that you have to worry about.
 +
 
 +
Once the move set is built then DrivePool goes to work moving the files around using background I/O
 +
priority and a technique that doesn't lock any files.
 +
 
 +
If a system is shut down or the service is stopped while a move is in progress, DrivePool cleanly aborts
 +
the balancing run immediately.
 +
 
 +
If the system crashes while a file move is in progress, the user's data is not affected.
 +
Worst case scenario is that they will find an extra .dptemp file which can be deleted safely since the
 +
source of the move is still intact.
 +
 
 +
==Distribution==
 
If you would like your balancing plug-in posted on the official page, send us a note with your balancing plug-in attached @ http://stablebit.com/contact
 
If you would like your balancing plug-in posted on the official page, send us a note with your balancing plug-in attached @ http://stablebit.com/contact

Revision as of 18:41, 20 November 2012

To install balancing plug-ins, you need DrivePool 1.2.0.6817 or later.

Requirements

Getting Started

  • Download the sample balancing plug-in solution below.
  • Extract it.
  • Run RunMe.cmd.
    This will initialize the source files for your plug-in and prepare it for being built.
  • Open the solution in Visual Studio and Build it.

Download: BalancingPlugin_src.2.zip

Revision: 2

Coding the Balancer

The balancing plug-in itself consists of 3 parts:

  • BalanceSettingsState - This holds properties which represent any settings that you would like to expose to the user in the UI.
  • SettingsUI - The WPF user control that exposes the settings as user interface elements.
  • Balancer - The balancing algorithm.

You may have noticed that this is the Model-View-Controller pattern, but I won't use this terminology here.

Settings

First, define the settings that your balancer wants to expose to the user as properties of the BalancerSettingsState. Put them all in the 'Settings properties' region.

See SampleProperty1 which is already there.

Don't forget to call OnChanged() after every set.

The system will Serialize and Deserialize your BalancerSettingsState for you automatically between restarts so you don't have to worry about saving anything.

Settings UI

The settings UI is a WPF control which will be data bound to an instance of BalancerSettingsState object.

Set up any controls that you would like to expose to the user and data bind them to the properties that you've set up in BalancerSettingsState.

The DataContext of SettingsUI will be set to an instance of BalancerSettingsState.

Make use of the SettingsSection control and the general outline in the sample to provide a consistent look for your plug-in.

No Settings

If you will not be exposing any settings to the user, simply delete the SettingsUI.xaml file.

Balancing Algorithm

Open Balancer.cs and write your balancing algorithm in the Balance() function.

You will be passed in an enumeration of BalanceStateInfos. Each BalanceStateInfo represents one or more disks part of the pool that are on the same physical disk.

DO NOT do any lengthy work in this function. DrivePool expects this operation to be very quick.

If you need to do some external work, such as query WMI or some other time consuming task, you must spawn a new thread in the constructor and do all the work there.

The rest of this document will describe the various operations that your balancing plug-in can perform.

Moving Files Around

The BalanceStateInfo.PoolParts property will enumerate each individual pool part that is on the same physical disk.

You can examine the various properties of each BalanceStateInfo in order to discover current file distributions and call MoveFiles() to move files around.

When requesting to move files you always pass a target ratio. This ratio is always calculated as a ratio of total disk capacity. 

// Example:
var moveResult = balanceState.MoveFiles(FileTypes.UnprotectedFiles, 0.25);

This means that you would like un-protected (or un-duplicated) files to use up 25% of this storage-unit.

DrivePool will go out and see whether it's possible to satisfy your request given the current file distribution across all the disks, it will try to satisfy as much of your request as possible and then give you back a moveResult which will contain information about how much data was actually scheduled to move.

You can also pass a MoveOptions instance as the third parameter for more control over the move operation.

Real-time File Placement Limit

In addition to moving files around you can also set real-time file placement limits like this: 

balanceState.SetLimit(LimitTypes.NoProtectedFiles);

Equalizing Pool Parts

Since you can only move files among the different balance states and not the individual pool parts, it may be advantageous to be able to equalize the disk space usage on all the pool parts that comprise a balance state. 

// Example:
balanceState.EqualizePoolParts();

Forcing an Immediate Balancing Pass

If your balancer requires the immediate moving of files, regardless of the trigger conditions or the balancing schedule, you can tell DrivePool that you want to start balancing immediately. 

// Example:
this.Actions.BalanceNow();

You CANNOT call this from the constructor of the balancer.

The user can disable this ability in balancing settings.

How it All Works

DrivePool keeps track of all file statistics in real-time and will periodically call your Balance() function in order to ask your balancer if it wants to modify the current file distribution.

DrivePool will do this for all the balancers in the stack, starting with the lowest priority balancer first.

At the end of this 'Calculation' pass DrivePool will determine how much data needs to be moved in terms of bytes and percent of total pool usage. It presents this to the user in the 'Pool condition" bar allowing the user to start a manual balancing pass at any time.

It will also evaluate any automatic balancing triggers and start an automatic balancing pass if required and possible.

The balancing pass will once again ask your plug-in to update the balancing model. It will then start to satisfy this model using the built-in 'FileMover' which is responsible for cataloging the (possibly) millions of files and building an optimal move set in order to satisfy the balancing model. This is no small feat, but nothing that you have to worry about.

Once the move set is built then DrivePool goes to work moving the files around using background I/O priority and a technique that doesn't lock any files.

If a system is shut down or the service is stopped while a move is in progress, DrivePool cleanly aborts the balancing run immediately.

If the system crashes while a file move is in progress, the user's data is not affected. Worst case scenario is that they will find an extra .dptemp file which can be deleted safely since the source of the move is still intact.

Distribution

If you would like your balancing plug-in posted on the official page, send us a note with your balancing plug-in attached @ http://stablebit.com/contact

Retrieved from "https://wiki.covecube.com/index.php?title=StableBit_DrivePool_-_Develop_Balancing_Plugins&oldid=409"