کانفیگ کردن پیشرفته EMC Isilon SmartConnect
In Part I, I covered SmartConnect Basic, an included feature of OneFS, that handles connections to Isilon node IP addresses based on a Round Robin connection policy.
What do you do if you want a more dynamic, more resilient, and better load balanced solution? You implement SmartConnect Advanced. Remember, that SmartConnect Advanced requires an additional license.
So why is SmartConnect Advanced so much better than SmartConnect Basic? From a previous post, I mentioned:
SmartConnect Basic has the following features:
- Static IP allocation to nodes
- Connection Policy Algorithms
- Round Robin (which node is next)
- No Rebalancing
- No IP Failover
SmartConnect Advanced has the following features:
- Dynamic IP allocation to nodes
- Connection Policy Algorithms
- Round Robin (which node is next)
- Connection Count (which is next given connections each node has)
- Network Throughput (which is next given the amount of throughput each node has)
- CPU Usage (which is next given the amount of CPU usage each node has)
- Automatic or Manual Rebalancing
- IP Failover Policy
- Round Robin (which node is next)
- Connection Count (which is next given connections each node has)
- Network Throughput (which is next given the amount of throughput each node has)
- CPU Usage (which is next given the amount of CPU usage each node has)
As can be seen by the feature set, SmartConnect Advanced allows for Dynamic IP allocation, more Connection Policies, Rebalancing, and Failover Policies. Quite a few more features.
Dynamic IP allocation to nodes
Dyanmic IP allocation to nodes gives some additional capabilities to the mix that may or may not be readily apparent. In Part I, the front end network (subnet0) had a range of 
172.16.1.11-13. With SmartConnect Basic each node had a single address. If the range had been 172.16.1.11-172.16.1.16, each node would still only have 1 IP address each.
With SmartConnect Advanced, all of the addresses are added to the pool. In my 3 node cluster, all 6 addresses are evenly distributed when using SmartConnect Advanced. I only have to change my IP allocation method from Static to Dynamic.
With IP addresses being statically allocated (SmartConnect Basic), I can only use 1 IP for each node in the cluster. With SmartConnect Advanced, I can use multiple IP addresses per node. As I add more nodes to the cluster, the IP addresses are automatically distributed across the nodes. In short, I configure my range up front, and as I grow the cluster, I don’t have to make configuration changes to my clients.
Connection Policy Algorithms
SmartConnect Basic only has a Round Robin connection policy. There is no intelligence on the decisioning of which IP to give out by SmartConnect, other than to say which IP is next in, what I would call a “hunt group.” Yeah, an old PBX analogy, but pretty accurate here. With Round Robin, we go to the first IP, then the second, then the third, and so on. Not much intelligence there…
SmartConnect Advanced adds a couple more connection policies, including Connection Count, Network Throughput, and CPU Usage.
These additional options add a lot of flexibility to determining which node is going to answer next for the cluster.
- Round Robin: The cluster issues the IP address for the next node in the “hunt group”
- Connection Count: The cluster issues the IP address for the node that has the least number of connections.
- Network Throughput: The cluster issues the IP address for the node that has the least amount of network throughput
- CPU Usage: The cluster issues the IP address for the node that has the lowest CPU utilization
Looking at these connection policy algorithms, we can see that there are, based on our use case/workload/etc, options that can add some intelligence to the balancing of connections to the cluster.
After changing my Connection policy to Connection Count, and running my PowerShell script mentioned in Part I, I can see that I am no longer using Round Robin for the resolution of cluster.isilon.jasemccarty.com.
In the graphic above, not only do I have additional IP addresses, I also have a different connection policy being demonstrated. Despite only having 3 nodes in my cluster, my cluster answers for 6 addresses. This is because SmartConnect Advanced is balancing all of the IP addresses all of the nodes. Also notice that the IP addresses are not simply moving from IP to IP as a Round Robin connection policy would use.
Based on the EMC Isilon whitepaper I mentioned in my VMware vSphere and EMC Isilon – One datastore or many? post, it would be easy to configure multiple, let’s say 6, NFS datastores on 3 nodes. It would look something like this:
- nfs1 on 172.16.1.11
- nfs2 on 172.16.1.12
- nfs3 on 172.16.1.13
- nfs4 on 172.16.1.14
- nfs5 on 172.16.1.15
- nfs6 on 172.16.1.16
But I only have 3 nodes? SmartConnect Advanced will spread the load out for me, automatically. And later, when I decide to add 3 more nodes… Wait for it… I don’t have to change my vSphere configuration, I only have perform a rebalance operation (I’ll go into that next) and each datastore will be on its own node.
Rebalancing
Another cool feature with SmartConnect Advanced is Rebalancing. This can be done either manually or automatically. By default the policy is Automatic Failback.
With the Rebalance policy set to Automatic Failback, the only times that a rebalance operation is triggered is when there is a change to
- the cluster membership
- the cluster’s external network configuration
- a member network interface
When set to Manual Failback, the policy does not redistribute IP addresses until a rebalance command is issued via the command line or Administrative Web Interface. I have even seen situations where a cron job has been used to rebalance IPs on the cluster during scheduled times of less activity, including maintenance windows, etc.
IP Failover Policy
The other feature that SmartConnect Advanced adds, is IP Failover Policy. This has also been referred to as NFS failover. It determines how to redistribute the IP addresses to the nodes in the SmartConnect pool in the event that one of the other nodes becomes unavailable. For this to work, the IP allocation method has to be set to Dynamic. All four connection policies are available to the failover policy.
An example of the IP failover policy would be:
- The nfs3 datastore is mapped to 172.16.1.13
- The IP 172.16.1.13 is assigned to Node 3
- Node 3 is no longer available
- The IP 172.16.1.13 is reassigned to Node 1 based on the IP failover policy
Think of a situation where multiple vSphere hosts are accessing nfs3 on IP 172.16.1.13 and that node needs to be brought offline for maintenance, a rolling upgrade, or other administrative action. It would require significant planning and possibly downtime, if that IP address could not be moved to anther node. SmartConnect Advanced can accommodate for times when nodes are no longer available.
Additional SmartConnect Advanced additions
In addition to the above features, SmartConnect Advanced adds the ability to have multiple SmartConnect zones. Yes, multiple SmartConnect Zones. SmartConnect Basic only allows for a single zone, but SmartConnect Advanced provides the ability to have multiple zones. Imagine having a cluster with a combination of X, S, or NL series Isilon nodes… We would create a General Use SmartConnect zone with the NL series nodes assigned, as well as a performance zone with the X or S series nodes.
The ability to have multiple zones gives the flexibility of pointing clients that require the most performance to the fastest nodes in a cluster, while pointing clients that require general use to general purpose nodes in a cluster. When using SmartConnect Basic all clients access the same SmartConnect zone.
I didn’t mention it in Part I, but it is entirely possible, with either version of SmartConnect, to have more nodes in a zone than IP addresses available. Node membership in a SmartConnect zone is configurable, but does not require all nodes to be in the zone. The ability to choose which nodes are in a particular zone coupled with the ability to present multiple zones (SmartConnect Advanced), provides a huge amount of flexibility.
More Information on SmartConnect
If you would like more information about SmartConnect, check out this whitepaper:SMARTCONNECT Optimize Scale-out Storage Performance and Availability
If you are attending EMC World 2012, come by the Hands On Labs and take HOL16 – Isilon Setup, Scaling, and Management Simplicity to have hands on experience with SmartConnect.