I’m finally getting off the schneid to write an entry that might hopefully be beneficial to the community.  I’ll do my best to make sure more of these will follow in the future.

A recent experience with a customer leads me into this discussion around performance optimization options when running an HP EVA 4×00/6×00/8×00 Storage Array.  As you may already know these storage arrays are commonly sold for their ease of implementation, management and use.  From that standpoint they are great, however if you have any need to understand how things are performing, you are limited in your options.  I’ll dig deeper in the future on the tools for performance monitoring, but for this post I’m going to describe a simple change on your Microsoft Windows-based hosts attached to the SAN that may help increase overall performance.

The HP EVA 4×00/6×00/8×00 Storage Array is an Assymetric Active/Active (AAA) Storage Array which simply means both storage controllers are active and able to service I/O requests, however only one controller manages a given vDisk or LUN.  That managing controller is the one that will ultimately have to service all I/O requests for the vDisks it owns.  From a write perspective, this is what it is and there are strategies for optimizing this traffic however on read activity this can result in unnecessary performance degradation on very active arrays because of how a read miss is handled.  I’ll explain what I mean…

An EVA 4×00/6×00/8×00 has a back-side channel for communication between the two controllers called the Mirror Port.  I believe this is nothing more than a point-to-point fiber channel link and each controller has two or three of these depending on the array version.  It is over these ports that communication between the controllers must occur and it is possible to saturate this communication link with data write and read proxy cache mirroring traffic.  We are going to focus on the proxy reads this time around and maybe look at the data writes at a later time.

A proxy read is a situation where a read comes into the controller that does NOT own a vDisk or LUN and the data requested is not currently in that proxy controller’s cache.  This is considered a Read Miss and a request is made to the owning controller to read the data into the non-owning controller’s cache.  Reading this data into cache will force the owning controller to push the data across the Mirror Port to the proxy controller.  While this is by design, it is not ideal because if your Mirror Port is under heavy utilization already this will just make things worse.  As a result we would like to see most or all read requests always come into the owning controller.  Fortunately, we have a thing called Asymmetric Logical Unit Access (ALUA) that EVAs support and can be utilized by the HP MPIO DSM on a Windows host to intelligently direct read traffic to the owning controller.

Now the downside is that the utilization of this ALUA feature is not enabled by default.  In fact, the default for Windows hosts is to leverage all paths and both controllers, essentially a round robin approach to path balancing and effectively sends half of all read traffic to the non-owning controller.  In order to enable the use of ALUA you must enable ALB (Adaptive Load Balancing) on each Microsoft Windows host currently running the HP MPIO DSM.  This can be done with the HP DSM CLI or the HP DSM Manager MMC plugin. 

Now this setting helped eliminate approximately 30-40% of all mirror port traffic for one customer that was running a very active storage array and had an immediate effect on reducing I/O latency.  The benefits will likely vary and could have no visible effects if your array is not under heavy utilization.

One way to monitor where you stand on performance and compare your stats before and after the change is to run the EVAPerf utility usually installed on your HP CommandView server.  Launch EVAPerf, which will open a Command Prompt and type:

EVAperf vdg –cont

This will provide you a view of the live stats from your EVA Disk Groups and specifically the Mirror Port MB/s will let you know how active your mirror port is currently.  I have been told that anything over 280 MB/s is reaching a threshold that could start displaying ill effects.  I have personally seen this reach upwards of 400 MB/s and significant I/O Latency came along with it.  This can display itself as lagging or locked VMs in a VMware vSphere 4 environment and very poorly performing Microsoft SQL 2008 Database servers.

Hopefully this will help someone troubleshooting an HP EVA Storage Array or planning a new implementation.  Comments and corrections are always welcome, we are all part of the same team, so let’s figure out how to make things work better.

This past year I made a point to organize myself in such a way so I could “live” out of my laptop bag, which really became more of my life bag for me.  Because space was limited I wanted to have the fewest physical items necessary to do my job.  As a result I started looking for tools that were either small or multifunctional.  My laptop along with the loaded applications is obviously my main tool and consumes the most space in my life bag.  However, I wanted to focus here on the other physical items in my bag that I could not do without.

These probably aren’t for everyone, but I can comfortably say that they have worked well for me.  I am always looking for new and better options, so please share if you know of better ones.

Notebook

Moleskine – For mindmapping or straight note taking – this is by far the best hardcover notebook I have ever used.  It is great when you don’t want to pull out the laptop or you need to quickly jot down a note.  I have used it to mindmap meetings, diagram solutions and generally record the details of my work life.  The books are available in hardcover (my preference) and softcover.  They also have various sizes and paper layouts.  I highly recommend having one of these.

www.moleskine.com

Pen

Franklin Covey Surepoint S3 or S4 pens – I searched long and hard for a good pen / pencil combo and at the time the best option I could find was the Surepoint S3 / S4 pens.  I initially purchased the S4 but found I really didn’t need the third ink color and I didn’t like the overall size of the pen.  That fact is what led me back to the S3, which also has a black pen, red pen, and a mechanical pencil.  This one I have used to death.  It is the perfect size and I really don’t miss the additional blue ink that the S4 had.  I believe some updated versions of this pen / pencil have since been released and would be worth a look.

www.franklincovey.com

Mouse

Microsoft 8000 Presenter Mouse – The one thing that I really dislike about most mice is that they are just mice.  I know they each feel different and many people have distinct preferences, but I have always preferred efficiency over comfort.  That’s when I started to look for a mouse that could serve several purposes, especially ones that could help me with my presentations.  The Microsoft mouse, which has been out for quite some time, has been great.  Since this is a Bluetooth device, attaching it to my bluetooth-enabled laptop was simple and I never have to use the Bluetooth dongle that came with the mouse.  I should also mention that it comes with a  handy hard case that makes throwing it into my life bag quick and keeps the device safe.

www.microsoft.com/hardware/presenter

Headphones

Shure – I can’t take credit for finding these headphones.  My brother actually gave them to me as a gift, but they are by far the best headset I have ever had and the hard carrying case that came with them is ideal.  These are a little pricey and probably not for everyone, but they are definitely a quality product.

www.shure.com

I have a mixture of other items in my bag as well, but these specifically have proven to be the most useful.

Through the years I have realized more and more that I truly enjoy working in the Information Technology field, but not necessarily for the same reasons that many other IT individuals do.  I am intrigued by the big picture, wanting to fully understand how IT can maximize its value and return to the business which it supports.  What is great about business is that every picture is unique in its own way and the technology used to support it can and probably will differ.  But the challenge is trying to find that perfect fit, those solutions that will drive innovation, establish simplicity and supportability of the environment, keep the return on investment high and ultimately help the business reach its strategic and concrete goals.

The analogy of a painter I think applies well here; at the start of the process a painter begins with a blank canvas and sets the objective of the painting, for example let’s say a fall landscape, a la Bob Ross.  Once they have established the WHAT, then they can define the HOW.  There are many ways they go about creating their landscape, but the key is that they continue to paint and don’t get too hung up on a specific detail or the need to repaint the same section of the painting over and over again. 

Technology for most businesses is typically only a means to an ends, unless, of course, your business is the technology.  The information technology department and the solutions they provide are truly business accelerators, either positive or negative.  Technology applied to a well run, disciplined and focused business will make it all that much better, while technology applied to a poorly run, directionless business could make it even less effective, so while I won’t tell you that your IT department will make or break your business, it certainly will contribute to its current direction.

There are several key principles I feel are important for any IT organization:

1. Allocate and spend money as if it were your own.
2. Evaluate every request, need, or want relative to the concrete objectives of the business.
3. Approach each assignment in a way that you will be proud of the results you deliver.
4. Operate as a team, disciplined and focused on a common goal and sharing in the results together, regardless if they are successes or failures.

I have had the privilege and sometimes misfortune of working in both a corporate IT department and an outside consulting firm.  While the views are different from either vantage point, the challenges they attempt to manage and solve are quite the same.  I believe that because IT sits in a unique and critical place within every business; we have the opportunity to see how each and every department functions, we often share in the frustrations and operational hurdles faced by many of the employees and we have what I believe is an enormous responsibility to help the business improve its daily operations and reach its overall goals.

Since the focus of the consulting firm I am currently working for is datacenter delivery and virtualization, some of the concrete topics will revolve around datacenter technologies such as server, storage and software virtualization, server and storage networking, and infrastructure applications.  The other topics will be abstract concepts around departmental and business structures, IT decision making, and operational effectiveness. 

Translation:  I’ll attempt to blog about the stuff I physically touch or see and the thoughts or ideas that pass through my head from practical observation. 

Hopefully some of those things will be useful, beneficial, insightful, or humorous to some of you out there and will ultimately contribute to the improvement of your operations.  If I am totally off base on anything I ever write about, please say so, this isn’t intended to be a soapbox but a collaboration of ideas with the goal of improving IT departments and making them the best that they can be.

If nothing else, this blog will be an excellent place for a technology geek to express his professional journey and any experiences along the way.