Gotcha #7 - AppDynamics and the Mysterious Case of the Vanishing Cookie

Not much of a mystery today in this article (the answer is alluded to in the title) but the process of identifying the problem, discovering why it was a problem, and then fixing it was quite a challenge.

Today's Gotcha is about Rational Performance Tester (v8.5.1.2 for anyone interested) and a vexing problem with successful test execution.

The problem originally arose when we discovered that our tests weren't always succeeding. After executing a schedule we would consistently have a large number of pages that wouldn't have 100% success status codes (located on the Server Health Detail view). There would be a few with 100% success rate, but consistently nearly every page was hovering between 94 and 98%. In some larger tests this would drop further, into the 80's or high 70's, but even a simple low volume test would have most pages at less than 100%.

To get a better handle on why and when these page errors were happening, I added a new counter to the default Page Throughput / Page Hit Rate graph (Add/Remove Performance Counters --> Page Performance Counter... --> Status Code Successes --> Percent Page Status Code Success [for Interval]).

Suddenly we could see this wavering line appear on the graph that would start off at 100%, dip sharply at the beginning of the test, before returning back up near 100% and stay fluctuating in the high 90% range.

Baffling when the very same test scripts in the past against the same application would run at 100% without trouble. Doubly-baffling when running the script as a standalone would work perfectly fine.

Following the Clues

The first clue came when we decided to rerun one of our test schedules and monitor the Test Execution of a user thread. Since the failures were reasonably consistent across all pages, chances are we might be able to catch it in the act.

After a couple of hours of watching successful page responses, we caught one. Right from the beginning of a new test script - a page failure. And then another, and another, and so on until every single page in that script failed in order. And the next script. And the next... but then the fourth script, for no apparent reason, began returning successful responses again.

Digging into the error, the only message we could find was a "Failed substitution ^csrfToken=.*$" message followed by a cascade of 500 - Server Error responses. The csrfToken appears in the HTTP Response header as a "Set-Cookie" value from the server (you can find an explanation of its purpose here). And for some reason, it wasn't being found.

Except... inspecting the response of that very first page - the csrfToken was right where it should be. Further digging into the application logs showed the same thing - on occasion our test would be sending an invalid csrfToken - the recorded token - which happens when the substitution couldn't be made.

But it was right there in the Response Header - right where we expected it to be. Except, not quite...

Smoking Gun

It looked like it was right where it was supposed to be - in among a half-dozen other cookies being set by the server. But a careful comparison with the original recording showed that it was 1 line down. There was a NEW Set-Cookie value that was inserted in the list just ahead of the csrfToken.

Inspecting and comparing successful test executions with failed ones showed that this new mysterious cookie only appeared in the failed tests, and wasn't to be found in the successful ones.

The server was randomly inserting a new cookie in its response... some of the time. And it was enough, by shifting the order of the Set-Cookie headers for RPT fail when trying to identify the csrfToken.

The culprit was a cookie called ADRUM_BTs and it looks like the following:

Set-Cookie: ADRUM_BTs="R:0|s:f"; Version=1; Max-Age=30; Expires=...date...; Path=/; Secure

There was no consistency in when or where it would appear. It would show up for a script execution or three, then disappear once more.

Solution

It turns out that we weren't the only ones to run the Mysterious Case of the Vanishing Cookie - other people had identified ADRUM_BTs as playing havoc with their scripting efforts.

The cookie in question belongs to AppDynamics End User Monitoring javascript package, a tool that can be used to monitor user behaviour and web application performance from the user's perspective. It was a new feature in the AppDynamics package that had been turned on in our Performance Testing environment recently to try and improve our measurement capabilities.

In the end, there was no way to remove this new cookie or adjust the system to consistently distribute it. In order to get our Performance Testing back up and running we had to remove all End-user Monitoring components - and suddenly our vanishing cookie was gone for the last time.

Gotcha #6 - Sometimes Naive Caching is Worse than None

There is a popular proverb about good intentions and the road to hell that is well known, but an alternate version of that proverb is phrased as "Hell is full of good meanings, but Heaven is full of good works". Unfortunately even good works can lead down the wrong path when they are undertaken with insufficient consideration for their impact.

This post is about one such work - about the implementation of a caching strategy on a large platform that is widely used across the industry. The work itself is good, smart, and done with significant forethought. However under certain conditions, the choices made ended up creating a system worse than if nothing had been done at all. This post explores the choices, circumstances, and effect that implementing an intelligent - but naive - caching system had on one particular project and why the result ended up being worse than if it had never been.

The names of all organizations, people, and software involved have been withheld.

6 Days to Go-Live

A war room had been convened. The production deployment of a new Public Access System (PAS) had gone smoothly over the weekend and everything was prepared to turn the switch and make it accessible in less than a week. The public information campaign had already been underway for some time, advertising the date, and getting the word out that this new offering was forthcoming. However, validation of the deployment had hit a snag - one member of the QA team had noticed something wrong.

At a critical step while making a new purchase on the PAS, the system would appear to pause. After a long delay (45s) it would return with a technical error. Repeating this process several times in a row would produce the same error across multiple sessions, across multiple machines. And then it would go away and everything would work smoothly again.

Until it happened again about an hour later.

And again an hour after that.

And then it went away. Combinations of testing, several people trying to test it in a variety of ways, and using precisely the same data - or different data - it couldn't be reproduced.

Until the next morning, when the problem showed up anew and again the following hour, before disappearing once more.

So began an urgent search to understand the problem began, testing the system, combing the logs, reviewing all the code tracing and analytic tools at our disposal to isolate and resolve the problem - in production.

Underlying this effort was the question - why wasn't this problem seen in any other environment? Through the myriads of Dev and QA environments, through multiple test cycles and performance testing, this pattern of problems had never been encountered before. At times a similar issue would be seen in isolated circumstances immediately following the restart of an environment, but once it was initialized everything was good.

So why was it happening in production, and nowhere else?

Architecture

The PAS is a clustered three-tier system using RESTful services to communicate between the web server and the Back-end Application (BEA). The BEA contains all the business code required to verify, validate, and persist purchase information.

The PAS cluster contains 2 nodes, and the BEA cluster contains 8, separated into two group of 4 - one group dedicated to the PAS and the other for direct Intranet access. As a true RESTful interface, no session information is tracked between requests to the BEA cluster. The load balancer provides a true round-robin.

When the system is active, most of the processing load takes place on the BEA cluster, the PAS operates as little more than a view rendering layer. When a page request comes in to the PAS, it opens a connection to one of the BEA nodes - passes along the request - and waits for a response. In the case where the BEA node takes an abnormally long time, or if there is a failure and no response is generated - the connection has a 45s timeout. If it takes longer than 45s, the PAS would close the connection and display a technical error message encouraging the customer to call their local company representative to complete the purchase.

The Trail of Evidence
When the problem was first raised, there was a great deal of speculation as to the possible cause. Database issues, connection issues, misconfiguration, load balancer issues - all possible sources of this technical error, but all things that would need to be verified and eliminated.

There were a number of observations made about the system that were unusual or worth investigating - including hung database threads and the presence of a query in the AWR (query performance statistics) report that was known to be problematic in other environments. But in hindsight there was only one that would prove to be important to the actual cause: the problem only happened in the morning when full-time employees arrived for work and began using the BEA system directly via the Intranet.

Now the Intranet BEA nodes are physically separate from the ones used by the PAS, but they do share a database. The problem only occurred during the morning hours when load on the BEA nodes and the database were at their highest. Not high enough to cause a bottleneck or result in performance problems - but not insignificant either. More on this later.

The most important piece of information came from our enterprise performance monitoring tool that was installed on all production servers. By drilling down, identifying, and isolating a single request that failed we could examine the performance of the BEA code for that single request.

And what we discovered was a function call that took 24s that involved retrieving a large but critical piece of reference data from the database, and storing it in the application cache. By itself, not enough to cause a timeout and result in a technical error - but in a confluence of circumstances this piece of reference data was then being used for a large and complex set of calculations that took ~20s to complete and in this particular instance was just barely slow enough to cross the 45s timeout threshold and cause the technical error to be displayed. 

Now this isn't unusual - the median time for this particular load-and-store operation was about 15-18s, and once it was complete the data would be cached and any future accesses would take 1/10th of a second or less. But in the situation where it did have to retrieve the data from the database while the db was under significant load caused it to slow just enough to trigger the problem.

But why was it happening at all? Why wasn't it happening elsewhere? And why wasn't it discovered during Performance Testing.

Unexpected Caching

The answer came in a conference call to the vendor. The cache on the BEA system operated as follows:

1. Objects in the cache are marked as stale at intervals of 15 minutes if they have not been accessed during that timeframe.
2. If an object was already marked as stale when the interval arrived, it would be evicted from the cache.
3. Every 4th interval (once per hour) all objects in the cache would be evicted regardless of their state.

This cache interval was configurable to a maximum of 30 minute intervals, extending the hourly global eviction from 1 hour to 2 hours - but no further. This means that every hour all reference data would be purged from the BEA system's memory and need to be reloaded the next time it was accessed... on each node.

The Core Problem

In order for the problem to occur, the following had to happen:

1. The BEA's cache was empty on the nodes that were used by the PAS.
2. The other Intranet BEA nodes had to be actively used by a large number of concurrent users to put sufficient load on the database to cause a retrieval delay for the large amount of reference data.
3. The PAS had to be inactive, with only 1 or 2 users online performing a new purchase. Other operations would cause the reference data to be loaded but not perform the complex calculations that would cause the response delay to exceed 45s.

An unlikely scenario it seems, right?

Except...

1. The BEA cache would be emptied on the hour, every hour.
2. The Intranet BEA nodes were being used by employees for their normal job, and would be heavily used every weekday for several hours during the morning.
3. The PAS was a limited-scope rollout, the only functions that it performed were New Purchase, and Continue a Saved Purchase.
4. The PAS was so new, that the initial expected load was only a handful of users per day. So the likelihood of having only 1-2 users performing a new purchase was very high.

These circumstances meant that it was very likely that any user of the system would run into the situation where all three conditions were met and cause their purchase attempt to fail.

This problem was further compounded by the round-robin load balancer, and the fact that the reference data needed to be loaded on each node separately. This means that not only was a user likely to encounter the technical error message, but if they attempted to retry their purchase their next page request would be directed to a different BEA node whose cache was still empty resulting in the same error message again.

The expected result is that for a majority of potential customers, each would need to retry their purchase five times before successfully completing it.

Why Wasn't This Caught?

The simple answer is the other environments were not busy enough. Dev, QA, Staging... all of these environments did not have sufficient database usage to cause enough delay to cross the 45s delay threshold. In addition, these environments typically were not configured in the dual-cluster architecture of production, so any activity directly on a BEA node would cause the reference data to be loaded into cache. The chances of a PAS new purchase occurring with an active database were essentially non-existent.

The exception was the Performance Testing environment which did have a fully configured production system, and sufficient database traffic during a test to cause the problem. The reason it wasn't discovered was that the PAS traffic was too high. Even a half-dozen simultaneous users was sufficient to cache the reference data across all the BEA nodes - and even if the error was observed near the beginning of the test once or twice, a few minor functional errors are not unexpected during a performance test and the absence of reoccurence over the next hour would be simply dismissed as an anomaly.

As soon as the circumstances of the problem were discovered, it became a simple matter to intentionally reproduce in Performance Test by adding load to the Intranet BEA nodes and manually attempting to test a new purchase.

Why Did This Happen?

This problem is caused directly by the implementation of a naive caching framework on the BEA system. The purpose behind clearing the cache regularly would be to ensure that reference data (if updated) could only be stale for a certain amount of time before being refreshed. The assumption was that the cache could be cleared every hour without causing a problem, which ended up not being the case. The BEA system had no facility with which to preload reference data, no functionality allowing it to be refreshed periodically except through eviction and reload on-demand, and no ability to persist reference data in a long-term no-expiry cache.

The reason this was more than just a normal problem, is that by implementing a limited cache the system was successfully able to mask the existence of the problem from developers, qa, and performance testing. The naive caching strategy allowed it to work well under load most of the time, and only appear as a problem in very specific circumstances.

Had there been no cache on the BEA system, this problem would have become abundantly clear in performance testing by resulting in terrible performance for that operation and also causing the error rate to approach 100%. But by masking the issue it managed to survive until mere days before go-live when it was discovered entirely by accident.

The Solution

The permanent solution will be to modify the BEA system to build a more sensible multi-use caching strategy with the ability to preload and periodically refresh reference data independent of a user request. Temporarily it has been resolved with an automated process that makes an appropriate service request to each BEA node every minute that triggers the reference data reload if it has been evicted in an attempt to complete that reload before a user encounters it.

Gotcha #5 - Network Performance, CSS, and HTML5

Today's article comes courtesy of understanding your customer's environment, and that emphasizing style over substance can break the bank.

Our story starts with an urgent request for assistance from a system administrator, a minor application update has resulted in a nearly five-fold increase in total network traffic at a remote office. On Monday morning, when the morning shift arrived network traffic began to spike. It grew, and grew, far beyond normal traffic levels for the office and stayed there, night and day, until the last shift signed off on Friday evening.



Think about this for a moment.

The application's network traffic did not increase five-fold. The entire office's network traffic consumption increased from ~2Mbps (24-hour average) to nearly 9Mbps (24-hour average) with peaks topping out at 10Mbps.

The first stage is always denial.

How could a minor update to a server-based web application that mostly consisted of a couple minor bug fixes, one new summary screen, and some UI tweaks be the problem? It must be something else, there isn't anything new or unusual in the application update.

Well...

Step 1: Isolating the Problem
Easy enough to do, by using more detailed reports we could see the initial spike in unusual network activity began at 10am on the Friday before, the same time as the outage window when the application update was deployed.

Also, there were no other maintenance activities ongoing at that time, no file transfers, and no other system updates scheduled or otherwise.

Hmm...

Step 2: Reproducing the Issue
To confirm, we conducted a test where all instances of the application on every workstation on the site was closed, and by so doing network traffic returned to its normal (low) stable state.

One by one, each workstation re-loaded the application and by time the last workstation was reconnected the network traffic was back at its five-fold peak. It was during this process of re-loading that the system administrator noticed something unusual. A new UI feature had been added to some screens, when a specific piece of data was in a warning state - the text box would flash orange. And every time a new screen loaded that happened to display one of these flashing orange boxes, network traffic would spike sharply.

In the end, the system administrator noticed 10 out of more than 100 workstations happened to be displaying this flashing orange box. Returning to shut down just these 10 flashing displays suddenly dropped network traffic back to normal once more. When they were re-loaded, network traffic would spike once more.

Step 3: Cause and Solution
So how can a simple CSS and HTML5 flashing text box on 10 screen possibly start pushing 4 times the total network traffic of an office of more than 100? The technique to highlight a text box had been used elsewhere in the application already, and the style control was rendered entirely in-browser with no additional server requests being made.

How can a box no larger than 400px by 300px with a lovely fade-in/fade-out HTML5 animation possibly consume 800Kbps per screen?

The Epiphany

Animated. For the first time in this application, the developer (to satisfy a requested requirement) used a simple animation cycle to flash the control orange on a 4-second cycle. It was a lovely effect, understated, but unmistakable and was easily visible when displayed on a large screen from across the room.

What wasn't clear however, was that each workstation did not use a browser to view the application. Instead, each workstation used a Remote Desktop (RDP) session to a centralized (offsite) server which then connected to the application in a browser window.

Instead of being rendered in a local browser session and consuming no network resources at all, the beautifully animated flashing textbox was being rendered and streamed as uncompressed full-motion video over the network to the site office.

10 workstations streaming uncompressed video for 24-hours a day, 5 days a week.

Step 4: Resolution
In the end the problem was easy to solve. Rip out the animation, and using a simple steady-state orange highlight to communicate the information.

But it was an important lesson. You must be aware of how your customer is using the tools that you build, and how the simplest of design choices can cause a major impact under the wrong circumstances.

The Black Art of Performance Requirements

BAWorld Winnipeg (October 7-9, 2015) at the RBC Convention Centre is now over, and I had a wonderful time listening to a full slate of excellent, interesting, informative speakers. I had the opportunity to meet many new people both in the Winnipeg Business Analyst community as well as others who joined us from out of town, and renew acquaintances and friendships with others that I haven't seen in awhile.

My own session, The Black Art of Performance Requirements went very well, I had a great time presenting and I'm glad I could provide something a little bit unique and different.

If you are interested in the slides from my presentation, I have made them available here. Download presentation for The Black Art of Performance Requirements (315kb).

ICPE 2015, City of Austin - Part 2

(Click here to read Part 1)

During the remainder of my stay in Austin, it never got as warm as when I arrived. It stayed a rather balmy 5-10 C for the next few days, which was beautiful as I could wander around comfortably in my shirtsleeves. I cannot say my opinion on the weather was exactly shared by the locals, most of whom were bundled up in parkas and toques (the following is translated from Canadian to English: "winter coats and woolen hats"). I think their attitude was best illustrated by a pair of women whom I stopped behind at an intersection waiting for the walk sign. I was in a t-shirt and carrying my windbreaker because I was quite comfortable, while the two of them were quite bundled up against the cold and I overheard one remark to the other "I can't believe how cold it is! I can't wait for it to warm up again, but they're saying its going to be in the 40s all week." For some reason they abruptly switched to a different topic when I passed them by.

As I alluded to in part 1 of this post, I have never been anywhere that is green in February. Now plenty of the trees had bare branches, and there patches of brown in the grass that spoke of it being winter, but it was still green. But besides that there were two things that struck me plainly on my first day in Austin.

The first was parking.

In the area of San Jancinto Blvd between 3rd and 6th, and east towards 290, downtown Austin is dotted with low and high-rise condo complexes, and if I might digress for a moment downtown Austin is a lovely place. It is open, populated, there are people out and about walking around throughout the day and well into the evening which may have something to do with 6th street. But people live in downtown Austin and it has a very friendly, clean, comfortable atmosphere to it from the short experience I had. I could easily picture living there, right in the heart of the city - something that I would be less inclined to do in my own city (Winnipeg). Now, I live near downtown Winnipeg and I have for nearly a decade, but always just outside of the true core area. In Austin I could easily live right in the core without a second thought.

What struck me though, was that every single parking garage or pad that I passed was locked up. There were no arms to block entrance, every one was locked behind a 6-foot metal fence and gate. Ok, there were a few public-access surface lots, but every private lot was fenced, gated, and monitored. Perhaps it was the section of the city I was in, but in my experience it was something to note.

The second just about scared the living bejeezus out of me.

The first evening that I was walking back to my hotel, the sun was setting and night was drawing close. Have you ever been suddenly accosted by a pack of wild demons from hell screeching in a chorus and hungry for your flesh? No? Are you sure? As I passed by a tree-lined side street on my walk, the entire line of them began shaking violently with an incredible cacophony of noise. After recovering from my unexpected heart attack I realized they were birds, an enormous flock of them that filled the trees in numbers unlike anything I've seen before. I later learned that I had a close encounter with what was likely an annoyance of grackles. If there has ever been a more appropriate name for a group of creatures, I haven't heard it.

My conference was being hosted at the Hilton Garden Inn Austin Downtown which I wasn't actually staying at, because of timing and funding it had no vacancy when I was booking a room. The hotel was a nice enough facility, but had one particularly unique feature.

And that would be the canal and walking paths that ran through it. The windows just on the left side of the photograph is the dining room where we had lunch which afforded a fantastic view of the canal and the paths that run along side it,

During the few days that I stayed I managed to dine at two of the most iconic... or should I say representative institutions in Austin. The first was of my own doing, I couldn't pass up visiting a gem of Texas history while I was here: The Driskill Hotel. The lobby is jaw-droppingly gorgeous, and the restaurant/bar is about as truly upper-crust Texas as any visitor could hope for from the beautifully polished woodwork, to the leather seating, and the bust of a steer that graces the wall above the mantle. It doesn't hurt that the food and drinks were fantastic as well, and though it was quiet (it was a Monday evening after all) a musician was busy plying his trade, singing sweet country songs on the small stage. I didn't take any pictures of the building while I was there, but a quick search will give you a good feel for it.

The second place was with all of the attendees at ICPE 2015 where we were treated to dinner and a show at a place that is true to the spirit of Austin: Esther's Follies. The show combining satire, political commentary, music, and magic was up-tempo and enormously entertaining. I gather that the edition that we saw was more international in appeal than previous ones as one of the local patrons made a comment that there was a lot less Austin-specific material than previous times they had been there. But, considering that our group of more than 100-odd people contained a majority of non-Americans (largely European, but it was very much a group that hailed from across the globe) the show was very well received, though I believe a few Rick Perry references went over their heads. The liberal leaning of the show's writers also seemed to agree quite well with our European friends.

(Part 3 will be coming soon!)

ICPE 2015, City of Austin - Part 1

This post is all about my impressions of my trip to Austin TX for ICPE '15 and the city in general. I have covered details about the conference itself in other posts.
ICPE 2015, Getting Published with ACM, and Performance Management
Book Club - Foundations of Software and System Performance Engineering

Dear Austin Texas - your town motto seems incredibly appropriate. I'm not referring to your official motto of "Live Music Capital of the World", though that was abundantly evident as well, I mean the unofficial one "Keep Austin Weird."

Flying to Austin is weird. Flying to Austin on the afternoon of Superbowl Sunday on a plane that is running an hour late and half-filled with Texan transplants that know they aren't going to land before kickoff is more so. Especially for a true-blood Canadian who reserves such passion for games that don't prominently feature an attention hogging rhythm impaired southpaw shark.

But the weirdness started earlier than that. For a city that is the capitol of the largest state in the lower 48 and the second-largest population, a city that boasts an international reputation as a cultural hub, it is hard to get to. I only had two real options booking my flight from Winnipeg (YWG) to Austin (AUS) if I wanted to avoid an overnight layover (I did) or arriving late at night and risking a delay causing me to be late for the start of ICPE '15 on Monday morning (also yes). Since I was travelling on February 1st from Winnipeg, delays or cancellations were very much a possibility. The two options I did have were to leave at 6:30am (please arrive at the airport two hours before departure if you are flying internationally) with a 6 hour layover in Minneapolis (MSP) before arriving in the early afternoon. Or leave at 7:30am with short layovers (less than an hour) in Edmonton (YEG), and Los Angeles (LAX) before arriving at dinnertime.

In the interests of not flying a circuitous route around half the continent and potentially missing flights in two different airports, I chose the earlier departure. After arriving and puttering around the Minneapolis airport for a number of hours I went to my departure gate early where the staff let us know that the flight was going to be a half-hour late. Half an hour later it was now an hour late. Shortly after that when our original departure time arrived they let us know that although we were schedule to fly on Delta - someone along the way failed to realize that we would require an actual airplane in order to do so... and that they would try to find a new airplane to replace the non-existent one that we had been scheduled to fly on, but we would be facing further delays or possible cancellation.

Fun!

Fortunately they were able to discover an airplane in Wichita that we could use. I won't claim to understand the logistics of airline flight scheduling, but when the plane from Wichita arrived it proceeded to unload a full compliment of passengers. I'm not sure if the passengers were intending to come to Minneapolis, or if they were merely abducted in order to justify providing us with transportation, but either way - we had a plane, and one that managed to arrive in Austin less than half an hour ahead of the flight from LAX.

To give you a little context about my experience arriving in Austin, let me tell you a little about myself. I have never been "south". I've never traveled overseas, never been to Mexico, or Cuba, or on any kind of warm vacation. I've driven coast-to-coast in Canada, put my feet in the Pacific Ocean in Victoria BC, and in the Atlantic along the red sand beaches of PEI. But the absolute furthest south I have ever been is Gary Indiana - just at the very tip of Lake Michigan. I have never been anywhere that doesn't have snow in February. When I left home at 4-something in the morning in a sweater and a windbreaker the temperature was -26 C (windchill of -44 C) and when I arrived in Austin it was +23 C, an apparent difference of 67 C (or 120 F), a change roughly equivalent of that experienced by mall shoppers in Florida when they exit the building on a summer's day. In fact, the day before it was just as cold and I managed to snap a picture of the most brilliant sun dogs I have ever witnessed.

In Austin I stayed at the Hyatt Place Austin Downtown which is a beautiful hotel in the heart of downtown across from the Convention Center and just a few blocks from 6th street, congress avenue bridge, and easy walking distance to the Capitol. The accommodations were fantastic, the staff were wonderful (and one of whom I discovered was an expat from Winnipeg no less), and they put me on the top floor facing east which gave me an amazing view of the morning sunrise.

Upon my arrival, seeing that I was in Texas I heeded that age old proverb about the Romans - and I turned on the Super Bowl just in time to see the halftime show which proved to be more baffling than I expected. Alas, as I was busy doing my final preparations for ICPE I was watching it in my room while I worked, so I had no one to translate for me - but I gather that one of Katy Perry's dancing sharks was some kind of fortune teller that was predicting the apocalypse that would be brought about by an enormous lion? But I may have gotten that wrong. I did though appreciate the amazingly entertaining final 5 minutes of the 4th quarter (7 hours in real time) which was about as dramatic an ending to a game as I have ever seen. It's just a pity that Russell Wilson wasn't using the punctured bicycle tire that Brady was playing with or that last interception might not have happened.

Link to Part 2.

Book Club - Foundations of Software and System Performance Engineering

I recently received and am currently reading a copy of Foundations of Software and System Performance Engineering that I ordered through Amazon by André Bondi, a Senior Staff Engineer working in performance at Siemens Corp.

I had the good fortune of meeting André Bondi this year at ICPE '15 (which is where I learned about Foundations) and he was a fascinating, enthusiastic, and wonderful person to talk to. After I had presented my paper on Defining Standards for Web Page Performance, he approached me and we had a wonderful discussion about performance requirements and the perspective of the end-user and he had great things to say about the work that I was doing. His energy and interest in the subject was plainly obvious. It was a pure pleasure to have the opportunity to meet and talk with him.

As for Foundations, I haven't read very far into it yet, about 60 pages or so, but I have gone through and skimmed each section. This book is shaping up to be a fantastic resource and introductory guide to Performance Engineering. Since it is based on a training course that Dr. Bondi developed to train performance engineering and testing teams, I would expect no less. The book covers the entire software lifecycle from the perspective of how it connects with Performance Engineering as a practice, from requirements, to metrics, analysis, workloads, testing, instrumentation, and validation, and how they work within Agile environments and how to communicate and work with stakeholders on the project.

I won't write a complete review of Foundations yet as I still have much more to read, but I can be quite sure that this book is going to take a prominent place on my bookshelf, and play an important role while I am training my own performance engineering team.

BAWorld Winnipeg 2015 - Oct 7 - 9

The schedule for BA World Winnipeg 2015 taking place at the RBC Convention Centre from October 7th to 9th has just been released, and I have received notice that my seminar proposal has been accepted.

I will be presenting The Black Art of Performance Requirements for the Modern Web on October 8th at 2:15pm where I will be discussing the disconnect between what we believe the typical end user needs in terms of system performance, what they say they want, and what they actually need. This seminar is based upon a real-life case study and the contrast that it presented with "industry standards" for performance.

Some of the material is based upon a research paper that I presented at ICPE 2015 in Austin TX and is available online at the ACM Digital Library or MNP Media Library.

The complete session description is as follows:

Performance is a critical consideration in any project, many projects fail due to not only poor performance, but project teams that don’t give performance due consideration. Much of the reason is a lack of general understanding about how to define performance requirements, what makes a good performance requirement, and how to elicit co-operation to ensure they are met. The Black Art of Performance Requirements sheds light upon what end users want, what they think they want, and what they actually need. It examines the failings of industry standards, the reasons the results of industry studies fail to provide usable recommendations, and how to salvage value out of existing literature. High profile performance failures such as Healthcare.gov and Examsoft are neither accidental nor unavoidable. They are the result of failure to comprehend, failure to plan, and failure to commit to a set of defined performance requirements. Based on a real-world case study, The Black Art of Performance Requirements presents a process for defining SMART performance requirements in co-operation with business, developers, and analysts. Taking two years of production performance data and connecting with end-user performance complaints during that time frame, this session demonstrates the effectiveness of defining requirements using this process and explores how to objectively evaluate system performance against those requirements.

1. Understand why users don't understand performance requirements, and learn how to define quality SMART performance requirements that will satisfy them anyway.
2. Learn how to obtain buy-in from business, developers, and analysts for meeting performance targets and resolving performance problems.
3. Learn how to measure, evaluate, and compare performance results against targets objectively.

For more information about performance requirements, testing, and how to improve your chances at project success, please see MNP Consulting - Performance Management. 

ICPE 2015, Getting Published with ACM, and Performance Management

It has been a busy year so far.

For much of 2014 I had been working on some research around Web Page Performance and understanding users expectations, what the industry believed about response times and how fast was 'fast enough', and how that compared to the real world. What I ended up finding is that much of the research that had been done in this field was, frankly, worthless. Oh, there were good points to be found, some useful results, and valuable nuggets to be unearthed - but on the whole most results drew conclusions that were being interpreted far to generally or their methodology was deeply flawed.

In short, there was a gaping hole that was asking to be filled. And the question was, how fast is 'fast enough'?

The most valuable conclusions I could glean from published works were as follows:

1. Faster is better
2. For systems that don't need to rely on real-time feedback, <1s is fast enough to be seamless for most request-response human-computer interactions be it locally or over the web
3. People will happily wait longer if they are given responsive feedback while they are waiting
4. Consistency is important

But from a practical perspective, what kind of performance level should I be aiming for? Is 5s fast enough? What about 3s? Is average response time sufficient, or do we need to use something stricter?

These kinds of questions led me to doing the research that eventually resulted in writing a paper that in September 2014 I submitted and was accepted by the International Conference on Performance Engineering (ICPE 2015).

That research paper is now officially available online at the MNP Library Defining Standards for Web Page Performance (also available via the ACM Digital Library with the following Citation Link).

In February of 2015 I then got to travel to Austin TX to present my paper at the conference. I will post my thoughts about the conference and the trip in an upcoming post, but in short it was a wonderful experience, I got to meet a lot of amazing people, the presentations were fantastic and I learned quite a bit.

Now, I am pursuing expanding our service offerings as part of MNP's Technology Consulting Team by officially introducing Performance Management.

Performance Management is all about improving the success rate of projects with a technology component. Performance is often a consideration that exists only as an afterthought on projects, something that can be taken care of after the important functional work has been completed. That very thinking has led to some spectacular failures in the real world notably including the 2014 Examsoft debacle, the 2013 launch of Healthcare.gov, the 2008 Edinburgh Fringe Festival ticketing system failure, and the 2007 scrapping of the new UK General Register Office System.

Performance Management is about planning for success, it is about helping organizations invest in quality and avoid making the mistakes of the past. This is something that has become a bit of a passion of mine, a purpose, and I intend to spend a lot more time using this space to talk about performance in the future.

RPT Custom Code - Ensuring Unique Logins for Looping Tests

A problem I have encountered while performance testing business applications with Rational Performance Tester is the uniqueness of logins. Often, business applications contain logic that will prevent duplicate logins from multiple sources at the same time, or will have workflow control or session replication or persistence that will result in interference if the same login is being used by more than one thread at the same time. A quick workaround to this problem can be to ensure there is a large pool of available logins to reduce the possibility of duplicates, however this is not always possible if a system is using Active Directory or LDAP, or if it SSO enabled. So the challenge being faced is, how do we bind a unique login to each virtual user thread in our test for the duration of the test?

The first approach would be to adjust the settings on your login datapool. When you add a datapool to your test an option exists to "Fetch only once per user".

In theory, this would be sufficient to ensure each thread will have a unique login. However in practice it seems to be not quite so simple. In a test configuration that uses multiple agents "Open mode" must be set to "Segmented" otherwise each agent will have a complete copy of the same list of accounts, resulting in duplication. In order to use Segmented in this manner though, your datapool must be larger than the number of threads in order to ensure sufficient rows are available in each segment. (IBM recommends you have 2x the number of records as threads to ensure balanced segmentation).

Despite the theory, I have run into the problem of threads exiting prematurely in a multiple user group/multiple agent/infinite loop test configuration with the error message "End of datapool reached". This is not an error we should be seeing. Reviewing the saved response data demonstrated that each thread was correctly using a single unique login, but somehow the error was consistent.

While attempting to debug the issue, I tried setting the "Wrap when last row is reached" property. Although successful in preventing threads from exiting prematurely, the wrap property appears to override the fetch-once property, returning me to a state of duplicate logins. Unfortunately, IBM's documentation does a poor job of explaining how each of these datapool properties interact with one another, so in order to overcome this issue I turned to writing my own piece of custom code to manage my logins.

The following custom code solution binds threads to a specific login on first access, and thereafter will always return the same login identifier for each subsequent request. It also segments the login map into groups that can be manually accessed (by passing the group name as the first parameter) or automatically by setting the User Group Name in your schedule.

This code is simplified and has a few limitations.

1. Distinct Agent Segments - Each Agent must use a distinct User Group Name because static objects are not shared in memory between agents. If two agents are assigned to the same User Group then duplicate logins will occur.
2. Configurability - The segments and login lists are hardcoded in this code segment, this could be overcome by adding an option to read logins from a file (or other data storage)
3. Order - Logins will always be returned in the same order for each subsequent test run, a piece of randomization code would allow them to be shuffled.

package export; 

import java.util.Arrays; 
import java.util.HashMap; 
import java.util.List; 
import java.util.Map; 

import com.ibm.rational.test.lt.kernel.IDataArea; 
import com.ibm.rational.test.lt.kernel.services.ITestExecutionServices; 
import com.ibm.rational.test.lt.kernel.services.ITestLogManager; 
import com.ibm.rational.test.lt.kernel.services.IVirtualUserInfo; 
import com.ibm.rational.test.lt.kernel.services.RPTCondition; 

/** 
 * !!Warning!!: This code is NOT agent safe. Each unique set of segmentation identifiers MUST be isolated to a single agent for this code to work correctly 
 * @author grempel 
 */ 
public class UniqueLoginManager implements 
                com.ibm.rational.test.lt.kernel.custom.ICustomCode2 { 
        
        private static Map<String, List<String>> loginsBySegment = new HashMap<String, List<String>>(); 
        private static Map<String, String> loginsByThread = new HashMap<String, String>(); 
        private static Map<String, Integer> indexBySegment = new HashMap<String, Integer>(); 
        private static boolean initialized = false; 

        /** 
         * Initializes static login maps by segmentation 
         */ 
        private synchronized void init() { 
                if(!initialized) { 
                        loginsBySegment.put("GRP1", Arrays.asList(new String[]{"grp1_login1","grp1_login2","grp1_login3"...}));
                        loginsBySegment.put("GRP2", Arrays.asList(new String[]{"grp2_login1","grp2_login2","grp2_login3"...}));
                        loginsBySegment.put("GRP3", Arrays.asList(new String[]{"grp3_login1","grp3_login2","grp3_login3"...}));
                        indexBySegment.put("GRP1", 0); 
                        indexBySegment.put("GRP2", 0); 
                        indexBySegment.put("GRP3", 0); 
                        initialized = true; 
                } 
        } 
        
        /** 
         * Instances of this will be created using the no-arg constructor. 
         */ 
        public UniqueLoginManager() { 
        } 
        
        /** 
         * Returns the previous login if the thread has previously requested a login during this test execution. Otherwise retrieves the next login from the list, binds it to the thread, and returns it. 
         * @param segmentId 
         * @param thread 
         * @return String.class login 
         */ 
        public synchronized String getLogin(String segmentId, String thread) { 
                if(loginsByThread.containsKey(thread)) { 
                        return loginsByThread.get(thread); 
                } else { 
                        init(); 
                        List<String> logins = loginsBySegment.get(segmentId); 
                        Integer index = indexBySegment.get(segmentId); 
                        if(logins!=null && logins.size()>0 && index<logins.size()) { 
                                String login = logins.get(index); 
                                indexBySegment.put(segmentId, index+1); 
                                loginsByThread.put(thread, login); 
                                return login; 
                        } else { 
                                //fail 
                                return null; 
                        } 
                } 
        } 

        /** 
         * Generate and manage unique logins across multiple user threads and user-defined segments. 
         * Warning: This class is not thread-safe for a segmentation identifier that is distributed across multiple agents. Each segmentation identifier 
         * must exist on a single agent to avoid login duplication. 
         * @param String.class[] args - arg0 = segmentation identifier (optional, if not included will use the UserGroupName as segmentation identifier) 
         */ 
        public String exec(ITestExecutionServices tes, String[] args) { 
                String segmentId = null; 
                ITestLogManager tlm = tes.getTestLogManager(); 
                
                IDataArea dataArea = tes.findDataArea(IDataArea.VIRTUALUSER); 
                IVirtualUserInfo virtualUserInfo = (IVirtualUserInfo)dataArea.get(IVirtualUserInfo.KEY); 
                String user = virtualUserInfo.getUserName(); 
                
                //If arg[0] not provided, use the UserGroupName as the segmentation identifier 
                if(args.length<1) { 
                        segmentId = virtualUserInfo.getUserGroupName(); 
                } else { 
                        segmentId = args[0]; 
                } 
                
                String login = getLogin(segmentId, user); 
                if(login==null) { 
                        tlm.reportErrorCondition(RPTCondition.CustomCodeAlert); 
                        return null; 
                } 
                
                return login; 
        } 
}