Breaking the Chains of Analysis Paralysis - The Black Box (Part 1 of 7)

This is the first article in a series of 7 that I will be putting up in the coming weeks about Analysis Paralysis. The purpose of these articles is to highlight primary causes of AP among people who play games, and discuss strategies that you can apply as a designer to reduce the potential for AP.

Each article in this series will go into depth on one potential cause of AP, examples of situations where it can be problematic, and present some solutions that can improve the situation.

Although these articles are specific to Boardgames, the concepts of Analysis Paralysis apply widely to everyday life. These specific examples can easily be translated to apply to systems in general, including business systems and enterprise reporting & dashboards.

1. The Black Box
2. The Paradox of Choice – Dec 13
3. The Prisoner’s Dilemma – Dec 20
4. The Maze to Victory – Dec 27
5. Relationship Status: It’s Complicated – Jan 3
6. Sophie’s Choice – Jan 10
7. Bigger vs Better – Jan 17

Additional articles will be released weekly, we hope you will read, share, and comment on them. Tell me what you think, and how you might apply some of these concepts and solutions to your business.

The first article discusses The Black Box, the problem of information obscurity. Read Analysis Paralysis - The Black Box

Gotcha #7 - AppDynamics and the Mysterious Case of the Vanishing Cookie

Not much of a mystery today in this article (the answer is alluded to in the title) but the process of identifying the problem, discovering why it was a problem, and then fixing it was quite a challenge.

Today's Gotcha is about Rational Performance Tester (v8.5.1.2 for anyone interested) and a vexing problem with successful test execution.

The problem originally arose when we discovered that our tests weren't always succeeding. After executing a schedule we would consistently have a large number of pages that wouldn't have 100% success status codes (located on the Server Health Detail view). There would be a few with 100% success rate, but consistently nearly every page was hovering between 94 and 98%. In some larger tests this would drop further, into the 80's or high 70's, but even a simple low volume test would have most pages at less than 100%.

To get a better handle on why and when these page errors were happening, I added a new counter to the default Page Throughput / Page Hit Rate graph (Add/Remove Performance Counters --> Page Performance Counter... --> Status Code Successes --> Percent Page Status Code Success [for Interval]).

Suddenly we could see this wavering line appear on the graph that would start off at 100%, dip sharply at the beginning of the test, before returning back up near 100% and stay fluctuating in the high 90% range.

Baffling when the very same test scripts in the past against the same application would run at 100% without trouble. Doubly-baffling when running the script as a standalone would work perfectly fine.

Following the Clues

The first clue came when we decided to rerun one of our test schedules and monitor the Test Execution of a user thread. Since the failures were reasonably consistent across all pages, chances are we might be able to catch it in the act.

After a couple of hours of watching successful page responses, we caught one. Right from the beginning of a new test script - a page failure. And then another, and another, and so on until every single page in that script failed in order. And the next script. And the next... but then the fourth script, for no apparent reason, began returning successful responses again.

Digging into the error, the only message we could find was a "Failed substitution ^csrfToken=.*$" message followed by a cascade of 500 - Server Error responses. The csrfToken appears in the HTTP Response header as a "Set-Cookie" value from the server (you can find an explanation of its purpose here). And for some reason, it wasn't being found.

Except... inspecting the response of that very first page - the csrfToken was right where it should be. Further digging into the application logs showed the same thing - on occasion our test would be sending an invalid csrfToken - the recorded token - which happens when the substitution couldn't be made.

But it was right there in the Response Header - right where we expected it to be. Except, not quite...

Smoking Gun

It looked like it was right where it was supposed to be - in among a half-dozen other cookies being set by the server. But a careful comparison with the original recording showed that it was 1 line down. There was a NEW Set-Cookie value that was inserted in the list just ahead of the csrfToken.

Inspecting and comparing successful test executions with failed ones showed that this new mysterious cookie only appeared in the failed tests, and wasn't to be found in the successful ones.

The server was randomly inserting a new cookie in its response... some of the time. And it was enough, by shifting the order of the Set-Cookie headers for RPT fail when trying to identify the csrfToken.

The culprit was a cookie called ADRUM_BTs and it looks like the following:

Set-Cookie: ADRUM_BTs="R:0|s:f"; Version=1; Max-Age=30; Expires=...date...; Path=/; Secure

There was no consistency in when or where it would appear. It would show up for a script execution or three, then disappear once more.

Solution

It turns out that we weren't the only ones to run the Mysterious Case of the Vanishing Cookie - other people had identified ADRUM_BTs as playing havoc with their scripting efforts.

The cookie in question belongs to AppDynamics End User Monitoring javascript package, a tool that can be used to monitor user behaviour and web application performance from the user's perspective. It was a new feature in the AppDynamics package that had been turned on in our Performance Testing environment recently to try and improve our measurement capabilities.

In the end, there was no way to remove this new cookie or adjust the system to consistently distribute it. In order to get our Performance Testing back up and running we had to remove all End-user Monitoring components - and suddenly our vanishing cookie was gone for the last time.

Gotcha #6 - Sometimes Naive Caching is Worse than None

There is a popular proverb about good intentions and the road to hell that is well known, but an alternate version of that proverb is phrased as "Hell is full of good meanings, but Heaven is full of good works". Unfortunately even good works can lead down the wrong path when they are undertaken with insufficient consideration for their impact.

This post is about one such work - about the implementation of a caching strategy on a large platform that is widely used across the industry. The work itself is good, smart, and done with significant forethought. However under certain conditions, the choices made ended up creating a system worse than if nothing had been done at all. This post explores the choices, circumstances, and effect that implementing an intelligent - but naive - caching system had on one particular project and why the result ended up being worse than if it had never been.

The names of all organizations, people, and software involved have been withheld.

6 Days to Go-Live

A war room had been convened. The production deployment of a new Public Access System (PAS) had gone smoothly over the weekend and everything was prepared to turn the switch and make it accessible in less than a week. The public information campaign had already been underway for some time, advertising the date, and getting the word out that this new offering was forthcoming. However, validation of the deployment had hit a snag - one member of the QA team had noticed something wrong.

At a critical step while making a new purchase on the PAS, the system would appear to pause. After a long delay (45s) it would return with a technical error. Repeating this process several times in a row would produce the same error across multiple sessions, across multiple machines. And then it would go away and everything would work smoothly again.

Until it happened again about an hour later.

And again an hour after that.

And then it went away. Combinations of testing, several people trying to test it in a variety of ways, and using precisely the same data - or different data - it couldn't be reproduced.

Until the next morning, when the problem showed up anew and again the following hour, before disappearing once more.

So began an urgent search to understand the problem began, testing the system, combing the logs, reviewing all the code tracing and analytic tools at our disposal to isolate and resolve the problem - in production.

Underlying this effort was the question - why wasn't this problem seen in any other environment? Through the myriads of Dev and QA environments, through multiple test cycles and performance testing, this pattern of problems had never been encountered before. At times a similar issue would be seen in isolated circumstances immediately following the restart of an environment, but once it was initialized everything was good.

So why was it happening in production, and nowhere else?

Architecture

The PAS is a clustered three-tier system using RESTful services to communicate between the web server and the Back-end Application (BEA). The BEA contains all the business code required to verify, validate, and persist purchase information.

The PAS cluster contains 2 nodes, and the BEA cluster contains 8, separated into two group of 4 - one group dedicated to the PAS and the other for direct Intranet access. As a true RESTful interface, no session information is tracked between requests to the BEA cluster. The load balancer provides a true round-robin.

When the system is active, most of the processing load takes place on the BEA cluster, the PAS operates as little more than a view rendering layer. When a page request comes in to the PAS, it opens a connection to one of the BEA nodes - passes along the request - and waits for a response. In the case where the BEA node takes an abnormally long time, or if there is a failure and no response is generated - the connection has a 45s timeout. If it takes longer than 45s, the PAS would close the connection and display a technical error message encouraging the customer to call their local company representative to complete the purchase.

The Trail of Evidence
When the problem was first raised, there was a great deal of speculation as to the possible cause. Database issues, connection issues, misconfiguration, load balancer issues - all possible sources of this technical error, but all things that would need to be verified and eliminated.

There were a number of observations made about the system that were unusual or worth investigating - including hung database threads and the presence of a query in the AWR (query performance statistics) report that was known to be problematic in other environments. But in hindsight there was only one that would prove to be important to the actual cause: the problem only happened in the morning when full-time employees arrived for work and began using the BEA system directly via the Intranet.

Now the Intranet BEA nodes are physically separate from the ones used by the PAS, but they do share a database. The problem only occurred during the morning hours when load on the BEA nodes and the database were at their highest. Not high enough to cause a bottleneck or result in performance problems - but not insignificant either. More on this later.

The most important piece of information came from our enterprise performance monitoring tool that was installed on all production servers. By drilling down, identifying, and isolating a single request that failed we could examine the performance of the BEA code for that single request.

And what we discovered was a function call that took 24s that involved retrieving a large but critical piece of reference data from the database, and storing it in the application cache. By itself, not enough to cause a timeout and result in a technical error - but in a confluence of circumstances this piece of reference data was then being used for a large and complex set of calculations that took ~20s to complete and in this particular instance was just barely slow enough to cross the 45s timeout threshold and cause the technical error to be displayed. 

Now this isn't unusual - the median time for this particular load-and-store operation was about 15-18s, and once it was complete the data would be cached and any future accesses would take 1/10th of a second or less. But in the situation where it did have to retrieve the data from the database while the db was under significant load caused it to slow just enough to trigger the problem.

But why was it happening at all? Why wasn't it happening elsewhere? And why wasn't it discovered during Performance Testing.

Unexpected Caching

The answer came in a conference call to the vendor. The cache on the BEA system operated as follows:

1. Objects in the cache are marked as stale at intervals of 15 minutes if they have not been accessed during that timeframe.
2. If an object was already marked as stale when the interval arrived, it would be evicted from the cache.
3. Every 4th interval (once per hour) all objects in the cache would be evicted regardless of their state.

This cache interval was configurable to a maximum of 30 minute intervals, extending the hourly global eviction from 1 hour to 2 hours - but no further. This means that every hour all reference data would be purged from the BEA system's memory and need to be reloaded the next time it was accessed... on each node.

The Core Problem

In order for the problem to occur, the following had to happen:

1. The BEA's cache was empty on the nodes that were used by the PAS.
2. The other Intranet BEA nodes had to be actively used by a large number of concurrent users to put sufficient load on the database to cause a retrieval delay for the large amount of reference data.
3. The PAS had to be inactive, with only 1 or 2 users online performing a new purchase. Other operations would cause the reference data to be loaded but not perform the complex calculations that would cause the response delay to exceed 45s.

An unlikely scenario it seems, right?

Except...

1. The BEA cache would be emptied on the hour, every hour.
2. The Intranet BEA nodes were being used by employees for their normal job, and would be heavily used every weekday for several hours during the morning.
3. The PAS was a limited-scope rollout, the only functions that it performed were New Purchase, and Continue a Saved Purchase.
4. The PAS was so new, that the initial expected load was only a handful of users per day. So the likelihood of having only 1-2 users performing a new purchase was very high.

These circumstances meant that it was very likely that any user of the system would run into the situation where all three conditions were met and cause their purchase attempt to fail.

This problem was further compounded by the round-robin load balancer, and the fact that the reference data needed to be loaded on each node separately. This means that not only was a user likely to encounter the technical error message, but if they attempted to retry their purchase their next page request would be directed to a different BEA node whose cache was still empty resulting in the same error message again.

The expected result is that for a majority of potential customers, each would need to retry their purchase five times before successfully completing it.

Why Wasn't This Caught?

The simple answer is the other environments were not busy enough. Dev, QA, Staging... all of these environments did not have sufficient database usage to cause enough delay to cross the 45s delay threshold. In addition, these environments typically were not configured in the dual-cluster architecture of production, so any activity directly on a BEA node would cause the reference data to be loaded into cache. The chances of a PAS new purchase occurring with an active database were essentially non-existent.

The exception was the Performance Testing environment which did have a fully configured production system, and sufficient database traffic during a test to cause the problem. The reason it wasn't discovered was that the PAS traffic was too high. Even a half-dozen simultaneous users was sufficient to cache the reference data across all the BEA nodes - and even if the error was observed near the beginning of the test once or twice, a few minor functional errors are not unexpected during a performance test and the absence of reoccurence over the next hour would be simply dismissed as an anomaly.

As soon as the circumstances of the problem were discovered, it became a simple matter to intentionally reproduce in Performance Test by adding load to the Intranet BEA nodes and manually attempting to test a new purchase.

Why Did This Happen?

This problem is caused directly by the implementation of a naive caching framework on the BEA system. The purpose behind clearing the cache regularly would be to ensure that reference data (if updated) could only be stale for a certain amount of time before being refreshed. The assumption was that the cache could be cleared every hour without causing a problem, which ended up not being the case. The BEA system had no facility with which to preload reference data, no functionality allowing it to be refreshed periodically except through eviction and reload on-demand, and no ability to persist reference data in a long-term no-expiry cache.

The reason this was more than just a normal problem, is that by implementing a limited cache the system was successfully able to mask the existence of the problem from developers, qa, and performance testing. The naive caching strategy allowed it to work well under load most of the time, and only appear as a problem in very specific circumstances.

Had there been no cache on the BEA system, this problem would have become abundantly clear in performance testing by resulting in terrible performance for that operation and also causing the error rate to approach 100%. But by masking the issue it managed to survive until mere days before go-live when it was discovered entirely by accident.

The Solution

The permanent solution will be to modify the BEA system to build a more sensible multi-use caching strategy with the ability to preload and periodically refresh reference data independent of a user request. Temporarily it has been resolved with an automated process that makes an appropriate service request to each BEA node every minute that triggers the reference data reload if it has been evicted in an attempt to complete that reload before a user encounters it.

Gotcha #5 - Network Performance, CSS, and HTML5

Today's article comes courtesy of understanding your customer's environment, and that emphasizing style over substance can break the bank.

Our story starts with an urgent request for assistance from a system administrator, a minor application update has resulted in a nearly five-fold increase in total network traffic at a remote office. On Monday morning, when the morning shift arrived network traffic began to spike. It grew, and grew, far beyond normal traffic levels for the office and stayed there, night and day, until the last shift signed off on Friday evening.



Think about this for a moment.

The application's network traffic did not increase five-fold. The entire office's network traffic consumption increased from ~2Mbps (24-hour average) to nearly 9Mbps (24-hour average) with peaks topping out at 10Mbps.

The first stage is always denial.

How could a minor update to a server-based web application that mostly consisted of a couple minor bug fixes, one new summary screen, and some UI tweaks be the problem? It must be something else, there isn't anything new or unusual in the application update.

Well...

Step 1: Isolating the Problem
Easy enough to do, by using more detailed reports we could see the initial spike in unusual network activity began at 10am on the Friday before, the same time as the outage window when the application update was deployed.

Also, there were no other maintenance activities ongoing at that time, no file transfers, and no other system updates scheduled or otherwise.

Hmm...

Step 2: Reproducing the Issue
To confirm, we conducted a test where all instances of the application on every workstation on the site was closed, and by so doing network traffic returned to its normal (low) stable state.

One by one, each workstation re-loaded the application and by time the last workstation was reconnected the network traffic was back at its five-fold peak. It was during this process of re-loading that the system administrator noticed something unusual. A new UI feature had been added to some screens, when a specific piece of data was in a warning state - the text box would flash orange. And every time a new screen loaded that happened to display one of these flashing orange boxes, network traffic would spike sharply.

In the end, the system administrator noticed 10 out of more than 100 workstations happened to be displaying this flashing orange box. Returning to shut down just these 10 flashing displays suddenly dropped network traffic back to normal once more. When they were re-loaded, network traffic would spike once more.

Step 3: Cause and Solution
So how can a simple CSS and HTML5 flashing text box on 10 screen possibly start pushing 4 times the total network traffic of an office of more than 100? The technique to highlight a text box had been used elsewhere in the application already, and the style control was rendered entirely in-browser with no additional server requests being made.

How can a box no larger than 400px by 300px with a lovely fade-in/fade-out HTML5 animation possibly consume 800Kbps per screen?

The Epiphany

Animated. For the first time in this application, the developer (to satisfy a requested requirement) used a simple animation cycle to flash the control orange on a 4-second cycle. It was a lovely effect, understated, but unmistakable and was easily visible when displayed on a large screen from across the room.

What wasn't clear however, was that each workstation did not use a browser to view the application. Instead, each workstation used a Remote Desktop (RDP) session to a centralized (offsite) server which then connected to the application in a browser window.

Instead of being rendered in a local browser session and consuming no network resources at all, the beautifully animated flashing textbox was being rendered and streamed as uncompressed full-motion video over the network to the site office.

10 workstations streaming uncompressed video for 24-hours a day, 5 days a week.

Step 4: Resolution
In the end the problem was easy to solve. Rip out the animation, and using a simple steady-state orange highlight to communicate the information.

But it was an important lesson. You must be aware of how your customer is using the tools that you build, and how the simplest of design choices can cause a major impact under the wrong circumstances.

Research Study: Memory Test

Contest!

Anyone who signs in with an email address and completes the test before Feb 28 will be entered into a drawing for one of the top 2015/2016 indie games from Steam or an equal value Amazon book order.

Participate Here!

I'm conducting a study on memory and the ability for people to memorize short sequences of numbers. In order to do this I've built a small web app to conduct memory tests. It tracks both correct answers and time spent and scores your performance - so speed is better but not at the expense of answering incorrectly.

The purpose of this study is to test your short term memory skills, numeric analysis ability, and speed. The test is divided in 5 levels of 5 questions that increase in difficulty. You may complete them in any order you choose and at any time if you would like to do some now and return later to do the rest, but you must complete them all before the results can be processed.

The test is randomly generated, so you can take it as many times as you'd like to try and do better, and it tracks your history so you can see you best score if you do it more than once. Each question presents you with a small set of numbers to memorize, then asks you to answer a question about the numbers.

It's very simple, and quick to do. Please give it a try, and share it around - compete against your friends!

For SCIENCE!

RPT Custom Code - Ensuring Unique Logins for Looping Tests

A problem I have encountered while performance testing business applications with Rational Performance Tester is the uniqueness of logins. Often, business applications contain logic that will prevent duplicate logins from multiple sources at the same time, or will have workflow control or session replication or persistence that will result in interference if the same login is being used by more than one thread at the same time. A quick workaround to this problem can be to ensure there is a large pool of available logins to reduce the possibility of duplicates, however this is not always possible if a system is using Active Directory or LDAP, or if it SSO enabled. So the challenge being faced is, how do we bind a unique login to each virtual user thread in our test for the duration of the test?

The first approach would be to adjust the settings on your login datapool. When you add a datapool to your test an option exists to "Fetch only once per user".

In theory, this would be sufficient to ensure each thread will have a unique login. However in practice it seems to be not quite so simple. In a test configuration that uses multiple agents "Open mode" must be set to "Segmented" otherwise each agent will have a complete copy of the same list of accounts, resulting in duplication. In order to use Segmented in this manner though, your datapool must be larger than the number of threads in order to ensure sufficient rows are available in each segment. (IBM recommends you have 2x the number of records as threads to ensure balanced segmentation).

Despite the theory, I have run into the problem of threads exiting prematurely in a multiple user group/multiple agent/infinite loop test configuration with the error message "End of datapool reached". This is not an error we should be seeing. Reviewing the saved response data demonstrated that each thread was correctly using a single unique login, but somehow the error was consistent.

While attempting to debug the issue, I tried setting the "Wrap when last row is reached" property. Although successful in preventing threads from exiting prematurely, the wrap property appears to override the fetch-once property, returning me to a state of duplicate logins. Unfortunately, IBM's documentation does a poor job of explaining how each of these datapool properties interact with one another, so in order to overcome this issue I turned to writing my own piece of custom code to manage my logins.

The following custom code solution binds threads to a specific login on first access, and thereafter will always return the same login identifier for each subsequent request. It also segments the login map into groups that can be manually accessed (by passing the group name as the first parameter) or automatically by setting the User Group Name in your schedule.

This code is simplified and has a few limitations.

1. Distinct Agent Segments - Each Agent must use a distinct User Group Name because static objects are not shared in memory between agents. If two agents are assigned to the same User Group then duplicate logins will occur.
2. Configurability - The segments and login lists are hardcoded in this code segment, this could be overcome by adding an option to read logins from a file (or other data storage)
3. Order - Logins will always be returned in the same order for each subsequent test run, a piece of randomization code would allow them to be shuffled.

package export; 

import java.util.Arrays; 
import java.util.HashMap; 
import java.util.List; 
import java.util.Map; 

import com.ibm.rational.test.lt.kernel.IDataArea; 
import com.ibm.rational.test.lt.kernel.services.ITestExecutionServices; 
import com.ibm.rational.test.lt.kernel.services.ITestLogManager; 
import com.ibm.rational.test.lt.kernel.services.IVirtualUserInfo; 
import com.ibm.rational.test.lt.kernel.services.RPTCondition; 

/** 
 * !!Warning!!: This code is NOT agent safe. Each unique set of segmentation identifiers MUST be isolated to a single agent for this code to work correctly 
 * @author grempel 
 */ 
public class UniqueLoginManager implements 
                com.ibm.rational.test.lt.kernel.custom.ICustomCode2 { 
        
        private static Map<String, List<String>> loginsBySegment = new HashMap<String, List<String>>(); 
        private static Map<String, String> loginsByThread = new HashMap<String, String>(); 
        private static Map<String, Integer> indexBySegment = new HashMap<String, Integer>(); 
        private static boolean initialized = false; 

        /** 
         * Initializes static login maps by segmentation 
         */ 
        private synchronized void init() { 
                if(!initialized) { 
                        loginsBySegment.put("GRP1", Arrays.asList(new String[]{"grp1_login1","grp1_login2","grp1_login3"...}));
                        loginsBySegment.put("GRP2", Arrays.asList(new String[]{"grp2_login1","grp2_login2","grp2_login3"...}));
                        loginsBySegment.put("GRP3", Arrays.asList(new String[]{"grp3_login1","grp3_login2","grp3_login3"...}));
                        indexBySegment.put("GRP1", 0); 
                        indexBySegment.put("GRP2", 0); 
                        indexBySegment.put("GRP3", 0); 
                        initialized = true; 
                } 
        } 
        
        /** 
         * Instances of this will be created using the no-arg constructor. 
         */ 
        public UniqueLoginManager() { 
        } 
        
        /** 
         * Returns the previous login if the thread has previously requested a login during this test execution. Otherwise retrieves the next login from the list, binds it to the thread, and returns it. 
         * @param segmentId 
         * @param thread 
         * @return String.class login 
         */ 
        public synchronized String getLogin(String segmentId, String thread) { 
                if(loginsByThread.containsKey(thread)) { 
                        return loginsByThread.get(thread); 
                } else { 
                        init(); 
                        List<String> logins = loginsBySegment.get(segmentId); 
                        Integer index = indexBySegment.get(segmentId); 
                        if(logins!=null && logins.size()>0 && index<logins.size()) { 
                                String login = logins.get(index); 
                                indexBySegment.put(segmentId, index+1); 
                                loginsByThread.put(thread, login); 
                                return login; 
                        } else { 
                                //fail 
                                return null; 
                        } 
                } 
        } 

        /** 
         * Generate and manage unique logins across multiple user threads and user-defined segments. 
         * Warning: This class is not thread-safe for a segmentation identifier that is distributed across multiple agents. Each segmentation identifier 
         * must exist on a single agent to avoid login duplication. 
         * @param String.class[] args - arg0 = segmentation identifier (optional, if not included will use the UserGroupName as segmentation identifier) 
         */ 
        public String exec(ITestExecutionServices tes, String[] args) { 
                String segmentId = null; 
                ITestLogManager tlm = tes.getTestLogManager(); 
                
                IDataArea dataArea = tes.findDataArea(IDataArea.VIRTUALUSER); 
                IVirtualUserInfo virtualUserInfo = (IVirtualUserInfo)dataArea.get(IVirtualUserInfo.KEY); 
                String user = virtualUserInfo.getUserName(); 
                
                //If arg[0] not provided, use the UserGroupName as the segmentation identifier 
                if(args.length<1) { 
                        segmentId = virtualUserInfo.getUserGroupName(); 
                } else { 
                        segmentId = args[0]; 
                } 
                
                String login = getLogin(segmentId, user); 
                if(login==null) { 
                        tlm.reportErrorCondition(RPTCondition.CustomCodeAlert); 
                        return null; 
                } 
                
                return login; 
        } 
}

RPT Custom Code - Saving Response Data

How to build datapools by extracting generated data from a system using custom code in Rational Performance Tester.

A major challenge that I encounter consistently as a Performance Engineer is priming a system with known, consistent data that I can feed into datapools and use for testing. There are a variety of ways to accomplish this: conversion processes, generation tools, database scripts, restore/upgrade processes, etc. However, a project that I have been working on recently has a variety of issues with these strategies. The database is automatically generated and nearly impossible to decipher, not to mention that it changes with each new code release. The second issue is that the data generation tools are not being kept up to date by the developers, so they are usually broken with each new release. These issues have led me to the point of using my performance testing tool (Rational Performance Tester v8.5.2) as a workaround data priming tool.

This plan has its own challenges, primary identifiers are all system generated - and the only way I can reliably access the data once it is generated is using those primary identifiers. Granted, there are ways to obtain this information once it exists in the system, but they involved days if not weeks of effort to extract and cleanse the data into a form that can be used effectively, and would require significant assistance from a developer to do so. A better way would be to process and extract the desired information from the responses themselves as it is being generated by RPT.

Currently RPT does not have any simple write-back mechanism for references during a test. There may exist a way using references and variables - but if it does it is well hidden beneath the primary strengths of the tool, which are load generation, data substitution, and response analysis. This led me to consider the question "How would I enhance the tool to let me extract response data?". Since RPT is an Eclipse-based tool and supports the addition of custom code and plug-ins it should be entirely feasible to write some code to extract whatever I wanted from it.

So that is what I did - the example and the code presented below is a very simple, bare-bones way of extracting a reference from a page response, passing it into a block of custom code, and writing it to a CSV file in a thread-safe manner that will allow you to import the generated CSV back into your project as a datapool source. There are dozens of ways I can think of to use this code and enhance it if you want to write to a database, do data transformation, correlating reference data with existing datapools, or producing multiple outputs - but this should be sufficient to get you started if you are encountering problems bridging that gap from theoretical to practical.

Step 1: Create a Custom Code Class
With your recorded test open, right-click your top-level test element and select "Add --> Custom Code".

Selecting the new Custom code element will show the Custom Code element details in the right-side pane that lets you enter a Class name (in the format: "package.Class"). If this is the first time you have created a Custom Code class - click the "Generate Code" button to create a basic default class implementation. If you already have the Custom Code class you want to use, skip down to Step 4 for adding an existing Custom Code element to your test case.

This will generate a simple Custom Code class that you can start using in your test.

package export; 

import com.ibm.rational.test.lt.kernel.services.ITestExecutionServices; 

/** 
 * @author unknown 
 */ 
public class Test implements 
                com.ibm.rational.test.lt.kernel.custom.ICustomCode2 { 

        /** 
         * Instances of this will be created using the no-arg constructor. 
         */ 
        public Test() { 
        } 

        /** 
         * For javadoc of ICustomCode2 and ITestExecutionServices interfaces, select 'Help Contents' in the 
         * Help menu and select 'Extending Rational Performance Tester functionality' -> 'Extending test execution with custom code' 
         */ 
        public String exec(ITestExecutionServices tes, String[] args) { 
                return null; 
        } 


}

To organize my library of custom code, I created a new Performance Test Project called "CustomCodeReference". If you create this project as a Performance Test Project it will automatically include all of the required RPT libraries in its build path. This will also allow me to export the project into a .jar file that I can import into other projects, or simply reuse it as a dependency in other Performance Testing projects as needed.

Step 2: Create a Singleton File Writer
Create a new class file that will be used by our custom code as the singleton file writer to manage file creation, access, and synchronization between threads. In this example I have created the class "export.ExportAdministrator" in my CustomCodeReference project to be my singleton file writer class. A suggested enhancement to this structure would be to create a dedicated Factory class that returns an instance of an interface to allow for multiple optional implementations such as a CSV file implementation, an XML file implementation, or a database implementation.

package export;

import java.io.File;
import java.io.FileWriter;
import java.text.DateFormat;
import java.text.SimpleDateFormat;

/**
 * @author grempel
 * @date 2014-03-27
 *
 * Singleton class with factory to manage thread-safe file export
 * Usage:
 *   Call ExportAdministrator.getInstance() to obtain object reference
 *   Use instance write() or writeln() to export to file
 */
public class ExportAdministrator {
private static ExportAdministrator instance = null;
private FileWriter fw = null;

public ExportAdministrator() {
//Build string writer
DateFormat df = new SimpleDateFormat("yyyy-MM-dd HH-mm-ss");
File file = new File("C:\\rpt-log\\" + df.format(System.currentTimeMillis()) + " output.csv");
try {
fw = new FileWriter(file);
} catch (Exception e) {
e.printStackTrace();
}
}

/**
* Retrieve singleton instance of writer.
*/
public static synchronized ExportAdministrator getInstance() {
if(instance==null) {
instance = new ExportAdministrator();
}
return instance;
}

/**
* Write string to file
*/
public synchronized String write(String str) {
if(fw!=null) {
try {
fw.write(str);
fw.flush();
return "true";
} catch (Exception e) {
return e.toString();
}
}
return "failed";
}

/**
* Write string to file and start a new line
*/
public synchronized String writeln(String str) {
if(fw!=null) {
try {
fw.write(str);
fw.write("\r\n");
fw.flush();
return "true";
} catch (Exception e) {
return e.toString();
}
}
return "failed";
}

/**
* Clean up open file references
*/
@Override
protected void finalize() throws Throwable {
if(fw!=null) {
fw.close();
}
super.finalize();
}

}

This class will generate a timestamped output file in the folder "C:\rpt-log\" on each agent that is running the test.

One note to make about the usage of this class. If you are using more than 1 virtual user to generate your data, you are best to compile the entire set of data you want to write to the file into 1 string before calling writeln(). If you attempt to call write() multiple times, you cannot guarantee that they will be sequential as another thread may have called the write() method in the meantime.

Step 3: Call File Writer from Custom Code
With our file writer (ExportAdministrator) singleton in place, we can now modify our custom code to submit whatever we want to write to that file. In this case I have created our Custom Code class called "export.ExportIdentifier" in my CustomCodeReference project.

This custom code expects 2 arguments to be passed to it, an output type: a simple text string that will allow you to group outputs based on the type of data being written - this allows you to reuse the code for multiple types of values in a single test; and an output value: the value you want to write. In addition to these, this custom code will also generate a timestamp that will be written to the file in order to determine the write operation sequence.

package export;

import com.ibm.rational.test.lt.kernel.services.ITestExecutionServices;
import com.ibm.rational.test.lt.kernel.services.ITestLogManager;
import com.ibm.rational.test.lt.kernel.services.RPTCondition;

/**
 * @author grempel
 * @date 2014-03-27
 * 
 * Usage:
 *   Add Custom Code reference to a test
 *   Set Class name = /CustomCodeReference/export.ExportIdentifier
 *   Add Arguments
 *     - Text: Value Type String
 *     - Text or Reference: Value String
 */
public class ExportIdentifier implements
com.ibm.rational.test.lt.kernel.custom.ICustomCode2 {
/**
* Instances of this will be created using the no-arg constructor.
*/
public ExportIdentifier() {
}

/**
* For javadoc of ICustomCode2 and ITestExecutionServices interfaces, select 'Help Contents' in the
* Help menu and select 'Extending Rational Performance Tester functionality' -> 'Extending test execution with custom code'
* @param String.class[] args - arg0 = value type, arg1 = value to write
*/
public String exec(ITestExecutionServices tes, String[] args) {
//Validate arguments
String type = null;
String value = null;
ITestLogManager tlm = tes.getTestLogManager();
if(args.length<=1) {
tlm.reportErrorCondition(RPTCondition.CustomCodeAlert);
return null;
}
type = args[0];
value = args[1];
Long timestamp = System.currentTimeMillis();

//Generate CSV string to write to file
String result = timestamp.toString() + "," + type + "," + value;

//Submit string to writer
String error = ExportAdministrator.getInstance().writeln(result);

//Catch errors and return to RPT for validation and handling
if(error.equalsIgnoreCase("true")) {
return result;
}

return error;
}

}

Step 4: Add Custom Code to a Test
To use this custom code in your test, add a Custom Code element to your test (see Step 1 for reference). Instead of generating a new class, we will be specifying our existing class in the Class Name field with the format: "/ProjectName/package.ClassName". In the above example this will be "/CustomCodeReference/export.ExportIdentifier".

Next we will add the arguments to the custom code element details. The first argument is the output type. Click the "Text" button on the right and specify the text to submit as the type. This can be different for each Custom Code element that you add to your test, but it works best as a hard-coded value in order to ensure it remains the same for each iteration.

Then click "Add" and select a data source object or reference to use as the output value. In this example I have set the output type to "TestAccountName" and the output value to the "Username" variable from my datapool. This example is redundant, I already have my usernames in a datapool - normally I would insert the custom code after a page response that contains some data that I want to write to a file, and select a reference instead.

This will now allow me to export any generated data that exists in the response record to an external source.