Lessons learned during a DR implementation

During the last few weeks I've been engaged in the implementation of a disaster recovery solution for a payment system. The task is nearing its completion; usually that's the period where you get the earth move away under your feet due to some unforeseen factor or a detail somehow missed. Thankfully that doesn't appear to be the case in this particular project.

It's always a nice feeling to get such an important and complicated project under your belt. During the course of this project I haven't really picked up any particular technology skills worth mentioning (except being able to break my personal record of the time needed to install the payment system from scratch). However, I gained some experience on practical DR planning and learned a few important lessons. Here are the most significant ones.

• Closely examine dependencies. In the case of a payment system, it's very difficult to have it operate solely on its own. The payment system usually interfaces with a lot of other entities. These can be the bank's host, external card management systems, Visa and MasterCard facilities, help desk system, middleware servers and others. Some of these are more important than others but it should be defined early on what is an acceptable disaster recovery setup. For example, it doesn't make much sense to have disaster recovery planning for the payment system but not the bank's host. These external systems must be a part of the disaster recovery planning as well.
• Make sure that there are well defined, written procedures for switching to the DR site. This appears deceptively trivial, right? Well, it's not as easy as it sounds. It really depends upon the organizational skills of each institution, but it can be a tricky exercise.
• Don't be afraid to break the norm and use the infrastructure available. In my case, keeping the disaster recovery in sync with the production site is officially achieved using replication software. However, the bank already had a SAN in place and that was already replicated to a disaster recovery site. By not following the standard, we managed to leverage what appears to be a vastly superior solution in the DR implementation.
• Test every possible scenario you can think off. By far the easiest DR scenario is one when everything goes smoothly and you can perform an orderly shutdown of all servers, then switch to the DR site. But what about a power outage in mid-transaction? What about a software failure severe enough to render the production site unworkable? Would the plan to switch to DR work under such circumstances? Test it and find out.
• Test exceptional scenarios under load. Everything runs smoothly in the test environment where you have a load of exactly 0.01 transactions per second. How about loading up the test system with 40 TPS and see how a switch to the DR site goes? That's what's going to happen in a real-world scenario.
• Bundle as few infrastructure upgrades as possible in a DR project. It's tempting to say that since we'll test everything, how about also upgrading our database server and also install those 25 patches to the payment system as well? Well, that's a thought and it can save you some time...if everything goes smoothly. But the last thing you want to do is end up chasing down problems due to upgrades while you should be testing the DR plan.

Documentation

I frequent the CodePlex site quite often. As a developer I try to get a glimpse of the efforts of other developers for various reasons. Some of the projects found in CodePlex are a one-year scheme, possibly a junior developer's attempt at fame but without serious motivation. Most of the projects tend to revolve around topics that appeal to the younger developers, especially projects revolving around ASP.Net and MVC, HTML, content management and other related popular technologies.

Occasionally, I find an open source project that stands out from the others. Some time ago I got interested in CommonLibrary.Net. This little project is a library of reusable code but it is different in several ways from other similar endeavors. A very encouraging point is that the project started out in 2009 and the main author is very active, committing regular check-ins and continuously evolving the library and posting releases with new features and fixes. Another feature that grasps the attention of the visitor is the large namespace and the multitude of developer helper classes that were fit into this project. Finally visitors browsing the code can quickly see that it's of unusually high quality, with the main author clearly knowing the tricks of the trade.

There is one sore point, and CommonLibrary.Net is by no means the only project at CodePlex that is problematic in this area. The quality of documentation is poor at best. There are examples of using the library and one project deliverable is a help file generated by the code XML comments. But it's not enough. The examples focus on showing how to use specific namespaces of the library but do not explain the general idea behind them. Browsing the code one can quickly see that there are several problems with the XML comments from which the help file is generated: parameters are not documented, comments are copy-pasted between overloads and are incorrect, class and method comments are frequently vague and a lot of protected or public members remain undocumented.

I think that there are several cases where an open source project can get away with little or no documentation at all. But a developer's library project simply isn't one of them. I would go as far as saying that the documentation may very well be the most important deliverable of such a project. Users of code libraries are themselves developers but that doesn't mean that they want, by default, to see the source code to understand what's being done under the hood. And the notion that a well laid out namespace and use of naming conventions are their own documentation is correct but way overblown if applied to the max at the expense of code comments. True, one can easily gather that ComLib.Scheduling.Scheduler has something to do with scheduling tasks but how are the tasks scheduled exactly? What is a task? What triggers a task? Is there something a task should not do? How can callers know the status of their tasks? Can they stop them, reschedule them, pause them? Can they add a task at dynamically at runtime? At start-up time? How can they gracefully stop all tasks when the program is shutting down?

Fortunately work is being done at CommonLibrary.Net to slowly fix that. It appears it might take a while but it's being done. Documentation for code libraries is indispensable.

Change management

When developers hear of "version control" they tend to think of CVS, SVN, Git or other relevant source code version control systems. The idea behind version control is to have a repository of the work done so far accompanied by the corresponding metadata (who changed a source code file, what was changed, was is the comment of the developer that committed a change, when was the change committed, etc). Version control is such a basic and essential facility that no developer or programming hobbyist who take their work seriously can live without it.

From the perspective of the end-user, the corresponding process to manage the life cycle and operation of a payment system is change management. It has a somewhat different meaning and it refers to the process used to track changes done to the payment system. Change management encompasses the complete range of activities that have to do with the configuration of a payment system. These may include installation of new modules, installation of patches, database or file configuration changes, operating system patches or configuration changes, the method each change is applied and other related information.

The main deliverable of change management is a detailed audit trail of installable modules, configuration changes and people that authorized and performed an installation or change to the payment system. This has the following benefits:

• A change management process ensures proper change authorization.
• It enforces different roles to people requesting a change and those who actually apply the change.
• A change management process can identify change prerequisites or areas affected by a change and it becomes easier to properly manage any related issues.
  
• Change reporting is feasible in a very structured fashion. It is easy to get a snapshot of the latest installed modules, patches or configuration. Likewise, it's also easy to find out the path of installs and changes that resulted in the current system. And it's also trivial to attribute changes to the business entity or person that authorized them. 
• Once in place, a properly used change management system forces you to formally evaluate each change and assess the related implications. Likewise, it ensures that a change is applied by the authorized people in the correct manner. In short, it forces you to do things the right way.

It appears that for some organizations change management is a difficult to grasp notion, despite its easily defined purpose. After all, what's it good for? All organizations have some kind of process that needs to be followed in order to authorize and make changes to a payment system, so what would change management add to this process? 

Well, sooner or later (most probably sooner) one of the following questions will pop up:

• "I want to create a new QA system, what do I need to install and what do I need to configure?"
• "Who authorized taking that patch live?"
• "I want to create an image of the production server for DR purposes. What's installed in production?"
• "Why did we make that change in the database?"
• "How is the system configuration different to the default configuration?"
• "What are the prerequisites before installing this patch in production?"

Without change management, answering each of the above is a time consuming and error-prone task. Change management is therefore very important. It doesn't matter how is it implemented. An organization may elect to create a home-grown change management system, purchase a license for an existing system or go out to a web-based or SaaS-based offering. The important thing is to have the process available and functioning.

Tokenization

For several reasons, the acquirers and banks in the part of the world that I live in have not seriously gone after the web merchants to make them become PCI compliant. I guess that PCI is one of those topics that everyone hates. I believe that for merchants in particular, the subject must really be viewed as a major nuisance.

The word is out, though, that Visa is getting stricter and that there's going to be a fury of activity. Well, it was about time. I'm not a supporter of PCI just for the sake of PCI. But if you think about it, Visa is doing the merchants and the acquirers a favor. Security holes and procedural gaps that could potentially hurt the merchant and the acquirer will be assessed and addressed and that's a good thing. A security breach can be serious enough to close you down. Even if you put monetary losses aside, a breach can generate enough negative publicity to put you out of business.

One way to minimize the scope of PCI is to introduce tokenization. Web merchants that use this technology can get to a point where no sensitive data ever gets into their systems, regardless of what those systems are. Only a card token and possibly a transaction token are stored. This information, if stolen or intercepted, is useless to a data thief and cannot be used to send fraudulent transactions from some other part of the world.

There obviously need to be changes in the authorization flow if tokenization is to be introduced. For the acquirer the exercise is not exactly straightforward. Tokenization is not the typical service that acquirers provide to their merchants, at least not at this point in time. For banks that provide acquiring services to web merchants (and hence have smaller numbers of e-commerce transactions), the economics of the business case are even trickier. This is the reason why there are service providers with offerings that are centered mostly around tokenization and data security.

As payment systems continue to evolve, they will doubtlessly include tokenization as part of the standard transaction flow and the window of opportunity for tokenization service providers will close. Securing the authorization process with card and transaction token will be the first step. Handling recurring payments is somewhat trickier and could be a batch-based process but eventually payment systems will provide a solution for that business need as part of their standard out-of-the-box packaging as well.

Ownership continued

Some time after blogging about "Ownership" it so happened that I got into a meeting with a company that provides payment consulting services. The people of that consultancy company seemed nice and they certainly have been around the block a few times. After discussing a migration project, I was left with the impression that they knew what they were talking about.

As the conversation unfolded, we started broaching the topic of testing and certification. I quickly discovered that the consultants had a very different view of how testing and certification should be handled. I am of the opinion that when an end-user (a bank) wants to in-source their payment system, they should take ownership of all testing and certification activities and immerse themselves in these tasks.

To my initial surprise, I discovered that the consultants favored the extreme opposite approach. They pretty much wanted to come up with the test cases, execute the tests and be in charge of certification cycles. In short, they felt that it is preferable to them to get all over these tasks themselves than help the end-user do them.

"How can the end-user ever get off the ground if they do not own these processes?", I thought to myself. "How can the end-user even operate their own payment system and innovate with it if they can't even go through a test cycle themselves?".I presented these concerns to them. Their response was along the lines "We can provide this service to the end-user".

Some times I'm pretty slow but at this point I grasped the obvious difference between us. The consultants are trained to act on behalf of the end-user. They are hired by the end-user and most of the time adopt the viewpoint of the end-user. When they are successful in a project, they think and act as the end-user and in essence they are the end-user. On the other end of the spectrum lies the vendor that provides the payment system software - the vendor would be very comfortable just selling a license. I happen to sit comfortably in the middle. I work with payment system vendors and, when I'm successful, I help the end-users to integrate the payment system in their organization.

In the end, the consultants virtually re-validated my opinion: for an in-sourced payment system, the end-user must step up and take ownership of the system. Whether this is achieved by the end-user's own resources or by getting hired guns to do the job is a different matter. I'm not by default against getting consultants to run an operation and take on all the things that an end-user might find distasteful. Sometimes the conditions and the economics may be right to do just that, perhaps for a large retailer, a specialized processor or a global bank. But in the long run it just doesn't make sense for the average case. If continuous integration, patching, testing and certification activities appear to be more than an end-user can bear, there's a perfect alternative which is out-sourcing.

Complexity

Looking back at the 90s, the tools of the trade of programmers implementing payment solutions were horrific by today's standards. A great deal of work was done using Telnet and I remember that I quickly learned to hate the blue, text-only interface of the TTWin terminal emulator. Everything was slow. It wasn't a problem with the terminal emulator or with the text-based environment in itself, just a lack of programming accelerators and facilities. Whenever a project came along that allowed development outside of the proprietary box running the EFT switch, the contrast was evident.

Auxiliary tools were also non-existent or downright primitive. Message protocol simulators were basic and moody. Terminal simulators were starting to emerge but were costly and sometimes hard to use. Writing and executing tests was done with ad-hoc tools so esoteric that it was often too tempting to just do the whole thing manually. Traces were hard to read. Network protocols like SNA and X.25 required considerable expertise to get a hold of. And if you had to remotely connect to another system to do something, you had to go through a modem - it often turned out that the bandwidth of myself getting up and physically commuting to the remote system was considerably larger than that of the modem.

Well, nowadays the tools have been considerably upgraded. We're now coding in Eclipse or Studio, two very powerful IDEs with great facilities. Workstations have become very powerful. Dual monitor setups are becoming the norm, providing valuable additional screen real-estate. Simulators are sometimes build-in to the payment products and, if they aren't, they're easy to build or free to download or inexpensive to buy. Test facilities and test harness suites are abundant, easy to work with and sometimes ever free of charge. TCP/IP has dominated and has become the network protocol of choice; the implementation of other network protocols is usually left to dedicated network devices. Remote connections can be established in a secure and fast manner.

The weird thing though is this: while the state of the facilities and tools available has improved by an order of magnitude, productivity hasn't improved as fast. Back in the 90s the rule of thumb for estimating the implementation of an extensive stream message protocol was six months or more. Now it's three months or more. Way faster than before - after all, being able to do a job in half the time is a big improvement. But it's not an order of magnitude improvement.

Implementation of small-sized projects has benefited greatly from software evolution and maturity of tools. I have repeatedly been able to complete in a week projects that could easily take months in the bad old days. But the speed improvements diminish rapidly with the expanding size of projects. We're still faster but not by the same degree. And there's a very simple explanation for it. Complexity has increased. A gazillion new factors and acronyms have been introduced in our daily development cycle since the 90s. BCP, CNP, EMV, PCI, ISO 20022, NFC, CAP are some of the things that we have to keep in our heads.

This is by no means unique to the payments world. Software in general has become much more complicated. Attempts to simplify things and hide all this complexity behind frameworks, SDKs and APIs are partially successful but so far frameworks are not always able to help us keep up with increased complexity. Perhaps the abstractions provided by the frameworks will become better over time. Perhaps frameworks, like software, are themselves becoming too complex and contribute to the problem in their own way. The one thing that's clear is that human factor is becoming more important in our line of work. In the bad old days, a good programmer without payment systems experience could stand on his own two feet after a few months. This is not the case anymore. The individual has to capture and process a massive amount of information before beginning to see the light. Getting, and keeping, good people is now more important than ever.

Authorization scripting

Modern payment systems usually offer a multitude of configuration options that cater for a large variety of situations. The thing, though, with such configuration parameters is that they are meant to cover the different situations that are thought up by the system designers. The extent of configuration parameters is limited by the architectural fantasies of the system designers. When users of payment systems delve outside these limits (something that tends to happen quite often) there comes a moment when configuration just doesn't cut it.

When configuration can't do the job, the traditional way of implementing changes in a payment system is through adding code. This could manifest in many forms depending on the system architecture. An external message protocol may be coded as a plug-in. A component may allow user exits to be written. If code for core components is available, end users may hack away at it and change the system behavior to implement their requirements (a practice that they universally regret in the years to come).

At the business spectrum, end users don't like code changes for several reasons.

• They usually take a disproportionate amount of time to implement and thoroughly test.
• They carry a financial cost.
• Custom code tends to cater for uncommon situations and is generally not well covered by patches and upgrades.
• When piled up, custom code may erode the system in a kind of spaghetti state.

A not-so-new development that claims to minimize the need for custom code is scripting. What a scripting facility in a payment system offers is an abstraction of a lot of the intricacies of application languages, making it possible for non-programmers to quickly implement a script. In addition, scripts are generally interpreted in nature and thus more dynamic.

To the user, scripting seems like a panacea but is it? It's true that a scripting facility offers some advantages.

• The interpreted nature of scripts makes it easy to dynamically inject them in the authorization process without downtime.
• Some customizations are very easy to implement with a script.
• Implementation of a script generally requires less time and resources than a code change.
• Scripts can be much more easily implemented by end users than code changes can.

However, not everything is as it seems.

• By default, execution of scripts is an order of magnitude slower than execution of code.
• Just as with configuration, the scripting model places restrictions to the customizations that are possible with a scripting language.
• Making it easier to implement changes doesn't necessarily translate to a more manageable situation. Instead of code spaghetti you could have a script spaghetti of the same complexity, especially with script-happy users.
• The burden of script maintenance and proper alignment with system patches and upgrades is generally on the end user.

A proper scripting facility is obviously a nice addition to a payment system. It's better to have it that not to have it. But it should also be appropriately rated by end users. No scripting facility, regardless of how useful it may be, can make up for shortcomings of a system in other critical areas. And, regardless of the versatility of the scripting facility, there will always come a time when implementation of custom code will be inevitable. In that respect, a scripting facility is useful but a software development kit is what's really invaluable.