Maintaining financial data integrity and software quality are paramount concerns for any major financial institution. This is especially true when the firm in question is a Fortune 50 Bank, and the system involved manages a multi-billion dollar mortgage asset portfolio. Staying current on valuation requires 24 x 7 asset calculations on multiple sets of thousands of mortgages, including a check of the overall value of the portfolio which is accomplished every night. To meet these demands, the bank maintains a custom-built, 1000 node grid supercomputing platform for asset valuation, management and reporting. This system is core to the bank’s operation, is unique to the industry and has been under continuous development and expansion for over ten years. The grid computer system is subject to periodic upgrade and replacement to avoid hardware obsolescence.
Due to the extreme transaction volume, reliability, availability and functional performance required, the migration to the new platform with its multiple processing environments had to be verified as fully functional and demonstrate as fast or faster performance than the existing platform. Complicating this picture was the fact that no formal requirements or design documents had ever been developed for the system. Further, the system's user interface had been adapted to its highly expert end-user group which was more concerned about correct operation than usability.
SLI Global Solutions (SLI) was tasked by the bank with a project to support the planned migration with multiple measures of success, which included:
- Rapidly documented all components of the system;
- Developed corresponding functionality tests for all system components; Ensure the tests are uniform across all disparate operating environments;
- Executed tests to provide absolute evidence of replicable performance between the new platform and the existing platform;
- Managed the test execution effort to provide "on-the-fly" progress status, including exact completion status at any time of the work day;
- Fully conformed to SOX operational and auditing requirements; and
- Delivered the testing, management and reporting features cited above without purchasing or leasing any Commercial-Off-The-Shelf (COTS) products.
Our test program spanned 11 components, approximately 100 display screens, dozens of dynamically inserted 'task' panes within the display screens, and hundreds of different configuration and control combinations, most of which were tested in permutations across multiple settings. These tests fully covered all daily operations and also supported month-end and quarter-end reporting periods.
SLI rendered over 300 test cases into procedures with automated time journaling features to provide real-time tracking of both projected and actual schedule in order to ensure the system migration date would be met. Additionally, we developed six individual customized test suite dashboards that allowed test libraries to be adjusted to different patterns of functionality and performance in each of the six grid environments. We also provided innovative, SLI- proprietary data mining and reporting tools capable of pushing hundreds of test results via single display screens, with real-time "on-the-fly" update capabilities. These capabilities enabled the test team to update and report on percent testing complete, current status and projected time to complete testing within 90 seconds at any time during the work day.
In order to accurately report test results, SLI engineers visually rendered outcomes for the new system to compare existing vs. new grid system performance. This approach enabled the new performance results, even though occurring in non-replicable environment conditions, to be contrasted with prior parallel runs in the existing system.
The results of our program enabled the client to verify the readiness of the new grid system and facility on schedule and with no additional costs. During parallel operations, empirically derived performance test data matched our predictive results and the system delivered the enhanced performance that was anticipated in this multi-million $ migration. This result enabled our client to retire the existing system ahead of schedule and enjoy the cost savings and performance improvement of the new system almost a quarter earlier than anticipated. The net business benefit of this engagement was measured in multi-million dollars per year of operation – resulting in a very sound investment for the bank.
Said the Project Manager for the overall migration effort:
”Downtime of any length is very costly and deemed unacceptable: for this effort, the only acceptable risk was 'zero' risk. As we did not have the resources to satisfy the testing requirements within identified project schedule, we met with SLI Global and quickly decided that they had the expertise and personnel for the job. What made SLI stand out for us was their communication and flexibility. As with any project of this size, problems arise; SLI helped us rise to the occasion at every turn. On the bottom line, SLI was instrumental in helping us attain that goal of keeping acceptable risk at zero."