More than ever before, financial service organizations are using analytical models to power their core business processes. Ever-higher customer expectations together with new market players are some of the drivers that are responsible for accelerating the pace of change in the market. This calls for business decision makers to react quickly if they are to keep up with the competition.
I have seen some companies react to this challenge by investing in analytics or IT capabilities. However, analytical flexibility or advanced production technology on their own do not allow for a business to react quickly and optimally to market changes. Only when these two components are fully integrated so that the IT understands what the models need in a seamless process, can the business implement rapid changes to models and deploy them to production at the right time. Furthermore, to fully utilize analytical and operational capabilities, the two components should work efficiently together.
The integration itself often requires additional learning and training for the analytical and business departments within the organization. In this blog series, I have and will touch on some of the factors to keep in mind when implementing an integrated analytical system. In my first blog in this series, Eleven Best Practices for Building a Successful Production System, we discussed practical features needed to bring models into a financial organizations production system. In this blog, we will discuss the model lifecycle and what is meant by an integrated analytics system.
How to create a fully-integrated analytical system?
In a fully-integrated system, a model will typically be in a constant loop of adjusting, testing, and deploying to production.
1. Gather new data → 2. Refit/adapt models → 3. Build and select winning strategy → 4. Initial manual tests
When working in the development environment, all recent data for predicted and explanatory variables, as well as for model performance should be available. This data will be used to analyze models, adapt them, and to select the best strategy. When new data enters the system, this flow will be repeated. The faster and more effective this analysis is, the more efficient your reaction to the market will be. The winning strategy should be tested in the development environment as part of the development stage before moving the models to the full-scale testing environment.
4. Test it manually → 5. Deploy to test environment → 6. Test it automatically
The testing process is extremely important in order to avoid harming operational strategy performance. Manual testing includes verifying that business logic and mathematical/statistical methods were followed correctly. However, this is a very time consuming process so ideally it should be complemented by automatic testing that covers a wide range of cases. The testing process and suites should be designed well to allow for comprehensive and fast testing cycles. They should be executed on an offline environment which is similar to production.
7. Debug problematic cases and fix → Redeploy and retest
If a problem was identified during any of the testing stages, you need to be able to quickly find its source, fix the models, update the test suites and repeat the relevant testing stages. Therefore, it is vital to design tests that not only report the problem, but also pinpoint its location. This will allow for a quick investigation of possible issues.
8. Deploy to the production environment → 9. Model monitoring
Make sure to deploy models to production effectively and at the right time. For this to occur, you need to manually and automatically monitor your strategy and operational system’s performance. If something goes wrong, the longer it takes to fix the larger the impact on business results is going to be. For this reason, you would want to know about any issue as soon as it happens.
Having a fully-integrated system benefits the entire organization, with increased accuracy of data for better decision-making. A common problem for many financial institutions is the ability to respond to changes in the market environment. As can be seen above, by implementing a streamlined model lifecycle process, many of the hiccups that occur along the way can be eliminated at their core before they manifests into something greater. This is essential for building a production system that all stakeholders in the company can be confident in using.