A very well respected and senior investment bank CTO outlined a core issue with present system development and the needs for digital transformation of data management.

“The big problem is this: we’ve invested hundreds of billions of dollars in teaching database design with focus on consistency, data-entry validation, and normalization. Tens of thousands of people are experts in this “validate input, apply it to the current world, get new world that can easily be consumed by the next program” model. Digital transformation of event data management means we, in essence, throw a lot of that model out: data may be non-sensical at the time we input it, consistency becomes defined over a 2-d time plane (with rectangles of inconsistency/incompleteness.) My database schema designers, my front-end data entry screen writers, and my report developers all have a ton of techniques to unlearn, and a whole new way of looking at a system end-to-end that they need to learn. How do we get this rather large group of people to understand/embrace digital transformation in data management? And is it even possible (what if it’s a fundamentally beautiful and correct idea that is just inaccessible to 90% of the programming population?)

The CTO went further and summarised the challenge as being:-

“Given a large time critical data store, ideas like consistency become defined over the (entry time, valid time) 2 dimensions of time. The practical upshot of this is that we really can’t validate data entry/changes beyond trivial self-consistency checks of the record to be added (e.g. we have to allow a ship to unload cargo we currently believe it doesn’t have, etc.) In essence, everything becomes late-binding and late-validation and late-consistency checking. 

I’ve got about 1000 developers working in an event challenging world like this, and it’s difficult to get them to understand the implications of data management transformation. Maybe 5% really get it, but 95% still want to build lumbering state machines that interrogate the “current state” and transition to a new “current state.” 

I think this is the biggest practical problem. How do you retrain/educate people to use a very different (and inherently more complex) framework that affects system design from front to back?”

This is an excellent description of the main concerns business managers have in regards to digital transformation in data management. In essence, business events and importantly key aspects to business processes don’t always get presented in the right chronological sequence but the system developers need to accommodate these because this is what happens in the real world.  Quite understandably, the development of new systems is an obvious solution but developers need a roadmap to make best use of their current skillsets and enable them to move into this area with the minimal implication.

A Time Travel capability does just this. In fact the Time Travel development journey for addressing the digital transformation of data management moves the developer into a realm where it actually becomes easier to create new systems! The reason for this is that the first step for the developer is to forget about time! Sounds odd but the first and critical step in using the Time Travel environment is to devise a current view model for the business requirement and exclude any accommodation for date or time stamp columns and also exclude history tables. Quite simply just model the data entities as if only existing in the present moment. This step may sound inconsequential, but it’s the changing values and states of data in time which cause the greatest cost in system development.

By having a completely different starting point – the normal start point is adding timestamps into the data model – the developer is not having to learn anything new. Most developers are familiar with relational modelling and quite simply the hardest step in creating a transformed system with Time Travel is the creation of a current view relational model for the required business system. The writing of the  applications and reports is then as easy as falling off a log!

The even better news for CTOs is that the existing skills of the database schema designers, the front-end data entry screen writers, and the report developers can all be use BTDaaS and that there is absolutely no need “for a ton of techniques to be unlearnt.”

The CTO provided an example of a real world banking requirement – see below. The really good news is that a developer with an afternoon’s training developed a transformed solution using BTDaaS and was astounded how easy it was to address. Of course this is mostly down to the fact that the starting point is a current view schema and this business requirement was simple to model

Here’s a concrete example that only deals with historical events:

The original application is this:

1. during the day, various trades come in

2. around 4pm, a clearing house sends an aggregation trade that cancels out the day trades and books a single aggregated/net trade

3. trades get cancelled for various reasons.

4. payments/reports, etc, get run around 5pm based on the aggregations.

The system does various sanity checks (e.g. you can’t cancel a trade that is already aggregated, you can’t aggregate a trade that doesn’t exist, you can’t trade with a counter party who doesn’t exist, etc.)

Everything seems to work ok, the numbers add up, the payments get made, etc.

Then the auditors come in and ask questions like “does a cancel ever come in on an aggregated trade?” In turns out this happens now and then: the users of the system resolve this by getting on the phone, sorting things out, then maybe getting a new aggregation record sent, or getting the cancel revoked, or whatever. In any event, lots of pending transactions may be unentered into the system while they are being explored manually (and the system’s validation rules won’t let them be entered anyway.) So, lots of financial risk is not getting captured in a timely manner.

So, the firm decides to go with a new solution which will give them a real audit trail, etc. Naturally, data fidelity is important…

The new system needs to allow entry of, e.g. a trade, with a physical time of 6pm and an effective time of 3pm (a classic missed trade due to a communications outage.) And the 4pm/4pm aggregation transaction should be enterable into the system even if it references that missed trade that is not going to appear until 6pm physical time. And the 5pm job that wires the money now needs to ask for balances effective close of day, based on the current physical time, but now also needs to know that the result set of that query may be in some sense invalid (e.g. we aggregated a trade we know nothing about.). Quite quickly, this becomes a significant challenge.