Why would a reasonable person undertake one of the most difficult types of software development efforts out there? Very rarely the answer is because it just sounds like a wicked fun thing to do. Usually, the decision is driven by the realization that you, as a business, are backed into a corner and, despite months or years of Denial, Bargaining, Anger, and Depression you’ve finally come to accept that the pain of creating or updating this application is equal to or less than the current or inevitable pain of not doing so.
The root of this pain can be summed as follows:
Either:
Or:
In both instances, the business need is critical and failure represents a catastrophic breach of your profitability and viability.
Furthermore, I would caution that, in some cases, one or both of the above scenarios also contain the embedded threat of a security attack or failure to meet the security guidelines set by compliance standards like HIPAA, PCI and SOX.
So, it must be done. You’ve exhausted all other options, including launching a new career as a lobsterman (because, hey, even hauling traps at four in the morning in heavy seas in February seems moderately preferable). But before you throw yourself – begrudgingly – headlong into this compliance-focused software effort, there are a few things that you should know. Understanding what lies before you is key to managing expectations on cost, schedule and, indeed, some allowable level of grumpiness. So, let’s begin:
1. Unlocking complex business intelligence is iterative and takes time.
Imagine there is a duck sitting in your lap. Imagine this duck has all day to sit there because it is warm and you are a nice person. Now imagine that this duck casually asks you how your company does what it does. Imagine that the duck is three years old and at every turn asks, persistently, annoyingly, WHY?
How long would your explanation of how you do what you do take you? How many times would you need to go back and tweak your answers to better reflect reality and not your shorthand, in-most-cases, initial explanation? Imagine writing that down. Imagine documenting it all and then imagine building a piece of software against that explanation and then imagine releasing that to your staff. Would it work? Would it represent their daily workflow needs? If they had the duck on their lap, would their explanation be different?
This is your business intelligence. It’s all the rules and responses to situations you’ve internalized over the years you’ve worked in your field, doing what you do. Much of it, I would wager, is mostly muscle memory – things you don’t much think about, you just know. This is what we’ve got to get out of your head, or the company’s Head.
This Business Intelligence will become the basis for your database architecture, your reporting, the accuracy of your data, your rules engine, your confidence that the application is actually performant and not failing silently. It is the very heart of the application itself.
2. The application is only as good as your data and your database architecture.
It seems so simple: You want to enter data into the application. You want to run a process that crunches that data. You want the report to be accurate.
But it is far more complex than that – indeed, if it were that simple, you’d still be getting by with Excel spreadsheets and a gazillion embedded macros, right?
With Compliance-focused software, we can do all sorts of number crunching, but if the “numbers” we are crunching are wrong/missing/imaginary/Text instead of numerical/numerical instead of text/etc., the output will be wrong. GIGO: Garbage In, Garbage Out.
A senior developer here states it this way:
“There is a tension between having a reliable system that faithfully represents strict rules, based on laws, and allowing a system to capture and store real-world data as it is known by data entry staff.
If you require a field that a data entry person cannot determine, then either records get left out or the field gets fudged with some made-up value just to get past validation; on the other hand, if you allow data entry people to say “N/A” or “Unspecified” all over the place, then you can’t thoroughly and consistently apply business rules to the data.”
Kick in the head, right?
Closely tied to this is the concept of Reporting/Auditing, which so frequently is expressed glibly as, “Oh, and you know, reports. . . .”
Reports and audits are hugely misunderstood. Clients really do seem to think of them as infinitely flexible, and somewhat magically-born, side effects of the application itself. They focus a lot of time and energy thinking about how to get the data into the application and rarely spend much time thinking on how to get the data out in a meaningful way. Yes, making sure that the data is tagged and stored correctly in the database is crucial and database architecture is paramount here, but that doesn’t mean that data can be whimsically extracted into a report without effort. Thinking about exactly what you need to run reports on, both now and in the future, as well as how customizable those reports need to be will be a significant amount of that duck-lap time.
3. Legacy data is snarly.
You’ll want your old data. Very few companies can afford to abandon their history as, you know, history is pretty integral into that whole Compliance algorithm.
But migrating legacy data from an old system is difficult. It just is. If you’ve been following along with the above, you can extrapolate why. Abandon the concept that there’s “a script of something” that can be employed to just jam the legacy data into the new database and that the application will run fine. What happens, in reality, is a huge amount of manual data massaging to get the new application to accept the old data. It’s tedious, time consuming and frequently frustrating to everyone involved.
4. Be prepared to test. . . and test. . . and test. . . and test. . .
I think we can all agree that the objective is to have a system that produces results that are accurate and reproducible. That’s the whole point: To be able to enter data you’ll use to monitor compliance, to have that data be compliant with any laws or policies if applicable, and produce reports that are true and faithful representations of the data you collected.
The best way to achieve this is to test the application to within an inch of its life. Bang on it. Hard. Put it to users and have them try to break the application. Assign a few people who are really good at doing the math behind the reporting to provide the development team testers with lots of fake data and the right results based upon using that fake data. Have those same folks available to spot-check the application throughout development.
Managing your expectations will help you keep your cool, especially as you edge towards a beta-release and things appear to be devolving into chaos when data is run through the system. Remember that what you are undertaking is difficult but has been done before, and done successfully. Lean on your development team to keep you informed and empowered and know that your highly specialized understanding of your business and the problem domain is invaluable. With you covering that, and the development team covering the “under-the-hood” stuff, and with a healthy dollop of patience and good humor, you’ll be highly successful.
