It’s a safe assumption that time travel is impossible. After all, how many people from the future have you met? Then again, we don’t know how many people have traveled from our instant in time into the past. Presumably, someone who vanished into thin air might have gone to a different era using their personal time machine. It is fun to think about time travel and has made for plenty of science fiction plot lines, but what does time travel have to do with software testing?
There are a lot of IT applications that track the passage of time and use time to trigger events that operate on data. For example, if you’ve worked with an accounting system, there are month-end, quarterly, and year-end closing processes for “balancing the books”. You may want to have the system generate messages and warnings as the time nears for triggering one of these closing processes.
So, how do you test a system for handling events occurring in the future? In this article, I’ll discuss how you can use the “time traveling” features of modern software systems to test for future events. I’ll use a non-accounting example to show this. I’ll also discuss why testing by going “back in time” without changing the clock, while appealing at first, is not a particularly efficient way to test a time-based system. I’ll use an example from the EPA’s drinking water program to illustrate how you can turn your computer system into a time machine for software testing.
In this section I give some background about the Safe drinking Water Act (SDWA) and how the regulations under “the Act” use time tracking for compliance determination. Under SDWA, EPA develops and promulgates drinking water regulations for ensuring that your water is safe to drink. Rather than using a real drinking water rule, which can be quite complex and somewhat ambiguous, I’ll use an imaginary, simplified rule as an example in this posting to illustrate how to test a system that uses temporal, or time-based processing.
Our example drinking water rule (I’ll be clever and call it “Test Rule”) requires qualifying drinking water systems take samples from certain points in their treatment plants and sources of water once every three years. (A source of water is a well, river, lake, or even a tanker truck delivering water to an area that lost it’s water supply.) A certified laboratory uses approved analytical techniques (or methods) to determine the concentration of the six compounds in the sample. Following our simplified naming convention for this example, we’ll call these compounds 1 through compound 6.
According to our example rule, each compound has what EPA calls a Maximum Contaminant Level, or “MCL”. If the concentration of a particular compound exceeds its MCL, the system will generate a candidate MCL violation. Just for kicks, let’s also say that the analytical method used by the laboratory for determining the concentration for the six different compounds is the same for all of them because they are chemically related. Using a single analysis technique, the laboratory can determine the concentration of all six compounds in the sample.
What happens if the samples aren’t taken on time or not taken at all? In the former case, according to the Test Rule, the system should generate a candidate monitoring violation. If the samples aren’t reported on time, but eventually received late, the system should generate a candidate reporting violation. However, if the water system did take the sample on time, but the laboratory reported the sample results late, the system should automatically rescind the candidate monitoring violation (because the sample was taken), but leave the candidate reporting violation in place.
Currently, the drinking water program combines monitoring and reporting violations into a single “M&R” type violation. EPA has, for several years, discussed separating M&R violations into distinct monitoring and reporting violations (the “M” and “R” seen in the example schedule shown later). Maintaining separate M and R violations makes it easier to tell when water systems are reporting late as opposed to when they aren’t doing the sampling, which is more critical than reporting late.
EPA prefers that regulators track every requirement, so a water system could incur up to six MCL violations per sample — one for each compound. Likewise, if the water system is tardy in reporting samples, it could end up with six monitoring violations if it failed to do the monitoring and six reporting violations if it reported the results late for all six compounds.
The process of taking samples, sending them to a laboratory, doing the analysis, reporting the results, and entering the results into the regulatory system takes time. Drinking water regulations allow ten business days from the time a sample is taken to when the results are reported to the regulator.
Sample results are often entered by hand (yes, we are in the twenty-first century, but in many places the drinking water program hasn’t caught up), we need to add some time for processing the samples after receiving them from the laboratory. Three business days for the regulator’s data entry section to process and manually enter the sample results into the system is reasonable. However, this time doesn’t count for detecting monitoring and reporting violations. Instead, the date the water system (or its agent) collected the sample at the sampling point is when the 10 business day clock starts ticking.
Speaking of business days, the system’s scheduling features should use a calendar and not just count days. Much like a project scheduling system, the calendar keeps track of non-working days, such as weekends and holidays.
To avoid a monitoring violation, the water system must have taken the required sample between the beginning of the first day of the sampling period (the sample period start date, or “SPS date”) and before midnight on the last day of the sampling period (the sample period end date, or “SPE date”). Samples taken after this time period would trigger a monitoring violation (as does not taking a sample at all). The system should do nothing when the water system reports sample results before the SPS date (it should, however, save the sample results anyway and generate an alert to the regulator if the result exceeds the MCL).
Suppose the water system did take the sample between the SPS and SPE dates, but reported it to the regulator two weeks after the date the sample was taken (the sample taken date, or “ST date”). Depending on the calendar, that is not a problem if the sample came in on the 10th business day after the ST date. But if the sample came in 11 or more business days after the ST date, then the system would generate a candidate reporting violation.
The user, a compliance officer working for the regulator, makes the decision to accept or reject the candidate violation (accepting it makes it a “real” violation). So if the regulator wants to give the water system some leeway and not charge the water system with a violation, they can do that. The system keeps track of the date, time, and identity of the user making the decision for accountability.
Now you can see how separating the “M’s” from the “R’s” gives more insight into what a water system is doing. Have a water system piling up a lot of reporting violations, but hardly any monitoring violations? It could be because their laboratory is inefficiently processing samples. Or the water system might be slow in getting the samples to the laboratory. Either way, separating the M&R violation gives the regulator more information about what’s going on with the water system.
As water systems are located across the US – from the Virgin Islands to the Commonwealth of the Northern Marianas Islands – the system must also use local midnight to determine if the water system exceeded the period date in reporting sample results. This means the system must keep track of the timezone of the regulatory agency (and not the water system). So what started out as a seemingly simple date tracker is more complicated that that.
Example Sampling Schedule
In this section, I extend our example by presenting a sampling schedule mock-up for the “test” drinking water rule for regulating compounds 1 through 6 using maximum contaminant levels (MCL) as discussed above. The rule requires the water system to take samples from sources of water (wells, lakes, streams, etc.) and treatment plants. Table 1 below shows an example sampling schedule for water system XY-1234567. According to the schedule, samples are to be taken at Treatment Plant 1, sampling point 1, and at Intake 1 (a source of water), sampling point 1.
Sampling Schedule Table Heading Definitions
This section defines the information contained in the sampling schedule shown above it Table 1.
Water system – This is the identifier used by the regulator for the water system. For brevity, I have not included the official name of the water system.
Task type ID – This identifies the task used to create the scheduled requirement. The design approach used in the example system shown here uses tasks as the building blocks for creating all of the schedules used by the system. There are six different tasks building blocks used for the example sampling schedule — five for routine sample collection of each compound 1 through 3 and 5 and 6, and one for waiving collection of compound 4.
Requirement ID – Unique identifier for each requirement in the schedule.
Requirement Description – “Human readable” description of the task stated as a requirement. In this example, the requirements are “take a routine sample” or “waved sample” meaning don’t take a sample because the requirement to measure the compound was waived by the regulator.
Sample for – Indicates the compound to measure. Under the drinking water regulations, this can be a chemical (organic and inorganic), biological (such as a bacteria), and radiological.
SPS date – Sample Period Start date (described above)
SPE date – Sample Period End date (described above)
PEV date – Processing End Violation date (described above)
ST date – date the water system (or its agent) took the sample
PE Violation – Processing End Violation – one or more candidate violations the system generates if the ST date (see below) is after the PEV date.
MCL – Maximum Contaminant Level – if the concentration of the sample exceeds this value, the system will generate an MCL-type violation.
Sample Result – Concentration of the tested compound in the sample.
Note to those readers familiar with the drinking water program: I’ve simplified the sample schedule display for the purposes of this article. I’ve also avoided drinking water jargon for the names of many of the columns in the schedule to help with understanding the example.
About That Test Data…
As you can see in Table 1, I generated some test data to create the sampling schedule. This involved selecting a rule, a water system, and creating sampling requirements for two sampling points, each associated with a different facility (treatment and source of water). I would also need sample results data to match samples taken on time (within the SPS and SPE dates), reported on time (PEV date and the 10 business day reporting requirement) and determine if the results are in compliance (below the MCL level) or out of compliance (equal to or above the MCL level).
I should also include some data to test some “edge” conditions that may not normally happen in the real world, but could “upset” the system if they happen. For example, I may want to include test data showing the drinking water system submitting data before the SPS date.
I can obtain test data through a combination of making data up and using actual, historical data. The key to effective testing is that I must know the expected result of each test case before I begin actual testing. If using historical data from a legacy or earlier system, I must verify the data will generate the expected results. I cannot blindly assume results calculated by that system are correct. This is an especially important point for testing the system as users can accept or reject the candidate violations it generates. If the legacy system did not keep track of accept/reject actions, there is no way to tell if it calculated incorrect results or if a user overrode a correct result by rejecting it.
Another aspect of designing the test dataset is to pay careful attention to the date values. With a scheduling application like the example discussed in this article, the dates are more complex than just day-month-year — they also have a time of day component. For example, the PEV date 9/15/2020 is really “15 Sept 2020:24:00:00”. (By convention, 9/165/2020 begins the instant after midnight, 9/14/2020.) If the system generated the monitoring and reporting violations when the day changed to 9/15/2020, then it would have done so a day early (at most 23 hours and a hair under 59 minutes). Instead, the system should use the date 15 Sept 2020:24:00:00 as the cut-off for generating the monitoring and reporting violations
When building test datasets, focus on quality rather than quantity, especially if the decisions generated by the system have human safety implications. Ensure that your test team fully understands the test cases and the expected results. Think of the tests as a movie with different scenes. As time progresses and the movie plays on, new data comes onto the scene and the system reacts to it. The test team compares the outputs of the reactions to the expected results. If there is a difference between expected and actual test results, it should be because there is an error in the system and not in the testing process. All of this takes time, but the payoff is higher confidence in the decisions made by the system.
Temporal (or Time-Based) Functional Test Example
This section discusses temporal (or time-based) testing scenarios using the example sampling schedule given in Table 1 above. Before presenting the example testing scenarios, I’ll discuss testing scope because, after all, too much testing is not necessarily a good thing — unless time and cost isn’t a factor in your project.
Before assembling test data and writing test scenarios, it’s important to define the scope of testing. Exactly what is your team going to test? What’s “out of bounds” for testing? In the previous section I alluded to an example where the test weighs outcomes from an earlier, or legacy, system. This should be out of scope for testing because the results obtained by the earlier system are irrelevant to the new system for the reasons I discussed.
Also, as the system will generate candidate violations, any testing beyond that should also be out of scope. The key to testing large, complex software systems is to break the system down into smaller, more manageable components to test. In our example test here, we are interested in how the system uses it’s scheduling features to track requirements and do something when the plan doesn’t match what the water system actually did. We might also be interested in calculating MCL type violations for the test rule.
We can logically separate the process of accepting/rejecting/rescinding violations from our time-based testing. As the system will support several different kinds of schedules, all of which can result in candidate violations, it makes sense to elaborate, or (using older terminology) decompose testing into smaller, bite-size pieces. Therefore, to make our time-based testing more manageable, we’ll leave testing the system’s features allowing users to accept and reject candidate violations and to rescind previously accepted violations out of the scope of this round of testing (but do it in a different round).
We will, however, test the system’s ability to rescind candidate monitoring violations when the regulator receives sample results that were taken on time, but reported later after the PEV date or 10 days after the ST date if that date is before the SPE date. Logically, this scenario is crucial to generating candidate monitoring and reporting violations.
Table 2 below presents a set of test scenarios. Notice that there aren’t that many scenarios. I’ve purposely limited this list to the different transactions involved in testing the sampling schedule agains time. The purpose of the test is to verify the system is functioning as expected. If I created a thousand different scenarios, I wouldn’t be testing functionality so much as system performance, which is not functional testing.
You can download a copy of more test scenarios from my blog site here.
As you know, all computer systems use an internal clock to manage internal processing at the operating system (and lower) levels. In our example production system, it would use this internal clock to keep track of time and to trigger certain processes when a specified period of time has elapsed (like an alarm) or a point in time reached (date/time). Rather than using the internal clock for our testing regime, we’ll use a “pseudo-clock” that we can control. The test system will read the pseudo-clock instead of the internal clock. This way, we can make the system act as if it’s in the past or the future. We’ve created, in effect, a “time machine” for testing purposes.
If you downloaded the test scenarios, you’ll notice the two-level numbering scheme used to identify the scenarios (which can have more than one test with associated expected results and outcome). Each scenario has a synopsis (brief description) and a timestamp, which is the time setting for the pseudo-clock before the testing begins. As you work down through the scenarios, the timestamp changes, simulating the passage of days and weeks (and months and years). In the first scenario, we set the pseudo-clock to 5/15/2020:24:00:00 and then it starts counting time like the internal system clock.
The pseudo-clock ticks until the scheduled time, triggering (or not) an event. The beauty of using the pseudo-clock is you can do this over and over again — like the move Groundhog Day. It frees your testing from the constraint of time. You can use the pseudo-clock over and over again, rerunning the tests as needed.
If your team is using a business process management solution ( JBoss KEI and competing products), chances are it includes a modeling workbench where you can simulate the test scenarios shown in Table 2. These kinds of tools include a pseudo-clock for testing, validating and verifying time based business processes, like the scheduling discussed in this paper. If your team isn’t taking advantage of these tools, seriously consider doing so as they can make testing less complex, confusing, time consuming and costly.