[0:01]Hi, I'm Brenna Quirk, a Senior Technical Online Course Developer here at InterSystems. Joining me today, I have Eric Hemdal, Principal Technology Architect, Matt Giesmann, Senior Applications Developer, and Chad Severtsson, Service Executive. Today, we'll be talking about testing applications built with InterSystems products, and specifically, we'll be focusing on testing non-functional requirements. So to start us off, Chad, can you give us a quick definition of non-functional requirements and how they differ from functional requirements when it comes to testing? Sure. Um, to me the main difference is that functional testing is ensuring that 2 plus 2 equals 4 pretty consistently. Um, non-functional testing is everything else. Um, the way I really focus in on it is that non-functional testing for me is really about understanding the ROI of that particular feature, product, solution, whatever you happen to be building. It's thinking about it in terms of how much does it cost, and that's not just, you know, developer dollars, that's long-term maintenance and storage requirements, uh how much does it cost to maintain, how much does it cost to implement that solution per customer? Thinking about all of those things in contrast with the business value that is received from whatever the thing is that you're building. And it goes into even more things than that, it's looking at what's your RTO and RPO for things like that, are they actually even practical and possible based on the current design and implementation? Right, so, uh one last one would be, uh, does any of your design decisions require you to do battle with the laws of physics? That's another uh key non-functional test you want to make sure your application passes. Yeah, can you say a little bit more what you mean by that? Oh, I have a prop that lives on my desk, and that is that you should not be trying to violate the speed of light as part of any of your requirements or implementation strategy. That makes sense. I think that's a good rule to live by. Um, so Eric or Matt, do you have any specific examples of non-functional requirements you've seen in your testing? I've tended to focus more on the technical aspects of when we talk about non-functional requirements. Things like reliability, like how long does it have to stay up? Um, maintainability, how easy is an application to maintain and keep running and modify and extend what have you? Um, availability, uh how long does an application have to be available? Are there certain times of day or days of the week or months of the year when you absolutely have to have the application uh available for use? Um, installability, right? I've got this great package, but can I get it installed? Can I get it provisioned? Uh, or is is that a major implementation project? Uh those, you know, those are some non-functional uh things that we that I take a look at. And then there are more performance-oriented, uh non-functional requirements.
[3:19]Sure, the application works, but will it work when you've got 10,000 users? You know, yes, you can handle a request, but can you handle 10,000 of them coming in all at once? Uh what happens as you add users or if you're in a hospital, you add beds to the application, or you add interfaces to an interface engine? Uh can you scale? And, and how do you how do you make sure that the application is going to work, you know, at full volume or full speed? Right, yeah, that makes sense. So, I think, you know, for functional requirements, people probably have a pretty good idea of how to write tests and make sure that, you know, things meet the requirements, but how do you actually test some of these more non-functional requirements? Like you said, Brenna, um functional testing is probably more familiar, and um maybe there are more frameworks. It's easy to um or maybe not easy, but it it's it's pretty reasonable to have a good suite of tests, make sure that they cover the the things that your application uh needs to actually do and have those run consistently so that you're actually testing as your application uh goes over time. Um, non-functional tests um I guess are harder. Uh you need to know first and foremost, uh what your requirements are, and I think that's where I start, is just having clarity on how is your application actually used now and in the future? Um, and uh identifying those those those points that need to be reliable. Performance is a big one, and I uh um I guess in the suite of applications I'm I'm familiar with we have some level of performance testing built into our functional testing. Um, and from there we look at things like integration with other systems and what happens when those aren't aren't reliable, making sure that we're actually testing um that our application still is functional. Uh when the things that we don't control go haywire. Um so that's a starting place. Um, if I remember right, um app services has this great tool uh where you run your unit tests at the same time as also running a MON LBL, it's a line-by-line monitor as the name implies. So you can see not only code coverage, how much of this of my code is actually got tested, uh and also see the performance characteristics of the things being tested. So it's it's a great methodology to really combine the two, at least one area of the non-functional testing. Um, but but to me that that's only one part of the equation, I want to go back to something that both Eric and Matt uh had mentioned earlier, it's that with non-functional testing, the whole really is greater than some of the parts. You can't just do this non-functional testing in a vacuum with spherical cows. You really have to have representative user loads uh on the system, because of how things can interact between different modules and consume resources at different rates and such. So, just, you know, cranking one component up to production levels in your laptop isn't the same thing as running the whole set uh of things that are required in your orchestrator of applications. Mhm.
[6:51]So, one of the things I really lean into for non-functional testing these days is the idea of observability and the value you can extract from your system through that. Um, that'll give you just thinking about what metrics, what KPIs do I want for my application, will help start thinking about, oh, what are what are the other non-functional tests I should worry about here? And collecting those as part of your tests, will, you know, validate number one, that those work, uh and number two, you'll have an idea of what those KPIs look like at different simulated levels of load. So I'm curious, are there any uh existing uh tools that InterSystems offers that can be useful when doing these kind of non-functional tests? I really like the combination of system performance. Um it's a report that goes out and gathers a whole bunch of metrics about uh the Iris instance itself, but it also collects some things at the host OS level. Uh we have these big complicated data platforms, and you need to really understand all of the different layers and how they interact, uh because when you're just because you're doing a SQL table scan, that also involves something like reading from the disk. So it can have uh outsized impacts elsewhere uh if you have a performance issue, it just one small area of your application. Um the other one that deserves honorable mention again is Mon LBL for profiling, you know, a specific chunk of code to know how much time and resources each line takes. Um, and there's a small side benefit there is you can use it to detect things like infinite loops uh and such as well. One piece of using any of these tools is making sure that you're testing on something that actually represents your your application.
[8:33]Um, just just a few things that, you know, we do that um help with that are one, have test systems that actually are the the same infrastructure. If you're using a a certain VM for your live system, make sure that you're actually testing on a similar VM. Um, and uh, you know, where possible, sync real data from backups or or something like that at at a reasonable interval, so, um, you know, you're actually testing on um, what your application has. Not, as Chad said earlier, just a quick test on your laptop, with completely different hardware. I'm gonna push back on that a little bit, I I totally agree and in my ideal world, everybody would do it exactly that way. They'd have a separate duplicate set of infrastructure for their production, uh and for their test environments that are, you know, identical down to the last bite of memory. Um, and you'd have exactly the same data and you'd have exactly the same load. And that's where it gets complicated and costly. Um, just, you know, spinning up even a similar environment in AWS for a short period of time while you're doing your nightly builds each night, that can get pretty expensive pretty quickly. Um, and there's all sorts of regulatory requirements for what you can and cannot have, both, you know, federal government policy uh law-wise, and also, uh, company policy-wise, about what you're allowed to do with data in different locations, who can have access to it, that sort of thing. So for some customers, it's really difficult. Um, and I'd point to some of like the anonymizing tech, uh where you can take your existing data and run it through something that, you know, strips out those SSN's, replaces them with fake ones, et cetera, but does so consistently so you can have kind of the best of both worlds. Right, I imagine that'd be very important in more of like that kind of health care customer side of things. There are, for some domains, you can actually get uh fairly good synthetic data. Right, where it it's completely fake data, but it looks, but it looks good. In other domains that's that's tougher to do, and as Chad said, you know, there are a lot of often times legal requirements that say, you've you can't use this data, right, for whatever, for legal reasons or privacy reasons. And that makes it tough, that's part of why these non-functional requirements are so, that's why it's such a challenge. Right, because it it isn't easy. Um, absolutely testing uh at scale with the same kind of environment that you have in production is really important. Uh it's very very easy for something to look really good when you're running a test case, oh, I've got, you know, I I my my data loader is doing great. I can load 10,000 rows. Well, fine, but what happens when the customer wants to load 100 million? Um, you know, just running things and and running them at size and running them for a long time. Things shake out, and it's it's there are things you want to know about. Do you have any sort of success story example where testing these non-functional requirements proved useful? Or vice versa, do you have kind of a a failure story that really highlighted the necessity of this sort of testing? Yeah, I I wish I could point to a clear success story, uh but it it's very much like the intelligence agencies where only their failures are known. They, you know, you don't you don't hear about all the good stuff that they're theoretically doing behind the scenes. Um, it's the 2:00 a m calls saying our performance has tanked. Uh we don't know what could have possibly changed along the way and trying to unpack all of that and it usually comes down to, well, we push some code, and, well, we made sure all of the code worked, but it dramatically increased the demand on resources on the machine in a way we didn't expect because we didn't have the chance to run it at production scale prior to getting it into our production environment. Right. I I'd say that that's the explanation nine out of 10 times for issues like that. Yeah, sometimes um, you know, the the the value gained is a a running understanding of, uh, where your paint points might be.
[13:00]When you go to move that that big upgrade live, you've already thought through all these pieces, you've already thought about what the performance demands are, what what the integration demands with other systems and and all these things. So that when you go live, you've already thought about them so maybe you you you avoid disaster. Um, you know, just just from having these considerations in mind the whole time. I think, um the closest I have to a real success story, uh for this sort of thing is is um, have having integration tests that really do some load testing. Um at at least, and being able to to to review those over time and and see, okay, my my interaction with this other system um performance is degrading. Something is not correct there. Something's going wrong. Um, and then being able to dig into that and see like, oh, we're actually not being as efficient as we could be, um, with with this integration and and rewriting that to, um, you know, a point where all of our users stop performance benefits. Um, and that wasn't a crisis, but, you know, just that we're we're always thinking about these things and monitoring trends over time. Um, you know, all our users reap the benefit there, which is a success. Right, just paying attention to this sort of thing. That's a really good point. Um, one thing that you can do with an application is keep an eye on performance as you're in production. Right, one of the one of the most helpful things that we can get in looking at a performance problem is what does good performance look like? Right, so when you say, hey, my application is slow today, and we can go back, well, how was it last week? How was it last month? When did this start happening? As Chad mentioned, there's always some kind of change, something has changed and you have to find it. Right, um, the other thing that that I always keep in mind, you know, I always I always say it this way, right, when everything is right, and it still doesn't work the way you need it, then one of the things that's right is wrong, and the sooner you find it, the sooner you're going to solve the problem. Right, which is is kind of a funny way of saying, you know, make sure that you've tested your assumptions, make sure the things that you you think are true, really, really are true. Um I've been surprised by that more than once, so, you know, I always want to stop and and say, you know, how do I know that something that I think is going on, is really going on? Yeah, it's a really soub observation calling back to the classic Sherlock Holmes line, is all I'll say on that one, and you have to really test all of those possibilities to make sure that they are in fact still possible, and I'd say the same is true of the impossible ones, they can suddenly sometimes become possible, um themselves. Uh the dynamic environments we're in are shifting so quickly, um, and I think that that that adds to the challenge. It's not just a static analysis of a system that's sitting there anymore, it's a virtualized system, um, in a data center that you do not own, and there's a lot more that can change. And I'd say that the the number of non-functional testing related issues I've encountered has only risen uh with increased use of the cloud. Um, it's very helpful overall, um, but one of the main results of going from capital expenses to operational expenses is that everybody wants to run as lean as possible. So there's a whole lot less slack uh in each one of those systems before they tip over these days. Um, you're trying to run it 85, 90, 95% utilization, whereas before you because you're buying a server for five or so years, you're going to probably size it to be 20, 30, 40, 50% uh of load, giving you some room to grow because you don't want to go through that purchase process again. Um, very different today and so uh these types of issues are much more frequent and often much more severe. So something that we haven't touched on yet is actually testing for disaster recovery. So when those disasters come up that you mentioned, Chad, How do you know that you're going to be prepared? Uh, you don't unless you've actually tested it, and I I've seen this more than once where a customer decides to implement their disaster recovery plan, and they've never had the ability, the approvals, whatever was necessary to actually implement that plan in advance. Usually there's a cost, and I work with a lot of healthcare providers and that cost for them is inconveniencing providers and clinicians and the patients that they serve. And so nobody wants to just do a DR test in the middle of the week. Um, but there's been several occasions where it's been necessary to do a complete uh DR failover or a restore from backups along the way, um in order to get up and get live again, and that's the best option at that point is just just figure it out. Um, so a strategy I've been seeing more of lately is how can you construct it in as parallel of a way as possible so you can even start up your DR process, um, separately without impacting your production systems. Um, that's it's one possible approach. Um, so but but trying to find strategies like that where you can have your DR up and running. Um, one of my customers, uh their they're fantastic, they have the to me what is the gold standard for working with our products. They have a synchronous failover pair that's in one data center, it's always up, it's always running. Um, and then whenever they're doing their monthly maintenance, the OS patching, just, you know, server restarts, whatever they happen to be doing, they always, every month or every quarter, fail over to the other data center and run from there until the next window. And that's that's to me is the gold standard, they're ready for pretty much anything uh that can be thrown at them. Um, the data centers are in different zip codes, so it would take a heck of an event to knock both of them offline and prevent them from being operational. Yeah. And you want to work that plan out in gory detail because inevitably when you need to do that failover or you need to recover from that disaster, it's not going to be Monday at 9:00 in the morning. It's going to be, you know, Sunday at 2:00 a.m. when your primary system administrator is on vacation in the Bahamas. You need to decide what that range of things you want to cover is, and then absolutely have well-defined plans for each one of them. Right. Yeah, so much of this is knowing your needs, knowing your requirements for these situations, having a plan and actually putting that plan into action before it's the critical moment. Um, because if if you don't know all those things, then you have no plan, you are flying blind, and maybe it'll work out, but experience tells us, it it might not work out so well. Right. Right. I think all of this goes back to the thing I said earlier on, uh is that all of this additional testing beyond making sure that 2 plus 2 does in fact equal 4 in my application, is uh is understanding that return on investment. Uh making sure that you have the ability to continue to provide value in a variety of context to ensure that you're going to be available and it's going to remain confidential and it's going to be accessible uh along the way.
