Farming, not weeding: Changing our mental model of Legacy systems
We tend to talk about IT in analogies. Words like API, Microservice, Monolith, and Data Contract aren’t widely understood, so we use familiar examples to help us communicate.
Some of these analogies are dangerous, however. One mindset is particularly insidious: Software development as a factory. In this mindset, there’s a series of steps to design and “ship” code—which then rolls out the factory door, hopefully never to be seen.
A far better, and more realistic, analogy is farming. Left untended, any software system starts to decay. Sometimes it just gets slower as more people use it. Other times, the software outlasts its anticipated lifespan, and things go wrong.
For example, did you know that Microsoft Excel can only use dates later than January 1, 1900? If you want to calculate the age of someone born before that time, you need to write some Visual Basic code, in order to ensure spreadsheets are backwards compatible with earlier versions. Software is full of compromises that made sense at the time it was created.
As a farmer, you’re constantly watering and weeding. As a manager of software, you’re constantly refactoring and updating, and removing bugs. We need to think more like farmers and less like factories. After all, just as an untended farm turns to overgrowth, so an untended software becomes a legacy.
That word—legacy—is fraught with problems. Thanks to our partners at Code for Canada, I had a chance to sit down with Sean Boots, Amanda Clarke, Gordon Ross, and Afua Bruce to talk about legacy modernization. It’s a theme for this year’s FWD50 conference, because building shiny new things often requires connecting them to rusty old ones—or clearing out the old things entirely.
In the FWDThinking episode we recorded, we touched on a number of vital topics:
- Why legislation and technology need to be much closer together, so we pass laws that can be implemented in tech—and understand what tech we need to create to enact the policies we want.
- The challenges of getting several departments to use a common platform (such as a web form, a sign-in process, a translation tool, or a notification system) rather than building their own.
- How to break a monolithic system into components that can be upgraded, changed, and re-used independently of one another, so we can innovate in parallel.
- The complicated relationship between private-sector “capture” and open-source toolsets.
- How we decide what government should build, and what it should stay out of, on the digital frontier.
All opinions expressed in in these episodes are personal and do not reflect the opinions of the organizations for which our guests work.
Click to read the full transcript of this episode.
[00:00:00] Alistair Croll: [00:00:00] Hi, and welcome to another episode of FWDThinking, the podcast or video podcasts that we run in conjunction with some of the best minds in digital governments in Canada and around the world. This week, we’re especially proud to bring you an amazing discussion in partnership with Code for Canada. Code for Canada is a civic tech organization we have worked with since the early days of FWD50, and they remain one of the leading forces for civic engineering and collaboration within government. And so we’re absolutely thrilled to Luke Simcoe and the whole team at Code for Canada for helping us to put this together. The idea of legacy technology looms large over every government. We spend a lot of time talking about what we’re going to build as new, but we seldom spend as much time thinking about what we do with the existing systems. And those existing systems gather moss. Entropy is real and anything that isn’t brand new [00:01:00] quickly deprecates if it is intended to. So I’m going to welcome today a wide group of people from the public and private sector. So please join me in giving a very warm welcome, and I’ll let them introduce themselves briefly to Sean Boots who works in policy at the Canadian Digital Service. Amanda Clark who’s an Associate Professor at Carleton University. Gordon Ross who’s Vice President, OXD, and Afua Bruce is Head of Programs at DataKind with a storied past in the Federal Government in the US, the White House and the FBI. Hi everyone. Hello.
[00:01:36] I gave a little bit of an introduction, but let’s start with Sean. Sean, you were an early proponent of Digital Innovation at the Canadian Digital Services and now you work on policy there. I think you were part of CDS before it was really called CDS. What was that like?
[00:01:54] Sean Boots: [00:01:54] It was exciting. I think this is around 20, late 2016, early [00:02:00] 2017. And that was sort of when, you know, GDS in the UK, the Government Digital Services at its heyday, lots of cool stuff coming out of 18 FM and USDS and the United States and it really started as a grassroots kind of crew. Ryan Androsoff was a really, really key part of making that all happen who just said, there’s some really cool stuff happening in this space. You know, Canada has a huge opportunity to sort of, you know, make that happen. Yeah, and I feel so lucky that I have been part of that journey from about five people to just shy of a hundred nowadays.
[00:02:30] Alistair Croll: [00:02:30] And CDS in the latest federal budget in Canada is now a fully funded and recurring part of government, right?
[00:02:36] Sean Boots: [00:02:36] Yeah. We’re, we’re real new. It’s very exciting.
[00:02:40] Alistair Croll: [00:02:40] Yeah. That’s amazing. Amanda, how about you? You’ve spent a lot of time in academia, kind of looking at this, me outside and written extensively on the subjects of government IT and legacy modernization and stuff.
[00:02:51] Amanda Clarke: [00:02:51] Yeah. I mean, I think right now is a really interesting period to be a researcher on this topic because you know, I’d say [00:03:00] like having worked on this now for about 10 years, it’s like most of the time you could, you could capture the political imagination around some of them were obvious pain points, like services that were obviously failing or alternatively examples of how new digital methods and approaches were really bringing value to government. That kind of sells well, but the idea of doing the hard work of focusing on legacy systems that are often invisible infrastructure, that is nonetheless absolutely the kind of, the like sleeping giant of effective policy implementation that sets the terms of what you can and can’t do in government. You know, it was always treated like all that’ll be kicked down the line to some future government. It’s not something you would announce in a, a budget with a lot of splash. It certainly wouldn’t be in an election platform. And I, it’s really fascinating to watch that change now where, you know, the most recent federal budget focuses heavily on investing in, in you know, sort of generational IT renewal, [00:04:00] things like making CDS permanent. So I’m, I’m hopeful, but deeply cautious as well, which I know we’ll get into as the conversation unfolds. But it is definitely a fascinating time to see legacy IT being spoken by the Prime Minister, for example.
[00:04:16] Alistair Croll: [00:04:16] That sounds like a good bio to have on Twitter, no matter what, you know, hopeful, but, but careful. Gordon, tell us a bit about OXD and your work with government in legacy system modernization.
[00:04:29] Gordon Ross: [00:04:29] Sure. Yeah, we’ve as a digital services firm, you know, in a past generation, you know kind of a human centered, you know, software design firm and development firm here in Vancouver, we’ve worked alongside of the Provincial Government in BC, most recently the Provincial Government in Alberta and a little bit alongside with CDS recently as well, too although not on any legacy system replacements, although legacy systems were looming in the background as they always are in our discovery project that we do with CDS. [00:05:00] And you know, it’s, our journey often begins and ends with you know, a core legacy system and even just in reflecting about the word legacy, I think we’re often really talking about the combination of two concepts, right? We’re talking about perhaps a, a monolithic system, you know, as, as described, by say agile software types, like Martin Fowler and others whose architecture kind of reveals the bias and the tendencies of a past computing era, right. Sort of a client server paradigm or a mainframe paradigm you know. And if you’ve been around long enough in tech you’ve encountered these patterns and how at a previous time, that’s how you solved either computing problems or you know, CPU problems or data storage problems or network problems. That’s you’ve, you can architect these systems as such. And then I think we’re also talking about the specific languages and products and tool sets that are that these then are built on top of, so, you know, and, and across three projects that come to [00:06:00] mind that I’m personally involved with right now in BC and Alberta, you know, you scratch the surface on those and we have 40 year old plus legacy systems with technologies like Cobalt, PowerBuilder, Oracle forms, just to name a few things that were extremely popular at, in their day that we’re now having to grapple with and, and maybe a virtualized version of them or something. But so yeah, they’re, they’re fascinating because they have, you know, they were once the new shiny thing. Right. And they have a disposition and they have a history and they have an inertia to them. And in some ways, Honey’s expression about, you know, we want to become the new inertia. You know, we’re often there to work alongside of teams to kind of replace those things. But at the same time, recognizing that at some point in their history, there was another team that worked to build it, you know, to, to replace maybe a punch card system or, you know, a paper filing system, some of which is still there as well, too. So I find him, you know, difficult to work [00:07:00] with sometimes, fascinating to work with as well too, I guess.
[00:07:05] Alistair Croll: [00:07:05] So Afua thank you for bringing a, an international perspective to this. What do you do now? And what was your experience like when you were on the public service side of the house with legacy technology?
[00:07:19] Afua Bruce: [00:07:19] Yeah, thanks for asking. So I work at an organization now called DataKind. We’re a nonprofit that works around the world with volunteers around the world on using data science, machine learning and the social sector. Most of our partners are NGOs, but we do we do some work with governments through our NGO partners, actually. Prior to joining DataKind though, I spent several years working in the US for all service I entered as a career civil servant. So really with the intention to really be a civil servant and to really dig in deep working at institutions, including the FBI and the White House may have heard of them. And so have the opportunity to really you know, see the [00:08:00] wide variety of systems that are used internally to the us government and to some of the points that have already been raised. The, really to talk to some of the people who created those systems and have maintained those systems and have spent a lot of time and energy thinking about how to still deliver services with the systems that are there as well as with teams and tasked with modernizing systems transitioning them to new environments, transitioning and defining new processes that can again all bring it back to delivering services to people.
[00:08:32] Alistair Croll: [00:08:32] So Afua, when you talk to someone who built a legacy system, Like that rare and special, you know, COBOL speaking, have you ever had had a conversation with someone like that and said, like, what were you thinking? What what’s their mindset like? How did they feel about being branded as the author of the legacy system?
[00:08:51] Afua Bruce: [00:08:51] Well, I don’t usually, I don’t usually lead with what were you thinking? That’s not, not my style. But really just trying to [00:09:00] understand sort of the context around why systems were created. Right. To, again, some of the points that have already been raised, they were created for a reason and no one goes into government. Hmm. Most people don’t go into government with the explicit intention to dismantle systems and to make it not function. And so I think when you start these conversations, you have to be really intentional to say, well, what can we learn? How did we, how did we get here? And sort of what were the factors that led to the creation of these systems in this way? What are some of the factors that have led to the maintenance of the system in this way? Why, why has it not been a transition to something you were since then? Sometimes that is inertia. Sometimes it’s because people have tried and failed multiple times. Why did that happen? And so I think really just starting those conversations with trying to understand some of the historical context, even within a specific agency, around whatever legacy system you’re looking at.
[00:09:54] Alistair Croll: [00:09:54] So Amanda, one of the questions that keeps coming up, and you touched on this a little bit, [00:10:00] is that when we come up with a policy, we want to implement it. But if the people who are coming up with a policy, aren’t aware of what’s possible with the implementation or the technology, they can write laws that can’t be enforced. Let’s say you need universal digital identity and it doesn’t exist. And conversely, if they were more aware of what technology could and couldn’t do, they might change the priorities to make some necessary technologies that keep getting glossed over, more urgent so they could implement those policies. Can you talk a little about the dance between policy and implementation that happens in technology?
[00:10:32] Amanda Clarke: [00:10:32] Yeah. That is such a, an important question. And I think is actually, you know, I began with, I’m excited that these kinds of topics are on the political agenda now. And I think that’s the hook that will keep it on the political agenda is if you, if you kind of broaden the constituency of people who really care about legacy systems, so like the IT folks and the digital people are like fired up about this. All of us have long been concerned about this. But it’s when, [00:11:00] the policy advocates outside of government, the policy teams within government the political partisan machine of the government in power becomes aware that they can’t deliver on something because the systems aren’t there that, that this will start to be increasingly politicized and that’s precisely what we need. And I think what that’ll tap into is a really longstanding challenge at the heart of government policymaking, which is that we siloed off policy design and all of the legislative, and then civil service led activity around policy design from the implementation folks and the digital service teams that we’re now seeing cropping up globally are always talking about linking policy and implementation. And actually they they’re in chorus with decades of policy research, which identifies this as one of the reasons that governments often fail to deliver on some of their core promises. So technology is, I think the, the sexiest and most important issue that politicians and [00:12:00] policy advocate groups, that academics who are interested in their niche, little policy issue, whatever it might be, homelessness or pharmaceuticals. Like if you don’t understand tech and you want to study those things, you’re going to hit walls constantly cause you’ll be like, why can’t we do this? And it’s like, because the infrastructure isn’t there, the data’s not there, the databases don’t work together, et cetera.
[00:12:22] Alistair Croll: [00:12:22] Gordon, you mentioned this idea of a monolithic system. Can you maybe provide a little example of what’s the difference they want to come monolith versus something that’s made of composed microservices or a more modern architecture?
[00:12:36] Gordon Ross: [00:12:36] Sure. I mean monolithic systems, which again, architecturally we encounter tend to have, again, being designed in a different time, in a different place, you know, say sort of, I don’t know, 20 years or more ago. And oftentimes sort of the business logic the data, it’s all compressed often into a [00:13:00] particular sort of layer if you will, of the system. So for instance, I don’t know, working on our childcare subsidy project in Alberta you know, a big sprawling database with a huge data model, a ton of stored procedures where, again, the sort of logic of, you know, eligibility say for childcare or something like that is, is enshrined and coded inside of that database itself. You know, as opposed to the sort of more microservice oriented modern architectures where you know, it’s, it’s a web service, it’s an API. You, you ask it a question, you send it a request and it sends you back a nice discreet well, formatted, bundled up bit of data that you can then manipulate and go on with your day, you know and your, in your new system. Right. And this kind of.
[00:13:46]Alistair Croll: [00:13:46] I find that if I’m a government executive, who’s not super technical, you just use words like microservices and APIs and so on. How do you get to Amanda’s point? How do you, how do you translate that? Or vulgarized that [00:14:00] into, here’s why that matters, here’s why a monolith is bad. Here’s why a stored procedure will mess with your ability to implement policy. Like how do we make that? Are there a good analogies or clear explanations of why complexity and, and coupling between systems are bad and, and how do we get people to understand that? Because if it seems like if, if we’re still talking in terms that we understand as technologists, but the wider public doesn’t understand why a monolithic database is dangerous to public policy, we’re not going to make the kind of progress we need to make.
[00:14:34] Gordon Ross: [00:14:34] Sure. And perhaps it’s cliche. I mean, we often use you know, houses and homes, you know, and metaphors, right? If, if, if we can’t, if we can’t have a degree of literacy, you know, amongst our leaders. And again, not speaking of our work with recent partnership with Code for Canada on some work here in British Columbia, around literacy and digital literacy for leaders which I think is important. I think they should understand what a microservice [00:15:00] is. Maybe not how to code one, but you know, at least know what, what, what it is and we’re talking about it. But you know, we tend to, I tend at least to go to a good analogy or a good metaphor, right. You know, for anyone that’s ever lived in an old building, you know, we, we talk about sort of the plumbing or the wiring of the building or the materials of the building and try and anchor it against a good metaphor that, that they can understand. You know, and, and, and talk about the difference between things like modernization and transformation, or, you know, we’re going to take out the knob and tube wiring in your home and replace it with, you know, up-to-date modern wiring, but essentially your kitchen is going to be the same kitchen at the end of the day, you know or no, it’s going to be a brand new novel kitchen, and we’re going to have to plumb it in some new, amazing technology, like natural gas, you know, in order to get this new fangled, you know, range in or whatever. And these things sound. I dunno a bit trivial in some ways, but I think we do often refer to the physical and known world through analogy and metaphor to help land it with people who are like, it’s [00:16:00] a lot of jargon. It’s a lot of buzzwords. I don’t know what an API is. I don’t do. I really need to know, you know, I don’t know what a stored procedure is. I didn’t know that. And I don’t know now it’s not my business. I’m not interested. Right. So. We, we tend to use again, lead with analogy or metaphor to try and bridge the gap. And sometimes it works and other times it’s like, you know, while it, well, there’s a great Venn diagram and images of organization, Gareth Morgan’s book about metaphors used for organizations, mechanical, organic, all that. And it says a metaphor helps you see the things that are similar but ignore all the things that are different on the side, right? The risk of the risk of using metaphor cause legacy system, isn’t an old house. You know, that shares some of its properties. So.
[00:16:42] Alistair Croll: [00:16:42] I think, but it’s talking about those properties, the idea that I can go buy a new fridge and plug it in, and my fridge will use a one 10 volt outlet and if it’s got a water filter, then it uses a standard three quarter inch water pipe with the right threading and so on. And I can go make that thing modular. So if it breaks, I can replace it something [00:17:00] else. If there’s a more efficient fridge that appears in the market, I can plug it in. People don’t necessarily, we take that for granted in a house.
[00:17:07] And so we don’t talk about how difficult it is to swap out the fridge in a computing analogy, but it does seem like what we’re trying to get to is being able to go to Home Depot and buy a fridge and plug it in and it just works. Rather than having to build a house that includes refrigeration systems. And if you’d like to change your fridge, you got to go get a new house, which is what we’re. Yeah.
[00:17:31] Gordon Ross: [00:17:31] And the best of the best thing about standards is there’s so many to choose from. Right? So, you know, all of these things, it’s like, there, there wasn’t even, you know, there’s not even an electricity standard, right because everyone was using their own custom voltage or something. And you know, there’s not even. So, you know, you often find yourself in these funny instances where there’s competing standards. And again, that’s where it’s the house metaphor breaks down a bit. We’ve been living with standards and other parts of our lives for a long time, but maybe not necessarily in, you know, in technology spaces and in large [00:18:00] bureaucracies.
[00:18:00] Alistair Croll: [00:18:00] Sean, can you think of examples where the available IT has constrained policy or made it harder to deliver something that people thought should be easy?
[00:18:13] Sean Boots: [00:18:13] Yeah. I mean, there’s, there’s a lot of examples of sort of that come up where, you know, things that the government did that appeared easy was because some kind of IT system was already in place that was making that possible. And things that were difficult is just because, you know, the system didn’t exist, the data didn’t exist, the infrastructure doesn’t exist.
[00:18:34] CERB is a really interesting, kind of like exception that proves the rule where, you know, teams at the CRA or the Canada Revenue Agency and Employment and Social Development Canada, were able to sort of like tweak an existing system in order to get, you know, in order to get money to people who needed it when started the pandemic. That’s often not the case and there’s a lot of examples, both in Canada and other countries where, you know, teams want to do a thing or governments want to do a major program, but [00:19:00] you know, they didn’t have the data, they didn’t have the infrastructure. And often they might not actually realize that until after they’ve already made political commitments. Amanada, Im sure you want to add to that.
[00:19:10] Amanda Clarke: [00:19:10] I just want to ask a question though. I mean, on the CERB example, wasn’t there also this moment though, where all the people involved were kind of like this might explode, like we are fully anticipating that this won’t work, which like, frankly isn’t good enough. I mean, there’s a big accountability question there. I mean, if the central response benefit that Canada had to the pandemic fail to get checks out the door, like people would not be paying rent. They wouldn’t be paying for medication. So, I mean, I love that it worked, but I’m also like, I keep hearing people talk about how it’s a great example of why we should keep using COBOL and like not like, and I’m like, no, it’s scary. It’s scary that it works. And that’s scary to me.
[00:19:51] Sean Boots: [00:19:51] Yeah, no for sure. There’s so there’s some really interesting examples too, in the the States Marianne Bellotti from, from the States is like a [00:20:00] really brilliant engineer and author on this, talking about how for a lot of US state unemployment insurance websites that, you know, crashed under the load, it wasn’t actually the COBALT system. It was sort of the layers of new ish, but also legacy systems like Java based websites that kind of live in between people applying and the COBALT system behind the scenes. But I think, I think you’re, I think you’re right. And I think so The part that’s tricky is sort of the reality that we don’t necessarily know what programs or policies we might need in the future but I think it’s safe to say that the systems that we have right now, aren’t going to be good enough for future problems. And I guess the one of the underlying questions is how, how quickly can we turn around a new system that does what, you know, the government priority of the day, or like responding to an emergency of the day is because the typical turnaround time to sort of start a new it program is maybe like five to seven years in a typical government department. And it’s like, if [00:21:00] you have an emergency tomorrow that you need some kind of really critical thing that Canadians or people around the world are depending on, you know, five to seven years to turn around a thing isn’t good enough anymore
[00:21:11] Alistair Croll: [00:21:11] Also, it’s like a regime change too, right? Like, like the political will evaporate. Afua, at FWD50 last November Paul Glover said that Canada took a roll out of teams for communication. And they rolled it out to all government employees in six weeks. And I asked them what the plan was originally, he said it was about three years. And you know, the idea that three years is acceptable when six weeks was possible is something I don’t think that the public is quite aware of, but to come back to something that Gordon mentioned, when we talk about these microservices, these building blocks in order to build something more quickly, we often have to be able to compose it out of pieces that the government systems need to deliver. And, and some examples of that, that I’ve seen at GDS and other governments [00:22:00] something like a form. A form is a very simple thing. You enter some data and you either check a box saying, I promise this is me, or you log in and now that data is stored, but there’s no need to build a form tool every time you need to collect data, you should just do a stand on the back of an approved form tool that has the right things for accessibility and works in multiple languages and all that stuff. You might also have one for publish and subscribe, which is simply telling a computer system, Hey, I’d like to publish something to you, or I’d like to subscribe to updates about that thing or notifications. It seems like these building blocks are composable elements of more complex systems. And then Sean, if we had these building blocks, we would be able to stand up things to top them much more quickly, but the building blocks themselves are very sexy, you know? No one’s going to say, boy, I, I promise to make a publish and subscribe system, right. In the liberal platform or conservative platform of particular election. And yet when we look at I’ll take an [00:23:00] example, Canada has a case management system, basically, it’s just like a CRM or a trouble ticket management thing but every time someone comes across the Canadian border, applies for refugee status, stuff like that, there’s a use of this case management system. That’s an incredibly complex and it’s been around for a long time, big system, costs a lot of money. It seems like trouble tickets should be one of those fundamental building blocks on which we can build other things. And you should just be able to go. I’d like to use that trouble ticketing system to manage whatever trouble tickets I might have. But we don’t seem to do that. How do we get from this legacy mindset of this system was built from one thing only to a more composed system where you have lots of different blocks and you’re dragging them together in useful ways. Can we get there with the systems we have now? And I guess I’ll ask Afua that. Because she’s learning all about Canadian systems in real time, as we have this conversation, what do you [00:24:00] think, Afua? How do we get from, from big systems with one purpose to convincing people that the, the building blocks are just as useful as the end product?
[00:24:10] Afua Bruce: [00:24:10] Yeah. Oh man. I love this question. Also as a side note, I really enjoyed learning about the Canadian systems in real time. $2,000 a month, every month. Amazing. Man. Impressive. Now I’ve lost my train of thought. Oh, building blocks. Right. So the reason why I find this so exciting is because the boring things sometimes are what matters the most. I often tell people, structure allows freedom, and I think this is true for as you define processes that you need to implement in order to make sure that people in your organization know how to get work done. It’s true. When it comes to coding into different systems, the structure allows freedom really applies to a lot of things now is making a form that can be built off of [00:25:00] repeatedly exciting, probably not to most people. As you mentioned, Alistair, it’s not going to be exciting for a campaign. It’s not going to be something that might lead to protest in the street about, we really want the building blocks of these systems. But it is what is necessary and it is what matters. And so I think, you know, as we think about what matters in digital services in the civic tech space more broadly. I think that we really need as a community to really turn our eyes toward some of these things that just aren’t exciting, but are truly necessary and really make sure that we’re putting perhaps even more value on those, the other work. I think this idea, I also think of it sometimes as sort of a digital infrastructure and that, you know, infrastructure, if you don’t have roads, you can’t actually get anywhere roads, particularly exciting. To someone, not to me. But they are absolutely necessary to be able to have communities connected to each other, to be able to get to your jobs. I think the building blocks that you’re describing [00:26:00] are the same. They require the same investment into what we’re doing. They require the same dedication to make sure that services can continue to be delivered even as technology changes, even as requirements change, even as policies and laws change, that then require changes to these forms that we’ve just talked about or something like that. We really need to make sure we’re investing in some of what’s traditionally considered a little bit on the boring side.
[00:26:30] Alistair Croll: [00:26:30] Well, well, you know, power line, power grids, taxes aside, and highway infrastructure and other elements of public infrastructure on which we build things, you know, the US has spending a tremendous amount of money on an infantry infrastructure plan, but there’s still this arm wrestling about what constitutes infrastructure. Is daycare childcare, considered infrastructure, right? How do we get people to realize that we need to redraw the lines of what’s [00:27:00] infrastructure because everyone’s okay when the government builds a highway, but they’re not usually okay when the government builds like a competing email service for all its citizens although that’s something that they may want to build. Sean, you want to take that? How do we, how do we get people to think about redrawing the line for what government infrastructure is?
[00:27:19] Sean Boots: [00:27:19] Yeah. Afua, if you want it to, to jump in, I’ve got some thoughts, but.
[00:27:23] Afua Bruce: [00:27:23] Thank you. I’ll just say really quickly. I think that we need to, we need to also to do a better job of making the case to the average citizen you know, saying that, you know, creating a new email system for one agency. That’s fine. Creating another email system for another agency. That’s fine. I think we need to do a better job of tying it to, if you have people in so many different agencies, always creating new email systems, it actually means that the systems that are then used to deliver the, the CERB benefit don’t get as much attention as they want. It means that the systems that you need to make sure that you [00:28:00] can get access to your, your water and to other critical services you have, don’t actually have the attention that you want and the pain points that you have in accessing these other services. Part of the reason why those are taking so long to update is because we’re busy spending so much time on recreating the wheel so many different times in so many different places, Sean, back to you.
[00:28:20] Sean Boots: [00:28:20] Yeah. I think, I think reinventing the wheel so many times in so many different places is a really good way of summarizing that, cause I think you see a lot of the, sort of like myth of uniqueness in a lot of government institutions. That’s like, our department or our institution, or a tiny division is so different from everyone else that we need our own very unique system just for us. And that leads to a lot of, sort of, yeah. Reinventing the wheel over engineering things versus kind of like Alistair, you were talking about sort of like the idea that you have these like common building blocks, common platforms that, you know, meet 80 to 90% of an organization’s needs, but that are already ready to go that are already being used [00:29:00] all the time. I think the other, the other part of it is that, and you see this with like, you know, like the notified platform in the UK, that our team has also adopted like different, different kinds of platform tools. One thing that people don’t talk about very much is the fact that in these sort of government, as a platform teams, you have a group that is actively keeping it up to date actively, you know, actively maintaining it, actively improving it. This is not just about the building block. It’s about the fact that you have a team behind it which is often not the case for a lot of other typical government. It projects where, you know, you spent five years building it, then you’re done. And, you know, it’ll just sit there and keep on theoretically, keep on working for the next 10 to 20 to 30 years until all of a sudden, you know, it’s, you know, today’s COBALT system that is like falling apart. And so it’s, it’s, it’s like, there’s, there’s the systems themselves and the sort of architecture like Gordon was talking about. But then there’s also the sort of organizational environment, which is basically like: Are you an organization that continually iterates and improves [00:30:00] your stuff or you an organization that finished building it finishes building it and then just sort of like we’ll use it to sit and kind of fall apart.
[00:30:07] Alistair Croll: [00:30:07] Well, this is, I think one of the biggest challenges when I’ve talked to non-technical government executives and Gordon, you kind of touched on this a little bit, but since we’re talking analogies, the difference between a factory and a farm is that when you put someone in the factory produces something, it rolls off the conveyor belt in someone can now run it. And when you have a farm, you tend to it, you water it, you plant it, you weed it. And if you don’t keep doing those things, entropy kicks in and it does seem like we need to get people to realize that because IT is much more like farming and much less like factories we have to, when we launch a product or a service, understand that there’s gotta be a farmer that works there too. So Gordon and Amanda, I would love to hear from you. How, how do we deal with this tension around the, you know, training people that it’s [00:31:00] not fire and forget, it’s this ongoing thing and that the service comes with humans. But also how do we balance public and private sector interests in that technology because government’s not going to build it all itself.
[00:31:14] Amanda Clarke: [00:31:14] I’ll, I’ll jump in. Yeah. I mean, I think the, the two points there, like one, you just need better education, better training. I think we have to really think about how we define the core competencies of public managers today. And that is partially a continuing education question because there’s a cohort of existing executives who really need to understand the need to like think, you know, to Afuas’s points and Sean’s points just now around, you know, thinking in terms of platforms and common components and ensuring there’s interoperability so that you’re not like rebuilding what the department next door has already built. So, that is like just a tr training and [00:32:00] education piece. I think there’s many people in government, a very few who really understand that. And because of that, the second point you raised around kind of private sector influence here. I mean, evidently the private sector is, is going to be an important partner in, in resolving the challenges of existing legacy systems and in setting up new systems going forward. But if we keep kind of buying the narratives and products of the same firms that got us into this mess, it’s kind of a recipe for disaster. And there’s, I think not a lot of incentive for many of these firms to advocate for common components or for governments to build their own digital service teams who are building more internally because that cuts at their profits. And I, I am, when I, when I, you know, I led with this idea that I’m hopeful, but also deeply cautious. I think the thing that gets me most worried is how embedded the vendor capture just is, you know, I mean, it, it runs so deep in government whether it’s evident in [00:33:00] free quasi free executive training that some of these firms are running for current government employees. The kind of ways in which they embed themselves in departments and become like quasi employees, so that you’ve got, you know, the big management consultancies and the big tech firms working hand in hand with public servants to design systems and to think about legacy renewal. Sometimes, I think, you know, the government of Canada in particular, and I know it’s not exclusively a problem there, but has a little bit of like Stockholm syndrome where we’re almost like in love with our captor, you know, the very same firms that got us into this mess are the ones we keep turning to to fix it. And it’s just, it’s very dangerous. And I think something we need to really talk a lot more about when we think about where we’re going forward in managing legacy systems and avoiding building new ones that we have to clean up in decades to come.
[00:33:49] Alistair Croll: [00:33:49] Gordon. Did you want to chime in on analogies here?
[00:33:53] Gordon Ross: [00:33:53] I mean, I’m, I’ve got a couple of trains of thought. One is obviously the role of the private sector of which I’m a [00:34:00] part you know, and having partnered with government over the last 25 years, although given our size and our reach you know, a small but mighty here in Vancouver, we certainly don’t hav, the kind of engagements or contracts, to maintain and manage, you know, these legacy systems you know, the way, some firms that we work with do. You know, that said, I have to simultaneously respect some of the knowledge going back to the materials that I see in the sits down at sort of the developer level and the kind of craft involved and the sort of practice that those folks have often when it comes to knowing how to manipulate older technology and working within a whole bunch of constraints as well, too. Right. You know, Again, back to analogy and metaphor. You know, you, I live in, you know, I’m sitting in 110 year old house right now. Right. And you know, [00:35:00] somebody who I need to come in and fix some 110 year old piece of it, needs to be a skilled craft person to be able to do that. And sometimes we really, you know, I witnessed that on a daily basis. I, again, the value capture is, is definitely an issue. And there, there can feel like there’s a moat, you know, around these systems and as you approach the system, because we’re always there to like try and, you know, do something to it, right. Or change it or whatever. And, you know, we get to the moat and we’re like, Hey, we look up at the castle and you know, they shoot arrows down at the team that’s trying to come and touch the system because a good day for that team is kind of like you know, four nines. You know, don’t touch our system, it’s working, you know, stability. It’s a conservative force, right? Because, because there’s a business sitting on top of it still, they still have to print childcare, subsidy checks in Alberta. They still have to run the courts, you know, in BC or Alberta, they still have to, you know you know, name any, like it’s still to cut checks, you [00:36:00] know, for insurance. And so we’re there often to, to mess about with that at the same time, as someone’s trying to. So let’s stop touching the system, right. Get away sort of thing. Afua, did you want to comment on, on perhaps some of the tension there in terms of changing systems and keeping them running at the same time?
[00:36:18] Afua Bruce: [00:36:18] Yeah, absolutely. I mean again, the mission of government is to deliver services. Sort of let’s, let’s go with that for, for now. And so when you were changing systems, you can’t afford to let one of your systems fail. This is just not an option that is possible in the government sectors. Other sectors do perhaps to have that luxury. Sometimes governments think they have luxury things. Something that we see from time to time in the US governments, government websites. Yeah, decide to close every night for 10 to 14 hours because, because of system updates being required for 10 to 14 hours every single night. And so [00:37:00] there, there are a couple of things that I think really go into play here as a leader on this. I think one is recognizing that when we talk about updating legacy systems, we’re not just talking about updating the technology that’s there, but also the processes. So really also using it as an example, to think through what are the, what are the policies that should actually be in place? Not necessarily what policies are we enforcing now, but what policies, what people need to be included in the process, what people need to get access. And then how does that drive? Right. What processes actually need to be done. I think that needs to underscore any type of technology systems transformation that we talk about is just what are the true underlying policies and process changes that we need. Then on the other hand, I mean, we have to talk about procurement at some point, right? The procurement process really drives a lot of how do we have these systems in the first place? How do we maintain these systems? How do we then identify what new systems are built and how [00:38:00] they’re built. And so, efforts that various governments are doing to help shorten procurement processes, help think about what agile procurement looks like, how think about what it means to really just look at it who some of the vendors can be. Do you need to go back on to Amanda’s point to the same big vendors that you’ve already gone to? Can you that you’ve gone to repeatedly in the past, or can you take a look at some smaller vendors who might use different methodologies to then deliver things quicker, more efficiently? So you get down to that quick delivery and three weeks or six weeks, even though three years might have otherwise been acceptable.
[00:38:39] Alistair Croll: [00:38:39] I can keep asking questions, but I’m much more happy for all of you to just chat amongst yourselves. So don’t feel shy about chiming in, on this stuff. I guess my thoughts are on legacy are if we, if legacy stuff is a consequence of monolithic architecture, a lack of understanding that components work better, [00:39:00] a vendor lock-in or vendor capture, whatever you wanna call it. We’re talking about the largest buyers of IT infrastructure in the world, most governments and the strides that open source has taken are tremendous. The thing that the open source movement missed was that you could put a license into open source that required you to give the code to someone else. But the hard part was in the operation of it. Amazon and Google, all run on open source stuff. They just run it as a service. And so it does feel to me, like we took this idea of open source and then we realized that the hard part was implementing and running it and maintaining it so that just because it’s open source doesn’t mean it’s free, which was the old assumption, but government is the biggest provider of IT infrastructure in the world. So or the biggest buyer of IT infrastructure in the world. So how do we reconcile those two things? And I’m sure you all have opinions on that, but Amanda, I think I’d love to hear from you from the work you’ve done in the research you’ve done in this field, how we [00:40:00] reconcile, you know, open source software as a service, vendor capture and all those things. And do you think we’re going to see an upheaval in that model or is it going to be the status quo?
[00:40:10] Amanda Clarke: [00:40:10] I mean, as I mentioned before, I mean, a part of this is a, a re-education campaign within government. And I think it’s like an easy narrative to tell Afua well, put it very beautifully actually. I think if you were to sit down with those public managers or most ministers and explained to them that there is another way that it, you know, and use the, I mean, I’m always careful about, you know, it’s faster, it’s cheaper because sometimes that just means it’s like a worst product, but if you say it’s as good or better, like more public value meets user needs better solves your policy problem. Like, and Hey, by the way, we can do it in six weeks, not three years. Like that’s just easy to sell. We need to do more of that. Part of that is like putting more people in government who have that perspective. I think you’re really only going to see a lot of these changes if you see a shakeup of the CIO class and the executive class, like we just need, and you’re seeing that in the United States right [00:41:00] now we’ve been, I have a guest cameo for my mother-in-law. Hi, Jackie. And, and, I’m in her house right now. Speaking of houses and analogies of houses. So anyway, all that to say you’re seeing in the United States right now, like really, really exciting appointments to senior leadership positions, people who really get this. And that’s when you’re going to see the change. Like if the old guard is there it’s, it’s, there’s just too that the playbook is too ingrained and it is, it can be difficult to teach an old dog new tricks. And. I hope we can see that kind of a shake up in Canada as well.
[00:41:35] Gordon Ross: [00:41:35] We’re, we’re kind of, we’re kind of back to the same question that was into that a second ago, around procurement though, in terms of the attitudes or beliefs that you know, there’s a builder by decision that’s faced, you know, with like a technology thing, there’s maybe something that’s off the shelf that we can get or a platform that we can install and configure. And some people truly believe ultimately that it’s not their competency as public servants [00:42:00] to design build and manage and maintain and operate these systems over time, because that distracts from the delivery of the service in another way. Because that’s headcount that they don’t want to have that that’s like not their core competency or not their, you know, not their kind of job description. Right, right. In terms of their mission or mandate or, or remit or purpose or whatever the, the thing is. Right. And it’s really interesting to kind of see that, you know, I’ve been around in government working alongside government long enough now to sort of see this sort of pendulum swing a few times, you know, from, from central central, you know, bring it all in, we’re going to do it. We’re going to be the ones who own this and are competent and are, you know hold the, the, the direction setting and the implementation of technology and the pendulum swinging out the other way to being like, you know what No, like it’s somebody else’s business to be able to do this. And to the sort of outsource, you know, extremes, right, where core systems of record in government are [00:43:00] entirely run by, you know, third party, private sector for profit organizations as well, too. I think ultimately the context of this is like, you know it’s both. You know, I don’t, I don’t think, I don’t think it’s a one size fits all, but the dualism is so strong often. Right. And this, this sort of like pendulum swing tends to happen, you know, with frequency in government in terms of it shops, CEOs, their kind of attitudes towards the things.
[00:43:29] Sean Boots: [00:43:29] Yeah. I think to, to Amanda, to your points are there too. I think there’s some interesting sort of like, like, yeah, I was like a few folks are mentioning sort of like the ethnography of what is it like to be in an IT executive or CEO role in government where your own future career paths almost always leads out the door to working for one of these really large management consultancies or it vendors. And I think that changes the, like the incentives and the relationships that people at that level. And up with there’s some really interesting stuff that Bianca Wylie in Toronto is publishing these days about sort of [00:44:00] the, the value of essentially like publicly owned technology and infrastructure as a counterpoint to the sort of like commercialization or like vendor intermediate interface with government that you see with things like, you know, like the sort of software ecosystem around tax filing and stuff like that as two different kinds of poles for how, how people interact with government services.
[00:44:24] Alistair Croll: [00:44:24] Sean, you mentioned earlier the idea of these component services that people could use across departments, but nobody does. They all build their own cause they think they’re unique. How are the incentives in government set up so that everybody wants to build their own forms tool instead of using the common one?
[00:44:43] Sean Boots: [00:44:43] That’s a great question. I think it’s, I think it’s. Part of the sort of waterfall easiness of the process is that are actually on, you know, on the outside of the actual software part where, you know, a given department is going to be planning what their sort of [00:45:00] future system looks like as a very like word document people around a boardroom table kind of approach. Write that all down, lock it all in. Before, any kind of, sort of design research, talking to the users, actually getting a feel for what’s needed. And so it’s kind of like the uniqueness comes from the fact that that’s being done in a vacuum without any kind of input from actual users or feedback. And then it gets locked into budget proposals, treasury board approvals, you know, internal plans, whole bunch of waterfall documentation that even if way down the road you say, Hey, you know what three of the departments built the same thing could have just used theirs. It’s already too late because you already have $80 million that you’ve been given by the finance department to go build your unique thing and there’s no way to walk that back. And so it’s, it’s, it’s the, I think the bigger challenge, and this is something maybe for a future webcast is how do you, how do you de waterfall not just government software development, but government program design [00:46:00] legislative and approval processes, which is a really complicated question.
[00:46:03] Gordon Ross: [00:46:03] I, I just think like that I’m experiencing this right now as a portfolio of projects working in government that have kind of common components across these things, like the coordination. I think the coordination costs, if you’re on a team, working on a project, working on a system, sometimes the coordination costs seem higher when you’ve got to go off and collaborate and find the commonalities to determine whether or not you can reuse these things. There’s maybe not a ready at hand thing that’s already built. And you’re kind of like your incentives on a project, you know, or like you’ve got a timeline, you’ve got a budget, you’ve got promises that you’re supposed to be, you know, fulfilling on. You’ve got your thing that you need to ship, right. So oftentimes kind of the future benefit of reuse or even the current benefit of reuse is somehow misestimated or I don’t know deemed, deemed somehow, not as, you know, high as the sort of coordination costs to just try and get it done. Right. [00:47:00] Like my, you know, I’m kind of in my lane, we’re in our project. I know there’s that team over there, they’re working on something similar, but like, it’s going to take me off of my path which is like, by the way, you know, strategy and delivery and all that like I got to ship a thing, you know, at the end of the month. Right? So sometimes that gets in the way, just at a really pragmatic level across teams. And you know, that, that was a conversation this morning with a bunch of practitioners across a number of projects in a provincial context that we were working in this exact conversation.
[00:47:26] Afua Bruce: [00:47:26] I don’t know that I can add much more to what Gordon has already said. So I think just emphasizing the fact that, you know, it’s. Although, you know, my ideal state might be for everyone to sort of build off of the same building blocks. Part of the reason why it’s not done is because it’s challenging to do it. Not because, not just because everyone thinks they’re special, which everyone of course, very special, very unique. Every agency is. So unique and so different and has such different requirements than anyone else has ever possibly thought of. But also the effort to how all the [00:48:00] conversations between different agencies to determine what those common requirements are, and then determine what a level of effort is required to build on those and to do that over and over and over again. Right. And with every change that you need. I think it’s important to just name that is not a trivial exercise to do this. And that also leads to part of the pushback or resistance to doing this type of work. And so, as we think about how we message this difference, we have to acknowledge the amount of effort that will go into making this work long-term for folks in order to then build something that could actually make that difference.
[00:48:33] Alistair Croll: [00:48:33] Well, I know you’re all busy saving the world and building important things. So I’m going to ask you to give me a 30 second answer to the following question. First. Question for all of you, here we go. 30 seconds. This is your, your recap. You know, that scene in the Matrix where Neo plugs in and sits up and goes, I know Kung Fu. If you had that magic brain Jack, and you could plug it into the head of public senior, public [00:49:00] sector executive, what would you download into their brains? What one experience or a piece of knowledge or understanding or perspective would you download it in their brains to produce the greatest improvement in government technology modernization that you could and I’m going to go with Sean first, cause he’s got a big grin on like he’s figured out the answer already.
[00:49:22] Sean Boots: [00:49:22] Not, not the complete answer, but I would just say small is beautiful. If your thing is like a hundred million dollars in five years, I can almost promise it’s totally gonna fail. If it’s like, I don’t know, half a million dollars and you’re going to finish it in six months. Maybe we’ll actually finish it in seven and you do the next thing and you need the next thing. Doing things small is always better than doing a giant thing, even though all the government incentives are like, Oh, we’ll just do one budget proposal for the giantest, greatest thing ever, all the incentives or the other around. But if you can do things that are small, you’re way more likely to succeed.
[00:49:55] Alistair Croll: [00:49:55] Alright, Amanda, what would you download into someone’s brain with your magic matrix machine?
[00:50:00] [00:50:00] Amanda Clarke: [00:50:00] I would dial it in the brain the ability to, or the, the knowledge that it’s so key to say, who else has tried this? Like, who else could I talk to first so that you don’t reinvent the wheel?
[00:50:14] Gordon Ross: [00:50:14] I think, and it’s something we’ve said to recently, as soon as you engage with a legacy system on a digital service project, you inherit all of its history, all of its disposition, all of its ways of being and your project fundamentally changes after that, you know, the kind of Greenfield versus like legacy integration projects. As soon as you try and write a piece of data in particular writing data to legacy systems, you will encounter 20, 30, 40 years of history. And that is not to be dismissed easily. [00:51:00] I guess. It’s, it’s sort of, it will consume you. Entirely. You need to, you need to deal with that force and, and, respected as well too, I guess cause it’s, it’s complex.
[00:51:13] Alistair Croll: [00:51:13] All right, for channel your inner Trinity, what would you download into public sector?
[00:51:17] Afua Bruce: [00:51:17] The ability to answer the question, what do we really need to do here? And that can be, we need to find a policy. We need to fix some sort of historical wrongs built in, or we need to actually dig in and do some really serious redesigning of a system and getting other people to plug in. So what do we really need to do here?
[00:51:37] Alistair Croll: [00:51:37] And I think for me it would probably be who else is doing this already, which echoes some of what you said, but I don’t think we can find cross pollination until we find similarities. So thank you all very much for today. I think if I have a few takeaways from this amazing conversation, first of all, the term legacy is kind of a misnomer. It’s more about whether something is neglected or maintained. And the farm’s not [00:52:00] factory’s idea here that we are growing stuff and used to be weeded intended, nurtured otherwise it will get covered in overgrowth and become incredibly unmanageable that private sector dance with the private sector can exacerbate some of these issues because their lifecycle and maybe motivated by other things, shareholder, bottom line.
[00:52:21] Lock-in capture change in standards, whatever. The departmental models that we have today are not incented for cross departmental sharing. You want to get your budget, capture your budget and hold on to it. And finally, this idea that legislators need to be more tech informed and policy making and implementation can’t be taken separately that you’ve got to know what’s possible in order to write laws. And then in order to enact laws, you have to build the foundations for those things to be implementable. So great conversation. No, the answers here, but a lot of good questions. Thank you very much to our partners at Code for Canada, for making this possible and to all of you for spending some time with us. I know [00:53:00] you’ve got places to be and problems to solve. So thank you so much for being a part of this conversation. And we look forward to seeing you at FWD50 and elsewhere. Keep up the great work. We love you. Stay safe and we’ll talk again soon.
[00:53:28] I just realized I have paint on my face from painting, which is probably not a good thing, but people are going to wonder what it is. So let me just get that paint.