FWDThinking Episode 2: Empowering product managers

The second episode of FWDThinking dove into product management. Product managers are a must in the tech world, owning every facet of a product from conception to end-of-life. But as I learned, despite the technology side of digital government, the title isn’t that common in the public sector.

As someone who’s been a private sector product manager for over a decade of my career, I’ve heard the many complaints about trying to build products in government. “We can’t take risks,” they tell me. “And just try to ship an MVP; it’ll be bloated by committee before you ship anything at all.”

But it turns out that for every facet of public sector product management that’s hard, there’s a silver lining. For example, you can share openly, and innovate in public—unlike for-profit firms that need to keep intellectual property under wraps.

To find out more about empowering public sector product managers, I talked with three amazing digital government leaders:

  • Kathy Pham, a computer scientist who worked in the private sector before becoming a founding product and engineering manager for the US Digital Service. She teaches ethical product management at Harvard and runs the responsible computer science program at Mozilla.
  • Katherine Benjamin, who recently joined the City of New York as the deputy CTO for Digital Services in the Mayor’s Office of the Chief Technology Officer. She’s worked in Ontario’s digital service and the NHS in the UK. Her frontline work with hard-to reach populations is where she first became interested in how technology can be a way to include people.
  • Ayushi Roy, who leads the state and local practice at 18F, and works as a product manager on projects such as Medicaid and the Children’s Health Insurance programs. Before her Federal work, she built out the innovation team in Oakland, California after working on support for crime victims and mental health crises.

All three build technology products, and have fascinating perspectives on product delivery.


Learning what’s folklore

As Kathy said, it’s not clear what’s possible—and you have to discern “what really is not legal,” versus things people think can’t be done—what the group refers to as ‘folklore.’

Kathy shared the example, told by Erie Meyer, of people who were told they couldn’t survey students because of the US Paperwork Reduction Act. While this was untrue, it was folklore, which meant user research wasn’t getting done. “It’s not out of any kind of malice,” said Kathy. “It’s maybe that over time teams have passed along this idea that it’s not legal for a group to go talk to a group of students to understand education, for example. And someone might say, ‘well it’s because of something like the Paperwork Reduction Act.’ And they believe it’s true.”

So a public sector product manager has to bring others to this conclusion. Kathy explained that the PM has to tell others, “I can understand why you might interpret that it’s illegal to go talk to a group of students to understand what it’s like to apply to colleges, but really you can, and this is why it’s okay.” But as she pointed out, “you can pluck any product manager out of the private sector and they probably haven’t had to do that before. That, especially, is a big part that’s very, very different.”

Ayushi agreed, sharing an example of her own:

“When I started as the state and local Director at 18F, I was hearing a lot of different kinds of folklore about how 18F can’t work with state and local governments. They were like, “hi, welcome to your new position. And by the way, you can’t do what you’re being asked to do.”

Ayushi eventually found the Intergovernmental Cooperation Agreement (IGCA) that governs how 18F is allowed to work with other agencies at the federal state and local level as long as it has Federal funding, the support of the mayor or government, and a few other criteria. There’s so much folklore that make doors seem closed. Her answer is as simple as it is bold: “Retry all the doors.”

In her time in Ontario, Katherine also encountered the “urban legend of ‘you can’t speak to users’.” As she explained, “In Canada, there’s a certain period before the election when you can’t go out and engage with users. So we had a working hypothesis that the reason people were so frightened to talk to users is that they remember every election getting these notifications saying, “don’t talk to people!”


Low-hanging fruit

When I prodded our guests about many governments’ inability to take advantage of the latest and greatest technology, it was clear that there’s a flip side: Plenty of low-hanging fruit. Kathy pointed out that often, because the technology is modern, you can bring in proven approaches that have been thoroughly tested in other sectors. “We don’t have to go innovate for some like magical new thing,” said Kathy. “We have already known how to do that over here [in another industry] for a few years.”

We returned to this topic when I brought up some of the recent advances in AI later in the conversation. I think that’s what the kids call Foreshadowing. 


Capital-V MVP

In Product Management, much is made of the Minimum Viable Product (MVP)—the least complex thing you can deliver that proves the idea or removes the largest initial risk. As someone who’s written a lot about Lean Startup, where the MVP is Gospel, I believe firmly that “if you aren’t ashamed of your first release, you’ve waited too long.” I’ve even written about it in the past.

But that isn’t the case in government. One reason for this, as Katherine pointed out, is that “the reality of government is that, even with the best of intentions, you often don’t have time to circle back because something else becomes a priority. Katherine observes that this is where much digital inequality begins. “It’s certainly not intentional, but that’s ultimately the result.”

Her solution is to focus on people who have more amplified needs. “If you think about a distribution curve—on the two ends of that distribution curve, you’ll have very amplified needs,” she said. “If you can get it right for the tail ends, then you’re probably going to hit those people in the middle by default. So kind of just flipping that instead of, “let’s build it for the mainstream and then we’ll think about accessibility.”


Needs versus luxury

Ayushi goes beyond just designing for the outliers. One of the big differences between private- and public-sector product management is that when you’re in the public sector you’re “designing for a constituent user that is probably coming from a place of need, rather than luxury, right?” she said. “No-one goes to the medicaid.gov site because they have free time, or because they’re excited to do so. They probably have a horrible reason that they needed to get support.” As a result, what it means to be a product person focused on a constituent user is very different from what it means to be one focused on a consumer user.


Public sector product managers need their own job description

The role of product manager is one of the broadest, most influential jobs in an organization. But I can confirm from personal experience that it’s also a job with very little authority. You’re alternately an influencer, an advocate, and a curmudgeon, trying to get time and attention on your product rather than the myriad other things people could be working on.

In the tech world, this often means a background in business and computer science. With the resulting lack of diversity in hiring. My guests looked much farther afield in staffing product roles. At the USDS, everyone was called a Digital Service Expert; they only referred to themselves as product manager internally. And sprints were called “thin slices.”

“The field itself has to exist in the public sector,” said Kathy. “The field, as it exists in the private sector has to change. And if anything, PMs going to the public sector can help inform the product managers in the private sector, because they now have a much deeper sense of responsibility.” Ayushi agreed that “It has been happening, but it happens perhaps in distributed fashions, where there isn’t a single product owner, or a single product manager.”

Katherine said that in the private sector, “you’re having to start from scratch and articulate what a product manager is.” She’s seen private sector product managers struggle to make the transition. “If they’re coming in from the private sector and they’re used to being recognized as very senior with executive decision making skills. They’re still senior with executive decision making skills in government, but because of the hierarchy of government, people might not, first of all, immediately recognize that. And if you’re the type of person who responds to that in a negative way and becomes defensive, it doesn’t tend to go so well.”

Despite the challenges of the shift, Katherine doesn’t think recruiting is a problem. “I think digital government sells itself,” she offered. “There’s a lot of people who really want to join this, if you can just tell them what it is that they’re going to do.”

Kathy thinks that optimism may wane. “Candidly, over time, the recruitment becomes a lot harder,” she said. “At least for USDS, people who have the luxury of picking up and moving to Washington, DC, are just very different, right? So it makes it much harder to recruit for a very diverse group of people. It’s very biased towards a certain type of people. And it’s something that I have to think about constantly: ‘how do we change?’”


Ethics is complex, hard, and essential

We talked about the ethics of user research, from the risk of exploitation to the need to incorporate it throughout the design and delivery process. Kathy pointed out that this isn’t a new problem—but digital product managers have to catch up. In both private and public sector product management, “there’s a reckoning of tech and its harms,” she said. “There are so many fields, including user experience, research and science technology studies, and anthropology … that have been thinking about which communities are missing, which communities to include for a long time.”

Ayushi pointed out that the term “ethics” is an incredibly broad umbrella, and needs to be incorporated into every phase of civic engagement. What I took away from this is that ethical public sector product management is the work of ensuring that the things we build align with the values we hold true as a society—whether a bill of human rights, a constitution, or a moral code.


Working in the open is a superpower

Since we’re talking about empowerment, one of the superpowers that really came across in our conversation was collaboration. “One of the biggest pluses of the digital government movement,” said Katherine, “is just picking up the phone and chatting with someone. People will just show their homework.”

She remembered calling a colleague who had moved on, in order to learn from her mistakes. “I remember it was this great call in 2017 where she’s like, ‘what are you trying to do?’” Her friend replied without hesitation: “‘Here’s what I’m telling you. Bam. Bam, bam, bam. Whatever you do, don’t do this. Don’t do this. Don’t do this. The whole can of worms. Go in this direction.’ And that just saves you a month of work, so I think that that level of collaboration is really helpful.”


We covered a lot!

There’s plenty more in this amazing conversation. I sat back for most of it and let them talk—they’re already planning virtual coffee as a result. We quoted Foucault, the democratization of knowledge, the wide variance in technology budgets, and even the recent announcement of the GPT-3 model in AI (spoiler alert: We have bigger things to tackle.)

I’m loving the easy flow of these FWDThinking discussions, and for me, this one was a particularly good chance to think hard about luxuries versus needs, the profitable middle and the necessary edges, and how we’re redefining public sector roles in a digital world.

All opinions expressed in in these episodes are personal and do not reflect the opinions of the organizations for which our guests work.

[00:00:00] Alistair Croll: [00:00:00] Hi! And welcome to the second edition of FWDThinking, which is a production of FWD50, the digital government conference, and is done in partnership with the Canada School of Public Service’s Digital Academy. Our hashtag is #FWDthinking, that’s F-W-D-Thinking, and we are going to get into a discussion with some absolutely brilliant people who spend their lives thinking about how to build products and services that work for everyone- really keeping in line with FWD50’s mandate of using technology to make society better for all. 

The theme today is about empowering product managers but as you’re going to see, we really talk about what’s different between private and public sector, because often people bemoan the restrictions and strictures that are placed on public servants because of regulation and governance. But it turns out that public service product management has a whole bunch of advantages too. 

So, I hope you enjoy the [00:01:00] fascinating conversation that we’re about to have. Let’s dig into what it looks like to empower product managers in a digital age within government. 

We have some amazing guests joining us. So I’d like to welcome Kathy Pham, who works at Mozilla, making the internet better for everyone, Katherine Benjamin, who is the deputy CTO for New York, and Ayushi Roy who works at 18F coordinating with regional and local and state governments on product delivery. 

Hi Kathy!

Kathy Pham: [00:01:31] Hi! 

Alistair Croll: [00:01:32] Hi, Katherine. And hi Ayushi. How are you? 

Ayushi Roy: [00:01:38] Hello. Thanks for having us. 

Alistair Croll: [00:01:40] Thank you for being here. I know all of you are busy saving the world, and we’re really glad that you were able to take some time off to talk to us about what that’s like. 

Why don’t we start with you, Kathy? Why don’t you tell us a little bit about who you are and what you do. 

Kathy Pham: [00:01:52] Hi, I’m Kathy. I’m a computer scientist and product person who had spent about a decade [00:02:00] in the private sector in big tech companies, before coming over to the United States Government at the US Digital Service and was a founding product and engineering member there building out some of our teams and products over at USDS for about three and a half, almost four years or so. 

And then from there, I came over to Harvard to teach a class on product management and start the ethical tech group here at the Berkman Klein Center, as well as, run the responsible computer science program at Mozilla, and recently just launched something called the Mozilla Builders Fix-the-Internet Incubator, where we fund and support teams, product teams and startups, to fix the internet across a range of topics from collaboration  society to surveillance capitalism, artificial intelligence, search and beyond. So it has been really rewarding to tie together the academic side with the practical side, with [00:03:00] the ethics and responsibility side, and some of the government and policy sides as well.

And I’m really excited to be here. 

Alistair Croll: [00:03:07] Awesome. And how about Katherine?

Katherine Benjamin: [00:03:10] Hey everybody! Great to be here and joining such an illustrious group of experts. 

I recently joined the City of New York as the deputy CTO for Digital Services in the Mayor’s Office of the Chief Technology Officer. Before that I was with the Ontario Digital Service where I launched the Rapid Prototyping lab out of Kitchener, Waterloo. And I also worked in the UK for NHS England, as part of their Digital Services team. So my background actually started more in frontline work, usually in the context of healthcare, and it was through doing frontline work with harder to reach populations, where I became really interested in how technology can be a way to include people, particularly harder to reach communities.

So very excited to be here and chat with you all today. 

Alistair Croll: [00:03:54] Awesome. And finally, Ayushi. 

Ayushi Roy: [00:03:58] Hi all. Thank you so [00:04:00] much for having me here today. Really excited to speak to the both of you. 

I am currently the lead of the state and local practice at 18F. I’m also a product manager there, currently working on a project with Medicaid and the Children’s Health Insurance programs serving 9.6 million low income children across the country here in the US and we’ve been building out a lot of really incredible things that I’m excited to share here. I, before 18F, helped to build out the innovation team in Oakland, California. And how to, I guess in some ways a similar start to you Katherine, working on crime victims and medical treatment support for mental health and other crises- what’d it look like to build out a hotline there. 

So I’ve been doing this work for awhile. Really excited to share in all our product management glory with both of you. Thanks for having me. 

Alistair Croll: [00:04:53] Yeah, it’s great to have you all here. And I think, I’m a product manager by trade too, so we can all commiserate over the challenges of [00:05:00] building products at scale.

I got tons of things I want to get to in conversations. Obviously, I’m sure all of you do. But first up I wanted to ask, especially since both Kathy and Ayushi come from the private sector as well, and I’ve done some work in, you know, on that side of things: what’s the biggest change that you faced in building products when you went from the private sector into the public sector?

Kathy Pham: [00:05:22] So many differences. I remember when we first built out the hiring pipeline for USDS and thinking through, you know, what competencies and skill sets we wanted for a PM. And so many of us, because of our background, the private sector, we had a set list of what we wanted. And I think what many of us learned as you add a whole level of understanding of complicated bureaucratic systems and policies that every product manager has to know. When you’re told- and Erie Meyer has a whole talk she does on this- when you’re told that something is illegal, you have to go and figure out what’s legal and [00:06:00] what really is not legal, or what’s like folklore. And that’s a big part of your job. In the public sector, and not that anyone is particularly malicious, it’s just there are things that folks aren’t sure of. 

And that’s something for the most part, especially if you are in a big technology company, it’s almost the opposite. Where you are like: “You want to build something? Go, go, go, go fast. And like put it out in beta, put it out in alpha. Just get someone to use it.” And that’s a big part of it. Understanding bureaucracy and complicated systems and all the people. And I think having a deep respect for why the bureaucracy exists, but also after you have that respect to go and break it. And I think that’s… 

Alistair Croll: [00:06:38] Can you expand on whatyou mean by folklore? Cause that’s a really interesting term.

Kathy Pham: [00:06:41] Yeah. It’s, again it’s not out of any kind of malice, it’s maybe that over time teams have passed along this idea that it’s not legal for a group to go talk to a group of students to understand education, for example. And someone might say, well it’s because of something like [00:07:00] the Paperwork Reduction Act- and again Erie Meyer, one of the co-founders of USDS has a great talk on this. And they believe it’s true. 

But when you really, you know, bring in lawyers, other policy folks, and really dive into the actual laws you’re like: “Oh no, it’s not true. That’s not true.” And part of the job is now to bring everyone else along and say, “Well, this is the role and this is how I can understand why you might interpret that it’s illegal to go talk to a group of students to understand how it’s like to apply to colleges, but really it’s you can and this is why it’s okay.” And that’s it. Pretty much you can pluck any private sector product manager and they probably haven’t had to do that before. And I think that especially is a big part that’s very, very different. Yeah. 

And when I say folklore, that’s what I mean, something that’s been kind of passed around as truth. 

Ayushi Roy: [00:07:50] I can share an example of folklore too  Kathy, that I feel just might get some laughs in this crowd.

[00:08:00] We, when I started as the state and local Director at 18F, I was hearing a lot of different kinds of folklore about how 18F can’t work with state and local governments. That was the starting point. They were like: “Hi, welcome to your new position. And by the way, you can’t do what you’re being asked to do. ” And I was like: “Hmm, this is interesting. Let’s look into this further.” 

And found out that the Intergovernmental Cooperation Agreement or the IGCA, which governs how 18F is allowed to work with other agencies at the federal state and local level. It does allow for us to work with state and local governments, but they have four really significant clauses that guide how we’re allowed to work with state local. It has to have some federal funding. It has to have an executive sponsor, like the mayor or governor’s signature. It has to have, you know, so on and so forth. And people, it has to be in procurement, it can’t just be a billed project without some sort of acquisitions aspect or component. 

And all of these different clauses [00:09:00] combined people were like: “Can’t do. Just, just can’t do. It’s not what we’re going. We can’t do state and local.” 

And I think there’s so many pieces of folklore like that, which now effect how we see doors to be closed. And as a result for history, for generation or generations- we’ve only been around for like what, five years- for years, we will then use to retry all doors. Right? 

Kathy Pham: [00:09:25] Yeah. 

Ayushi Roy: [00:09:25] We assume that these doors are actually closed. And I think there is a really fine balance like you put it Kathy, about respecting the reason why the bureaucracy is there. I’ve definitely come to respect and actually really appreciate a lot of how the government chooses to derisk or avoid risk. And at the same time, then having the courage to open those seemingly closed doors.  

Alistair Croll: [00:09:46] It does seem like there’s a real paradox here though. If you’re, you know, there’s an old saying that you can’t fix the problems of today with the technology of today. You have to sort of find, or maybe that’s the problems of tomorrow, but you have to go and find a new [00:10:00] possible solution.

And so constitutions and rules and laws are living documents. And many of the rules that were created, you know, in the era of fax machines no longer apply in the era of FaceTime. And I think one of the things that we’re seeing with the current COVID pandemic is it’s proving to people. I mentioned on the last episode, I have doctor friends who would never have done remote patient consultations.

And now they’re like, I don’t want to go back to the office because they’re able to get a specialist and the patient on the phone at the same time, have better quality for everybody involved, there’s less risk of: “Am I sharing information I’m allowed to share?” Because the patient said it on the zoom call and they’re like: “Everybody’s happier and it’s cheaper.” Right?

 So I think there are examples like this, where, how do you question the folklore because your job is to innovate, not just on products, but on what’s possible. How do you question the folklore so you can develop better products without becoming that person that sort of people discount cause they’re always, you know, trying to change the way things are and they should stop meddling and all those kinds of [00:11:00] anecdotes. 

Kathy Pham: [00:11:00] I want to add to when you mentioned you can’t fix the problems of today with the technologies of today. I actually think in government you can just cause sometimes the tech is a bit farther back and you’re like: “Oh, there’s all this tech today that I can bring in. And we don’t have to go innovate for some like magical new thing. We want this very secure, strong infrastructure that we actually know how to do over here already for a few years. And we’re a little bit farther behind, and let’s go bring that in.” 

And I think to the folklore piece, I think that, you know, product managers in the private sector had the luxury to say- and I’m doing this now with some of the startups I advise: “We’re going to do an MVP. We only care about these type of users right now, and we’ll worry about the other ones later.” That’s a luxury you have when you’re a little startup or a company. 

And to some extent in government, you might maybe want to think about that a little bit too, so you can get something out the door, but for the most part you can’t right? How are you going to pick which people [00:12:00] in your country need healthcare and which veterans deserve access? And there’s like this deeper responsibility inside a public sector setting where you do want to go a little bit faster, but at the same time, balance that with this level of deep responsibility we have to every single person that we have to serve. And we just can’t, you know, decide to like: “I’ll worry about this other group a year later.”

Alistair Croll: [00:12:20] That’s the MVP with a capital V, right? The V is much bigger. 

Katherine, do you find that’s true in New York? 

Katherine Benjamin: [00:12:27] Well I think just across the board, so to the earlier point around, you know, how do you start to bring in some of these different ways of working? I mean, I think one of the best things you do when you come in to a digital government team is find your like person who’s been there for a really long time and approach them with humility and curiosity- not them, but the work. So where is this urban legend of ‘you can’t speak to users’ coming from- I’m so glad that everyone was mentioning that cause that was also in the back of my mind.

I remember in Ontario, there was a similar one and we [00:13:00] hypothesized that what people were thinking was they remember getting those alerts before elections where, in Canada there’s a certain period before the election when you can’t go out and engage with users. So we had a working hypothesis that the reason people were so frightened to talk to users is they remember every election getting these notifications like: “Don’t talk to people!” And so they’re like: “Okay, I won’t talk to people.” And then when you go to try to talk to people, they’re like: “Never!” So we weren’t able, we didn’t have to go in with lawyers, but that was sort of a hypothesis we had. 

But I think to the point of, you know, I absolutely agree with Kathy’s observation that it’s true in government. It is. You just really don’t want to go in with a target user. There often isn’t a target user and it’s very difficult to balance what would work in the private sector and has been proven as an effective model, with how you want to include people in government context. Particularly because if someone could guarantee to me that we were going to circle back and add additional users later, I might be more comfortable with it. But the reality of government is even with the best of intentions, even with the best of intentions, you often don’t [00:14:00] have time to circle back because something else becomes a priority. And actually, I think that’s where a lot of inequality comes in. And it’s certainly not intentional, but that’s ultimately the result.

The flip side of that is that there is an opportunity, those of us working more on the design side is, you know, there is this idea of focusing on people who have more amplified needs. So if you think about a distribution curve, on the two ends of that distribution curve, you’ll have very amplified needs, like vary a lot and vary a little, on any given variable. And often you can kind of just focus on those tail ends and not so much on the mainstream. Cause if you can get it right for the tail ends, then you’re probably going to hit those people in the middle by default. So kind of just flipping that instead of being like: “Let’s build it for the mainstream and then we’ll think about accessibility.”

The other way you could do it is: “Let’s build it for accessibility and then we’ll check that it works for everyone.” Then it’s the whole skateboard ramp analogy or wheelchair ramp: you build a wheelchair ramp and it’s not just for somebody to take their wheelchair. It’s going to be for people who have strollers and it’s going to be for people who skateboard and everyone benefits from the infrastructure investment of making [00:15:00] that ramp.

Alistair Croll: [00:15:00] Yeah. That’s a great analogy. 

Ayushi Roy: [00:15:02] I’m here for the curb cuts! 

This is something that actually, this is perhaps a shameless plug, but one things I do in my own personal time beyond 18F life is run this podcast about design governance and, the first actual season is literally all about designing for the margins, and what it means to start by designing for those margins. And inevitably that kind of covers the middle of the distribution curve. And there’s so much good literature around this, but I actually think it goes even further than just designing for the margins. I think it even goes into the way that we see product management, the way that we teach and train in product management. 

You know, kind of coming back to your initial question Alistair, about the difference between private and public sector product management, one of the things that I think is also not [00:16:00] really, carried over or the differences, right? Is that in one situation, I think you’re designing for a user, a consumer user, and then in the other you’re designing for a constituent user that is probably coming from a place of need rather than luxury, right? Like no one goes to the medicaid.gov site because they have free time or because they’re excited to do so. They probably have a horrible reason that they needed to get support or they’re, you know, we stigmatize poverty in this country. And as a result force, a lot of people to, you know, look to us for those support mechanisms and services. And as a result, what it means to then be a product person, or a product designer who is then focused on a constituent user rather than a consumer user, means very different things, right? 

Alistair Croll: [00:16:52] Yeah, I love that idea of need versus luxury cuz that really, you know, you’re not looking at a profit margin. You’re looking at where you can do the most [00:17:00] good and you can do the most good by helping the person who is most vulnerable cause there’s the biggest Delta there, right? So if your scoreboard is, how do you do that? And when you said that about accessibility, it made me think, I was watching a video the other day of a robot delivery unit that couldn’t get up a sidewalk. So, you know, that ramp might’ve been designed for accessibility, but it turns out all our robot delivery tools need ramps too.

And there was another, we had a person at FWD50 the first year who’s blind and we were talking, he said: “Look, this is the best and the worst of times.” Because on the one hand, you know, you’ve got apps on a phone that may not be accessible and so you can’t access this. But at the same time, people are now designing, not just for the blind, but for anybody who’s using Alexa or Siri, which needs to be able to crawl websites with properly structured, you know, JSON codafide code.

So it turns out that the accessibility is also really good for automation, because then you can very easily build something that can scrape the structured information that’s often needed to provide an assistant [00:18:00] reader the data it needs and stuff like that. So it does seem like a really interesting angle.

When you talk about bringing in teams and finding the right talent, how do you go about recruiting product managers? I mean, first of all, you got to convince them that they should follow the public service and do the right thing. But how do you look for someone who’s going to be able to make that transition from move fast and break things and all the damage that has caused, to move carefully and make good decisions?

Kathy Pham: [00:18:30] Yeah. Oh, man, this is such a big topic. I have so many thoughts on this. One, I think that the way, at least more traditional consumer technology companies, have hired for product managers are part of the problem. And that’s the world I came in, right? If you look at many product manager, product management job descriptions, at least in private sector, it’s a few years of computer science or engineering experience, some years of product management or something like that, and then a couple of [00:19:00] other things. Or in like, if you look for middle level product managers, it’s a few years of technical experience or equivalent, some years of product management, but then you don’t have a particular path for product managers because there’s no real career in that.

And so ultimately you have a bunch of people with a very similar background and it tends to be incredibly white and incredibly male in this whole field. And so when we are looking to hire them into government, you’re already from a pool that is quite restricted, with skill sets that might be relevant for consumer technology, but maybe a little bit different for government as well.

So I think it can get quite complicated. I think what that’s looked like for groups like USDS for example, is that we got just more creative with how we hire product managers. We might look for technical acumen, but that doesn’t mean it’s a computer science or engineering degree. And so we get very specific with what we’re looking for when we say: “What do we really mean by technical skill?” And technical isn’t binary, right? And so you don’t need an engineering degree to be [00:20:00] able to very effectively talk about cloud services or understand API. And so I think we just turned around a bunch of these job descriptions to be very specific on what we think we really, really need for the skill sets rather than these like high level  like another kind of folklore, which is like, good PM’s must have an engineering background and like an MBA, or a few years of experience in a big tech company. 

So we’ve changed how we do product. I think we have to change how we do job descriptions for product managers. And make them exist in the public sector! The whole concept of having a product manager in the public sector.

At USDS we called everyone a Digital Service Expert. PM roles were made up, we called ourselves that on the ground. And Ayushi you probably know that there are PM roles that have now existed across the federal government, but we named them something else to make it pass the hiring process.

So it’s complicated. The field itself has to exist in the public sector. The field, as it exists in the private sector [00:21:00] has to change. And if anything, PM’s going to the public sector can help inform the product managers in the private sector, because they now have a much deeper sense of responsibility.

Ayushi Roy: [00:21:11] Yeah, I’m so curious to see what the PM, very big quotes here, you know, ended up being defined as in the USDS space and also in the New York City context, Katherine. 

I mean at 18F one of the things that we laugh about within the product chapter is that the, even the language around product development is just, is directly copied and pasted from the private sector.

There’s so many other ways to do it, that we don’t. We don’t need to maintain even something simple as like the language of sprints. You don’t need to call it a sprint. You could call it the thin slice. That’s something that we joke about. You know, like we were trying to explain to some of our customers or partners here on the Medicaid team, but also in the department of justice team that 18F works with,  you know, the concept of like starting from the smallest,  most critical user need and [00:22:00] then expanding the pie until you have the full pie complete, right? 

And so we started talking about it from the standpoint of slices, like: “What’s your first slice of pie gonna look like? What’s your second slice of pie? You know? 

And I remember we had our last private contractor come in and they were like: “What is this language around slices? Like, why can’t you just, aren’t you doing modern dev?” Like, yes, we just don’t need to you, you know, like there’s ways to meet our government partners half way. 

Because to your point, I think Kathy what you’re getting to is like, the product work, it’s not that it hasn’t been happening in government. It has been happening, but it happens perhaps in distributed fashions, where there isn’t a single product owner, right? Or a single product manager. It’s perhaps been happening in ways that kind sometimes coalesce with project management. It happens in different ways, but it has happened. 

And so it’s important, I think it’s been really important. And I love this about this [00:23:00] work, is that we really try to amplify the work that’s already happening on the ground. And these different partner agencies, who are veterans of federal service or state and local government service, and understand product build out, but don’t really have the same kinds of terminology or it might be more used to waterfall instead of, you know, agile, agile dev, modern dev. 

But yeah, I’m curious how it shows up in New York or how you saw it show up in USDS. 

Katherine Benjamin: [00:23:27] Well, I think definitely product management in government, unless you’re somewhere that is one of the leading digital jurisdictions, you’re having to start from scratch and articulate what is a product manager- to Kathy’s really excellent point there. 

I’ve seen a lot of product managers struggle. If they’re coming in from private sector and they’re used to being recognized as very senior with executive decision making skills. They’re still senior with executive decision making skills in government, but because of the hierarchy of [00:24:00] government, people might not, first of all, immediately recognize that. And if you’re the type of person who responds to that in a negative way and becomes defensive, it doesn’t tend to go so well. So I’ve seen that play out. 

Fortunately, in New York we don’t have a ton of people who have the formal job title, Product Managers. We don’t have to articulate it quite so frequently. So it can be very intentional when we explain here’s the product we’re working on, and this is the person who’s going to be acting as product manager. So there is a lot of level setting around expectations, and then bringing in more traditional things to help people just frame what this job is. 

One example I talk about a lot is like, I think a lot of folks think product management is project management and they often think that’s something really junior. And so they sometimes are dismissive of product managers who are actually doing a really excellent job trying to corral stakeholders, trying to define the scope of  a product. So that can be really, challenging for incoming people. 

But I know one of the things that we’ve sort of been asked to discuss here is, you know, then how do you bring in these strong product [00:25:00] managers? And I think I’ve actually never had a really hard time recruiting into digital government. I think digital government sells itself. Once people know what it is that you’re talking about and they know the mission and they see the potential for change, I mean generally people are trying to bang down your door. 

I think the hardest part is the literal trying to hire someone, like what it means to hire in government. It’s always hard. It’s always a bit slow. 

And then, you know, if you can just get a job description up online that doesn’t look like a more traditional government job description, which is very, very long and extremely complex with tons of bullet points. And instead have a job description that’s much more user centered. It describes what the work is. It describes the potential for impact you have through joining that team. The jobs kind of sell themselves. 

I don’t know if that’s been your experience, Kathy, when you were more on government project or Ayushi when you’re working with 18F but, yeah, I found that oddly, there’s a lot of people who really want to join this, if you can just tell them what it is that they’re going to do. 

Kathy Pham: [00:25:58] I think, yeah. Two [00:26:00] things. On the part of the product management role itself, there’s like this interesting medium of on one hand, it’s good to have some people in the room who just have like core product management training. Cause then you can take that and you’re like, alright, this is what works in government. This is what doesn’t work in government. But you at least have some foundation versus like a free for all where you’re like, there’s no foundation. So to some level, we’d need some of those people. 

Then we need other people to come in who understand government and then like poke holes in ways where you’re like: “Oh, you’ve been like a PM in the tech sector for 10 years. Great! These skillsets are awesome. You should totally bring how you do user experience testing, but also managing stakeholders here is very, very different.” And work around that. So I think having a core team to help figure that out is incredibly helpful. 

And as far as hiring, I think in the early days, there’s a lot of momentum and excitement around you’re starting up something new and everyone wants to join and it’s super exciting. And people are willing to put aside all [00:27:00] sorts of things ranging from where they get to live, or they want to live in a certain area because that’s where all the tech companies are, or like family situations, and a number of other things to come to wherever the government needs them. But candidly over time, the recruitment becomes a lot harder. At least for USDS, people who have the luxury of picking up and moving to Washington, DC, are just very different, right? So it makes it much harder to recruit for a very diverse group of people. It’s very biased towards a certain type of people. And it’s something that I have to think about constantly of how do we change? 

Like the government pays. USDS was able to pay quite well at one of the highest levels in government. But unfortunately for some of the engineers and designers, it’s not the same as some of the tech companies. So, it made so that there might be people who are deeply bought into the mission, but for family reasons, for you know, maybe they’re first generation immigrants and they’re like taking care of their entire family with their tech salaries, they can’t just pick up and [00:28:00] go to a government. And, yeah, that just kind of affects the type of people we can recruit. 

Alistair Croll: [00:28:07] So I have a quick question.

Something we did at FWD50 last year was what we call a time travel workshop. So we start out by doing back casting, which is where you say: “Imagine it’s thirty years from now and we fixed climate change. What did we do?” And people break up into groups and they each write a story about how they got there, right? Some kind of fictional utopian story about solving a problem. Usually it’s a positive outcome. And then with that story, everybody gets up and tells the stories. And now you build sort of a timeline of how we get there. And then you do the second part, which is a premortem, which is where, like you’re doing a postmortem beforehand of like: “Okay, this happened, why didn’t it work? What went wrong?” And then you have this really interesting narrative where you’re going from 30 years forward and the desire to come back and then simultaneously looking at the obstacles. And it’s a really good way to kick off a new product, a new [00:29:00] idea. 

So we thought it was a great model and it’s a useful thing to share with people who are listening. But the challenge of course is, who’s there to tell the stories? Because I think when we have to ask ourselves who isn’t present, who isn’t there to say: “This is what goes wrong.” Who isn’t in the room to say: “Hey, you boneheads, that future you have outlined is not realistic for someone of my race, creed, color, socioeconomic background, varied level of ability, you know, neuro typicality, whatever that is.”

How do we make sure if you have this sort of product management mindset of we’ve got to deliver it, actions speak louder than words, let’s get something out there- how do we make sure that we’re conscious of who isn’t present and include them in building that initial narrative so we have a product roadmap from that? So how do you think we should bring those people in?

Katherine Benjamin: [00:29:51] Well, in some ways I think of it similar to a user research ethics. So anyone, I think everyone here has done research [00:30:00] ethics training, and if anyone hasn’t done it, you know, part of it is that at any given point, when you’re working with human subjects or excuse me, any sort of subjects. I know it sounds absurd to say including like plants, I do have a friend who’s literally doing this with trees .

Alistair Croll: [00:30:13] Algorithm is soon, right? Pretty soon it’ll be AIs, you know? 

Katherine Benjamin: [00:30:18] Sure! But we need to be thinking it at every stage and like constantly checking ourselves. Like the worst mistake you can make, and I tell this to students all the time, is thinking like: “Oh cool. I did this tick box of research ethics clearance. Now I’m good to go. I’m not going to hurt anyone.” It actually needs to be something that you keep in your mind all of the time. And I think to a certain extent when it comes to how are we building our product teams, who is present, who isn’t present, that needs to be at literally every single step. And you need to be constantly looking around the room, constantly asking who isn’t there. 

And then it’s more than just looking with your eyes and saying who isn’t there. It’s also looking at your methodology. So for instance, if you’re doing user research recruitment, are you doing something like convenience sampling? [00:31:00] Which totally I get, in a jam, just speak to 10 people you know about renewing car tax online. But who are those 10 people? And if they look just like you, what sort of answers are you going to get?

And then similarly, when you’re getting tons of data coming in, the teams and whether it’s your product manager, or probably wouldn’t be, would probably be more your user researcher, looking at that data that you’ve got in from, you know, all of your different sources and prioritizing what becomes the user story? What becomes the top of the backlog? What becomes the Epic that we’re going to tackle first? All of that comes out of who are you speaking to, who’s there, who’s not there. And I guess at the broadest level also, what’s the policy intent behind the product that’s being built? 

So that’s not a very specific answer, but I think that’s one of the lenses that increasingly we have to bring into product management.

Kathy Pham: [00:31:46] I love you brought up research ethics just because I think that, especially now, you know, we’re talking now about product management and the, in the public sector, but also in the, in the private sector, there’s like this reckoning of, of tech and its harms. And there’s so many fields, including [00:32:00] user experience, research and science technology studies, and anthropology and all these other fields that have been thinking about which communities are missing, which communities to include for a long time.

And I feel like government has. Two interesting things with hierarchies there’s like the actual hierarchy that exists in government who has power ranging from, you know, president, secretary, at least in the U S down and then same with different levels of government is like real hierarchies. And then tech teams themselves have hierarchies of who’s the most valuable voice in the room, whether or not anyone talks about it, where it’s usually. Engineers usually have the strongest voice and are seen as probably the most important people on tech teams and product managers. and then for pretty much ridiculous reasons, designers and UX people are sometimes seen as a voice that you bring in a little bit later.

And so these ideas around who’s included in research, it doesn’t always make it into the product. So in this theme of like empowering product teams, how do we make that perspective equal? If not [00:33:00] even more important than the actual code itself, that, that we write. 

Ayushi Roy: [00:33:06] I find the conversation around ethics.

Yeah. I find that one word also. So funny. Cause it’s like the biggest umbrella term for so many other conversations. but the one thing that comes up for me that, that. I heard both of you say, but I want to just like kind of explicitly name is the X of engagement. So I find particularly, and I should qualify with this by saying I have a background as an urban planner and urban designer.

So a lot of the way that I see, civic participation, civic engagement, presbytery politics really informs the way that I choose. To build, right? The way that I choose to build digital products. I think one thing I’ve learned, in having both of these hats sometimes worn simultaneously, right? My urban planner hat and my product manager hat. Especially as I’ve done work with the state and local, [00:34:00] agencies with 18, is that. It’s very extended sometimes for product management and product build out to really only involve users from the prioritization phase and on. But it very rarely involves users from the ideation phase. And there’s actually like by most standard, you know, I’m happy to drop further links in the Twitter or the YouTubes or wherever, you know, it ends up going.

But there’s lots of amazing literature around the six, eight basically phases that most academics believe is involved in civic participation or civic engagement, whether it be in like a town hall format or, you know, whatever, more digital formats. And I think what’s interesting is that product work often only involves the last four of this eight phases of engagement. And I, I’m always so curious what it would look like for us to really step into our power actually. And bring users in on the [00:35:00] first four phases as well, spreading with ideation, starting with framework building, because then you really are changing. You’re flipping the whole conversation on its head and you’re allowing users to be not just users in a transactive nature, which I, I personally find user testing sometimes be very transactive, and actually involve users from the very get, go around, being co-players, right? Like they are coal builders of this project with us. They aren’t just providing often unpaid feedback to us, which, and I bring that up because I think that’s particularly important in the government context. Right? When you aren’t a luxury users or consumers, when they are constituent users coming from a place of need.

Right. They are going to tend to be at least in the U S context, more of color, more low income, more marginalized, variety of ways. And for us to then take up their time and not compensate them for the work that they are putting in feels deeply appropriate. Yeah. Right. And so, what does it mean to again, think about the ethics of engagement, where we [00:36:00] involve folks from an earlier place.

And that is our responsibility as product people as product teams, right. To do so. 

Kathy Pham: [00:36:08] Ayushi, how do you, you made me think of two things, one, How do you think about, especially in the government context, people who are not necessarily how tech defines as the users, but people who are deeply impacted by the product regardless, and they never get put into the bucket of this happens a lot with different government products, right? You’re like, they’re not users, but they’re deeply impacted by whatever it is that the government puts out, whether it’s a policy or a piece of tech, as well. And then just a comment I have, I agree with you, I’ve been, I’ve always been so floored by how groups will treat user experience research where at some group will be seen as like you’re lucky to get to be in the room with us as we show you your product and for your free time while paying other people, hundreds of thousands of dollars to be consultants on the product. And you’re like, But these marginalized communities are the ones that are getting you way more valuable feedback that any of these other like fancy [00:37:00] consulting companies that are doing this work for you.

And, but the disparity is so stark. 

Ayushi Roy: [00:37:06] Yes, yes. Your second point. So strongly the, the gap is appalling is endlessly appalling and mind blowing to me. as far as, you know, a response to your first question, and I know this is, this could be a much longer conversation, so I don’t want to, you know, have the space here.

But one thing that I have started, doing years, I mean, this is pre 18F is kind of just mapping out the source blueprint. Right? So, there’s so many different kinds of designers, right? There are the original designers, there’s UX designers and researchers, but then there’s also service designers.

And I think services and work is so, so valuable to have upfront and like the discovery stage of product built out, for his exact reason where you just mentioned Kathy, which is how do you actually identify the people that are impacted [00:38:00] regardless of whether they’re actually users or not. And I think in the public space that is so valuable because, there are more than just users that are going to be impacted, especially when a lot of services and digital products or, or newly digital products are actually, sitting, I’m a very visual thinker, so I want to almost draw this out, but we’re kind of sitting at the crossroads  of multiple agencies, right? Or multiple, or this, or let’s put it this way. This same group of users might need, same group of constituents might need, might be users from multiple types of products. Right? So, let me give a simple example of something that I’m actually working on right now, which is, children’s health insurance program is the largest addition to Medicaid and Medicaid was started in the sixties and it’s currently serving about 10 million children across the U S who are coming from low income families. And these are often children whose parents are unemployed. So, already here, you’re looking at the dependence of people that are most [00:39:00] likely under the current COVID situation, applying for unemployment insurance from their local, state or agency. Right? But what’s wild, to me, what’s wild to me as an outsider, right? Is that, or as an observer, is that the unemployment insurance services that are provided the department of labor in no way overlap or, by wraparound services with the centers for Medicaid and Medicare service delivery arms that similarly have federally funded, and state administered programs for the children of those exact people, applying for unemployment insurance. And so, you know, you have clearly here these people like the parents of the people applying for chip. Are impacted, but not users. Right. And what does it mean for us to then build out this product?

That’s just one example. I know their are a bagillion examples, but yeah, it’s, it’s really important to what I, the way I’ve been able to think about this so far. [00:40:00] And I so encourage, you know, if you guys have other ideas, both Kathy, Katherine, but also any other viewers here, You know, ideas around what it means to really thoughtfully create a service blueprint for a service or product before you actually build it out to ensure that are not only capturing users, which are also capturing internal and external stakeholders.

Typically the administrators of said product or service because they will also tend to be women and women of color in the US context. And then you’re also addressing those that might be tangentially impacted via wraparound services elsewhere in the, in the state. Yeah, those are just some of the groups that we tried to capture.

Katherine Benjamin: [00:40:39] Ayushi, I’m so glad you brought that up. Cause I feel like you just touched on a thread that like links to so many different topics. So like you I’m a visual thinker and then my background’s also service design. And so as you’re talking, what I’m imagining is, at the least under the creative comments, Damien Newman’s squiggle design.

So it looks like if you haven’t seen it, it’s like squiggle, squiggle, mess, mess, mess, mess, mess, mess, mess, and [00:41:00] then increasingly becomes more linear. And it’s like a metaphor for product design cycles. the point being that, one of the things I’ve noticed just about, product design generally, but especially in digital government is there’s a huge amount of pressure for us to not have that squiggly messy phase. That by design it’s unavoidable if you’re actually doing collaborative authentically, engaging participatory process, like to suggest that participatory process is like super easy.

We’ll do it in a week. We’ll be done. The answers will be, I don’t care what you’re talking about. That’s just not the case. And so it was very difficult when you talk about empowered product teams and what does it mean to be an empowered product team? One of the things is having enough air cover to look at that service blueprint.

And what’s going to happen in variably if your team’s outstanding, which I  hope they are is that quickly you’ll you might get accused of doing scope creep because someone will say, we’re not asking you to redo all of that stuff over there. We’re asking you to build a widget and you’re like, right. But I can’t build a widget if we don’t look at all of this, because how do we [00:42:00] know we’re not going to insert name of hideous outcome. And that tension. I don’t have a perfect answer to, but that I think is a specific example of this macro question of what is an empowered product team. I would hope that an empowered product team has permission to ask those questions and that’s where you’ll see product development, product development, bump up against service development, bump up against systems development.

Alistair Croll: [00:42:23] Sorry, I’ve been looking down, I’m taking furious notes. There’s some great points coming up here. I really liked the idea again, the main theme of this, how do you empower product teams? And one of those things is give them budget to do ethical research. Without exploiting free labor.

That’s a great takeaway that I think doesn’t come up very often. Right? And I think that when we look at the, the model of minimum viable products, Audrey Tang from Taiwan told us about the gov g0v model. Where they have, a version of the, the government services @gov.tw, [00:43:00] but then if you replace the overwind gov with a zero, so it’s g0v.tw You get to see the current prototype for the thing that they’re working on. And so I loved that as a way of reconciling this tension of the MVP is you have a working thing and then you have a view showing the, the slice, if you will, that’s about to be added and people can go and try it if they want, but it doesn’t work for them. They just change one letter in the URL and it goes back and it’s sort of a little, like you have to be a little nerdy to know how to edit a URL.

So there’s a little bit of like, You know, guard rails just to try it. But I thought that was really interesting that you may really like what they had on G zero V, but it turns out it wasn’t accessible, so it goes away. And the next time you go there, it’s not there, but that was the nature of what it was.

And it seems to me the other thing we can do to empower product teams is to give them permission to have sandboxes that are sort of, you know, optional and you can play with them and then you do a commit and that’s the service that you’re actually building. How do you [00:44:00] feel about giving product teams the ability to test in a way like that, or what have you been able to do in your work to allow product teams to test quickly? Without the usual metrics that we think about for product management of just like adoption and so on. 

Kathy Pham: [00:44:17] I think one thing that was big in the United States is, the team that Marina Martin led over at the veterans affairs, with a lot of other brilliant, brilliant designers and product managers and engineers. Release things out in, in beta, and put the code up in github and ask for feedback.

And that was just unheard of. so that’s not quite a sandblast. One kind of in many ways it kind of is right. It’s you have something that isn’t fully fleshed out. You tell everyone that it’s not, and you’ve given people permission to put something out there that is not this 10 year contract to make a website.

And you, you solicit feedback and you make that all okay. And all the code is out there for everyone to see. And I think that’s, that’s one example and 18F has so many examples, I, [00:45:00] I think of this right. Of, Putting everything out in the open, you can go and actually, I just really admire. what 18F has done in general, of just how much writing and openness it’s done. So maybe not exactly a sandbox, but closer. 

Alistair Croll: [00:45:12] But that is a really interesting, you know, one of the things you talk about in the private sector is you’re very focused on the luxury and so on. But the private sector can’t usually work in the open because it’s proprietary trade secrets.

They want to extract a profit from arbitraging. The fact that they know something that you don’t, or they have code that you don’t, whereas in the public sector, it feels like that’s an advantage as you get to work in public without any worry about someone stealing your code. No? 

Kathy Pham: [00:45:39] No, no. Well it’s complicated.

Well, on one hand, yes, the veterans affairs did that. On the other hand, there’s a lot of pushback around putting any code out in the open. And if you’re working with a vendor or contractor, they’ll give you all sorts of legal, ease and contract stuff around. This is proprietary software. You’ll ruin our business. There’s no way we’ll tell you like to let you [00:46:00] release this in the open so that I just want to clarify. That was an example of a sandbox type thing, but it’s still quite, it’s not common. And candidly, there might be cases where you don’t want that to. I mean, I, and I am of the mindset that there are reasons why our governments operate in such a way that some things are just not open to everyone.

And we might want to keep those, some things that way, but I don’t know. You should Katherine, if you have other perspectives from, you know, your, your work as well. 

Katherine Benjamin: [00:46:28] We’ll briefly I’ll say so, I agree more in that, like. Broadest sense. So for instance, when we wrote this, the business case for user research, paying participants in Ontario, one of the first things I did was call up a colleague who I’d worked with in the UK, who was then in Australia, Lisa Reichel who then went to Atlassian, and one of the great, I’d say like probably one of the biggest pluses of the digital government movement. Is just picking up the phone and chatting with someone, and people will just show their homework. So, it was just, I remember it was this great call in like 2017 maybe where she’s like, what are you trying to do?

Oh [00:47:00] yeah. Here’s what I’m telling you. Bam. Bam, bam, bam. Whatever you do, don’t do this. Don’t do this. Don’t do this. The whole can of worms go this direction. And that just saves you like a month of work. so I think that that level of collaboration is really helpful. And especially if you can take it one step further and say to, you know, your jurisdiction, like, Hey, you don’t have to be afraid the sky’s not going to fall if we do this thing, here’s a list of all the jurisdictions who’ve done it. I’ve actually had them. And just being real about, like to Kathy’s point. There’s definitely. I mean, I come from a healthcare background. There’s definitely cases where like, you want to be ultra conservative. Like if someone can get hurt, like there’s no messing around, they’re like play it really safe.

But, you know, to the folklore stories, like be also real about what’s the real risk here. Like what’s the product we’re working on. If it’s just like a landing page for a PDF document. Then worst case scenario what’s going to happen, versus somebody getting hurt using a symptom checker online, like they’re completely different than 

Kathy Pham: [00:47:55] yeah.

Like I think in government, for sure. Looking at things, case by case, I mean, we have [00:48:00] cases where, you know, we’re like, yay, open data, open police data. And we’re like, and then release data. And then later on I’m like, Oh no, victim data was in there. We forgot. That’s not okay. That’s, you know, the, and those are scenarios that maybe, Some other smaller apps in the private sector never deal with.

So it’s just a really different level of risk and it makes you understand why there’s some level of bureaucracy. Cause some of this stuff is very sensitive. To your point of like, who are we hurting, Katherine.

Alistair Croll: [00:48:27] Yeah, I think, there’s a, an expression among data scientists that, algorithms that poop where they eat, they usually change the word poop, but the idea for predictive policing is that like you go to where the arrests are made because that’s where the crimes happened. But. All arrests happen with where the police officers are. And then that affects property values on Zillow, that’s hoovered up. And all of a sudden you’re seeing a decline in property that perpetuates the problem.

I think we need to be very conscious, especially as we automate a lot of these processes, that what we think of as data exhaust can actually be grist for a mill with very unintended consequences. And [00:49:00] it’s super hard to predict these things ahead of time. 

When you set out to, to define a policy of some kind or a new product, you’re going to build. You know, what’s the process that you go through to try and understand all the unintended consequences, all false positives and false negatives. I mean, other than sitting around just pretending the sky is falling and coming up with horrible list of things that might go wrong. How do you be aware of where the edge cases are that might get you in trouble at the outset?

Ayushi Roy: [00:49:28] I mean, just jumping off your example, Alistair like arrests are not, where crime is. Right? There’s a difference already between where crime exists and where arrests happen and that kind of, like trying to put our finger on the unknown unknowns sometimes feels like just reaching in the dark. Right? But I think that there are actually, sometimes there can be structured ways to allow for unknown unknowns to move into the light, [00:50:00] right?

From the dark, into the light, if you will. And. This is becoming a very odd, extended metaphor. I will stop with the extended metaphor now. I think like, you know, one of the things that we were actually talking about earlier around engagement or working in the open is a big way in which known unknowns can become, at the very least, an unknowns. Right? 

So one of the things that, 18F I think does really, really well is we do have a lot of documentation about how we work and we have documentation out in the open. We also will often have, you know, case studies about our efforts that we will write up, sort of the same time we might write up a post-mortem and have that in the public, not to mention all the actual work would make on, github is completely public.

And, I, you know, I was using 18F documentation before I was at 18F to build out some things. I know that each documentation has still been very useful. Thank you, Katherine. For many others, we’ve had calls at different state and local governments actually that have come [00:51:00] to, my co-director Alicia and I at 18F asking actually about documentation because Hey, we’re modernizing a similar legacy system. Could you help us? And that will then turn into some kind of formalized, you know, engagement and contract, but it all starts off with the distribution of knowledge, right? Cause knowledge is a form of power and it is so important for us to democratize that power, democratize that knowledge.

And, that’s a big way that not only do we benefit right by someone pointing out the unknown, unknown for our 18F team, but it also helps with us being able to share our learnings out with those teams, be more nascent, the digital service teams that are trying to be birthed all over the country and other countries.

Alistair Croll: [00:51:45] You’re making me wonder why there isn’t like a hacker news for digital government because there are places. I mean, hacker news has lots of issues too, but it is a place where you can go and vote things up and down and [00:52:00] have conversations about, you know, what’s working what isn’t and, and I mean, github certainly has some of that, but, it seems like one of the, you know, moving from public to private sector or private to public sector, there’s some things you gain and some things you lose, but before this conversation and listening to you, I hadn’t realized just how much of an advantage it is to be able to stand on the shoulders of others, which you generally can’t do in the private sector.

And it seems like that’s something, we should play up much more when it comes to empowering product teams in government is giving them access to budgets for user research, letting them redefine what the success metrics look like. So it’s not just like number of retweets or conversion rate or things like that.

I mean, even the term conversion rate is a horribly wrong term for public sector services. And so there’s, I think you’re making me realize that I’m taking furious notes here. This ability to work in the open and share best practices is something that, we often discount because when we’d bemoan, the [00:53:00] constraints that are on public service product managers, but in fact, that’s actually a superpower.

Ayushi Roy: [00:53:05] Well, one more thing to that. I’m sorry. Kathy, can I just add one more thing? And then I’ll actually the other thing I want to point out, Alistair. It’s important. So we, three of us here are a part of teams that have built products for the government and continue to do so, but it’s important to know that most government agencies do not have the kind of funding where they can hire an 18F to come in and build a thing, right? 

Like it’s exceptional to be part of a city that is as well resourced as New York City is. It’s very unusual for most governments to have that kind of resource capacity. And that’s what makes the knowledge distribution even more critical because most agencies will actually buy digital services, rather than be able to build them. 

And so being able to have public distributed information around how procurements happen, how agile [00:54:00] modular procurements can happen, which is the big thing the 18F has been working on. I know USDS has also put a lot of effort into is I think what’s even more important than building in the open is actually buying in the open, but, Kathy, to you.

Kathy Pham: [00:54:13] Yeah. I, I had a bit of a tenant, cause you were talking about, you know, making the unknown known and immediately fetched to the equity by design framework that talks about making the visible, the invisible visible. And in addition, the us all sharing stuff I’ve seen it from experience over and over again, that oftentimes when you’re banging your head against, against the wall, trying to figure out like what you should build or why can’t you build something or something’s really complicated.

And then you go out into the community and into the field and start talking to so many different people that you don’t normally bring into the room, whether it’s veteran’s doctors on the frontline, or like the people who take the records for the doctors at the veterans hospital that no one ever talks to.

And there are all these, all these folks that are. You know, bring the known into the room because they historically have just [00:55:00] never been, been asked and there’s so much, and there’s all these answers out there already. Or I feel like oftentimes on tech teams where like we have to like be here and think so hard and figure out the answers for ourselves.

And there’s so many answers out there. And how do we, create like a framework and actually both of you, Katherine, Ayushi talked about, frameworks on this to bring, bring that into the light so that we can build better products in the public public sector. 

Alistair Croll: [00:55:24] So I’ve got, we’re almost at a time, but I want to ask one more question and this one’s a bit of a, a bit of a change.

So over the weekend, GPT3 kind of exploded on the internet. Very briefly GPT3 is a model created by open AI that was trained on the internet, which is already a scary thing. And it has 150 billion dimensions of it’s a machine learning model. That’s a very, very powerful version of, of autotype when you type something into your phone and it suggests the next word.

Only in this case, it can suggest an entire treaties by bill Gates on vaccines [00:56:00] and people that built everything from search engines to speaking out loud and having a generate figma code that becomes a prototype to, Interactive dungeon games, like, like an RPG and they’ve built them in a couple of hours.

And so in the old days we used to write code and it produced data, but now you feed data and a prompt to an algorithm and it produces code. The problem is it’s not deterministic. So each time you ask it, you get a prompt, it produces a different result. It has tremendous power. I mean, there was a good example where someone provided one question, about American history and it wrote 10 other questions, and then they fed it the 10 questions and it went and got the answers. So now you’re a teacher. You just made a quiz in like two minutes, right? But one of those quizzes, one of those you don’t, you have to then have a human go back and review it and say, wait a minute, two of these questions are incredibly inappropriate because it has no sense of right and wrong, although it is terrifyingly powerful.

And I firmly believe that we’re going to look back at July, [00:57:00] 2020 and go that’s the moment we started to talk to the internet as an entity, rather than people on the internet. We’re moving from a world of determinism where you build something and you know what it does to a world where it’s very, non-deterministic like, I think this will probably be good, but it might not be. I can’t tell you what the ROI will be ahead of time. This is a little, like I said, it’s a little off topic, but, what are your thoughts on government adoption of AI and other non-deterministic technologies? How do we manage to put something out there that may break horribly and look politically incorrect at the same time, or very inappropriate or marginalizing at the same time we have politicians who go, Oh, don’t take those risks. I look terrible. 

Does that mean that the public sector is doomed to never take advantage of these incredible potential values that we get from machine learning technologies? 

Kathy Pham: [00:57:52] I have been building machine learning technologies since my freshman year of college. And so it’s not [00:58:00] new and  machine learning has been around for a long time, so I tend to be incredibly skeptical of all things, just seeing under the hood of how things work and even things that tend to be very flashy. Knowing how some of the bits and pieces put together. And so I think my general take on this and there’s a joke around USCS, we would say is something like lines of start talking to me about AI and machine learning once I can query a database in the system and get back the results of how many, how many veterans we have across a hospital. 

So on some sense, maybe it’s promising, assuming you might have a team in government that knows how to actually apply it. but at worst it could be someone packaging, something around this technology, selling it as, magical solution to, giving answers to something in government and then you buy it and it costs a lot of money, people don’t deploy it very well. And there’s all these other infrastructure problems in government that we have to have to solve for first, before we look to all these shiny, shiny things. So maybe if someone can see [00:59:00] how this is tied to some of the basic things that we need in government, otherwise it’s incredibly shiny and may or may not be that baked and might be really good at, you know, remaking a search engine or doing things that might seem kind of fun and flashy.

Like Watson was for healthcare and now 10 years later we realized actually, no, it’s not really good. And so. Yeah, I think the general sense is it’s not really about taking risks and maybe that’s the case where government is right in and being careful because it may or may not be the right piece of tech.

It’s like another piece of technology and you have to have people in the room to really understand if it’s relevant to what it is we’re building. It might be, but it just might be totally irrelevant as well. 

Alistair Croll: [00:59:43] Katherine. How do you feel about AI and stuff like that? 

Katherine Benjamin: [00:59:46] Well, Kathy summarized my feelings perfectly.

I was actually going to say once we don’t have fax machines, I’ll feel a lot better having that conversation. I think, one note I’ll add is, I remember in the [01:00:00] NHS, when they brought in, electronic patient records, one of the risks was always, or the conversation that was happening a lot was this is presenting, a whole new level of risk to patients about breaches of their patient data, which in some ways is true.

But I think the other thing to consider is that a lot of those risks were already there, but people weren’t looking at them. So there were still cases like a friend of mine. Anyway, I’m not even gonna say it, but they, people leave patient records in places they have to like, you know, there’s all sorts of breaches that happen.

And. Hopefully they’re contained and of course technology can make them broader. but like specifically a reason to not use patient records, I would hear is, well, you know, very vulnerable, hard to reach people are gonna be exploited by abusive partners, into sharing their patient records. And this is a new thing that isn’t happening right now, where I would argue that, that if someone’s partner is abusive and trying to get their patient records, whether it’s digital or not digital, that’s still going to happen. The sort of medium is isn’t the issue. So I think when we’re looking at these types of new technologies, it can be a distraction from trying to solve the current problems that we have with existing [01:01:00] technologies. So does it create new and scary trends? Absolutely. is it more scary than where we were before? Probably, but, let’s not focus on the forthcoming things before we solve issues that we have. And I think it’s a great way to forget the meaningful work that we’re having, particularly at this exact moment in 2020, in the summer 2020, and all that’s been happening. I’m like, let’s not lose focus on the immediate calls to action that we have.

Alistair Croll: [01:01:23] I love this concept because what I’m really hearing here is, there’s more than enough, low hanging fruits that we can just go fix this stuff. and you know, this idea empowering product managers means. Like making them feel that just bringing us into the 21st century is perfectly acceptable as progress for many of these things and that you get to work in the open and so on.

This is a really interesting reconsideration of how to empower product teams is don’t think about them like they were private sector product teams, and I love the idea of changing the language, 

Kathy Pham: [01:01:57] But it doesn’t mean like we shouldn’t. I hate the word innovate, [01:02:00] innovate, and do new stuff, right. Their cases are leap technologies all the time. Like you just skip over the mess and you make something new, but it’s more like in the world of maybe somewhat limited source resources. Like we shouldn’t only put all of our eggs towards these like flashy things while our house is still burning. And you’re like, “Oh, I’ll just buy like a new fountain”. And you’re like, “the house is burning just fix the house first!”

Alistair Croll: [01:02:21] And it does seem like I was talking once to a policing guy who told me, You know, he’s talking about the concerns people have around digital and the conclusion he reached was once upon a time we had a suspect and went and got data on them. And now we have data and go find suspects in it. And so I think the fear is that because it’s digital, you don’t have the coefficient of friction of the analog mechanisms that wasn’t really protection, but you know, who wants to go dumpster diving? That’s not much fun. Like you could always find those records, right. But you had to go dumpster diving and that was a lot of work. And now you can just search. 

So I think some of this stuff seems to be the fact that we are talking about digital technologies [01:03:00] means that we have to deal with the consequences of shifting from atoms to bits and a lot of this stuff.

So, this was a fascinating conversation and I know the three of you are going to go grab a virtual coffee right after this. but thank you all so much for being here. I, you know, in my notes here, I think I, I loved the points about legal versus folklore. About need versus luxury. Ayushi you mentioned, your podcast than designing for the margins.

Can you tell us quickly what that was? 

Ayushi Roy: [01:03:25] Oh man, you’re allowing me to shamelessly plug this? Thanks 

Alistair Croll: [01:03:29] Remember, we like to share. It isn’t the private, it’s not the private sector. 

Ayushi Roy: [01:03:33] That’s right. it is called “We Who Engage” and, you can find us on all the things, Apple podcast. We were named one of the best top hundred last year.

So check us out there. yeah, SoundCloud, just Google us. You’ll find it on my Twitter. but yeah, “We Who Engage”. We’re actually recording season three at the moment. so you’ll find two seasons already up there with 10 episodes a piece, fantastic [01:04:00] guests and conversation. 

Alistair Croll: [01:04:01] And Katherine and Katie – Kathy, sorry. do you, have things that you would like to share that people should go listen to or check out or read or watch. 

Katherine Benjamin: [01:04:10] I encourage everybody to complete the census. 

Kathy Pham: [01:04:14] Yes, complete the census!I have, we’re going to host an event maybe in a month or two around product management in the public interest. and I’ll probably post that on Twitter and beyond and what that means. So- 

Alistair Croll: [01:04:25] Well, and if you shoot us a link, we’ll put that in the FWD50 newsletter because we send out a newsletter every week with like past interviews and stuff too. 

Fantastic. Okay. Thank you all so much for being here. I’m very jealous. You should go have coffee, but I would love to be joining that coffee conversation, as someone who’s from the private sector, but spent a lot of time working with people in the, in the public sector, you’ve definitely opened my eyes to a few things I hadn’t thought about. I always talking about the drawbacks, of oh we can’t innovate as fast and we can’t try things out and we gotta navigate red tape, but I hadn’t realized that that also the ability to just put out the house [01:05:00] fire or  work in public, is, are, those are really super powers that redefine what it means to be a product manager, I think.

And I love this idea of, you mentioned the eight steps of product management Ayushi. I think it was you, you know, that idea of involving people in all those eight steps. If you have a link on that, we’ll post it in the, in the description for this video. 

Thank you all so much for being here and for those of you watching. we’ll see you soon with another episode of FWDThinking.