Security Cryptography Whatever
Security Cryptography Whatever
OMB Zero Trust Memo with Eric Mill
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
The US government released a memo about moving to a zero-trust network architecture. What does this mean? We have one of the authors, Eric Mill, on to explain it to us.
As always, your @SCWPod hosts are Deirdre Connolly (@durumcrustulum), Thomas Ptacek (@tqbf), and David Adrian (@davidcadrian).
Transcript:
https://securitycryptographywhatever.com/2022/06/10/omb-zero-trust-memo-with-eric-mill/
Links:
- OMB Memo
- Executive order on cybersecurity
- PIV card
- BeyondCorp
- HSTS Preloading
- Neither Rain, Nor Snow, Nor MITM
- EDR memo
- Technology Transformation Services (TTS)
- Is it Christmas?
"Security Cryptography Whatever" is hosted by Deirdre Connolly (@durumcrustulum), Thomas Ptacek (@tqbf), and David Adrian (@dadrian)
Hello. Welcome to security, cryptography, whatever. I'm Deirdre.
SPEAKER_00I'm David. Not it.
SPEAKER_04Who are you? Tom.
SPEAKER_00I'm Tom.
SPEAKER_04And our guest today is Eric Mill, who is currently at OMB, the Office of Management Budget, right? In the United States executive branch. Hi, Eric. How are you? Hello. Yay!
SPEAKER_02Hello, thanks for having me.
SPEAKER_04Who are you?
SPEAKER_02Who is this guy? So, yeah, Eric Mill. So I work for the Office of Management and Budget. That's what OMB stands for. It is, I'm I'm constantly having to describe that to friends and family. But it is so it's a part of the executive office of the president, which is what people typically refer to as the White House. And we're the part of EOP that focuses on agency, the on how agencies work. Obviously, the budget is a big part of that. It's also the management side of it, which is essentially where a lot of policies and oversight and other interesting interventionary stuff happens over agency operations. We're the office that is responsible, and my part of that of OMB is the office of the Federal Chief Information Officer. That's who I work for, is the Federal Chief Information Officer, Claire Martyrana. And our office does technology and cybersecurity policies for how agencies work. There's an obvious intersection with how their public-facing stuff happens as well, but we really focus on like policy for agencies. So for the folks in this audience who might be aware that about seven, eight years ago, there was an HTTPS policy that came out of the Obama White House around that time that mandated HTTPS for federal web services. The signature on that policy was the federal chief information officers. That's the sort of thing that this office does. Yeah, you didn't spend your whole career in the government, right? Like you've been elsewhere. Yeah, I have been elsewhere. The ratio is starting to add up to a lot of government time. But I started, I've been a software engineer and web developer for a long time. So I was somebody who really got into making websites in 1997 when terms like DHTML were still tossed around and I could entertain people in my high school library by showing them how I could make a page that made the screen flash colors and stuff. And I went to college for computer science to get a bachelor's, I went to Worcester Polytechnic Institute. And I, yeah, and for masters in. All right. I couldn't really imagine doing anything else eight hours a day, like the concept of having a full-time job was very dreary, and solving puzzles for fun was seemed like a pretty good way to spend it. So I I signed up for CS degree. That was also, this is back in 2001, was my freshman year, which was when the dot-com bubble burst. And so a lot of people were fleeing that major, and I hung tight to it because I couldn't imagine doing anything else. And so graduated in 05, worked out in the private sector for a little while. I worked for a small consultancy called ThoughtBot, which made what at the time in 2006 was a really big bet on Ruby on Rails, which was not whose future was not assured at the time. And I took a big pay cut to do it because Ruby just seemed really cool to me, and you can make fun of me for that later as you wish. Um but I sort of just spent a few years there, just becoming a functional software engineer. Ended up then moving to a nonprofit called the Sunlight Foundation in Washington, D.C. There's a brief interview there where I worked for a sort of politically focused, democratically leaning digital consultancy called Blue State Digital on their Obama campaign work. And then after that, I ended up moving to DC and working for the Sunlight Foundation, which was does not exist anymore, but at the time was a sort of 50-person strong nonprofit with a 15-person engineering team and a few designers doing all sorts of different things. So I was there for about five years before I jumped into the government around 2014 when the various digital service teams that now exist were starting up. So that's I did that also for another four or five years. That's where I started getting more formally into policy work and then ended up working in the Senate for a little while after that, and then just did a year again out in the private sector before this, where I was the lead PM for the Chrome browser for security. And then I just jumped back into government again when this came around. So that's the background there.
SPEAKER_04We're talking to you today because there was a memo put out by your office in January with the title Moving the US government towards zero trust cybersecurity principles. And for anyone who's been like vaguely paying attention to what the government recommends about how to secure systems, whether federal or in the United States, this was one that actually was like, oh, these are all good recommendations and not just like scraping the floor with bare minimums. And I think we all started to read it.
SPEAKER_01That's what Deirdre says. Or for anyone who's selling a security product, they definitely noticed this the hell they definitely noticed the hell out of this too.
SPEAKER_04Yeah. So Thomas read it in depth. The rest of us glanced at it.
SPEAKER_00We've all read it, but I think before we get started, it would be good to hear in the like two-paragraph summary of the old state of the world and the state of the world that this memo was recommending before we dive into details.
SPEAKER_02I guess that kind of starts with the term zero trust, but let me just give a little bit of the right the scenario that was kind of leading up to this, right? So this is done under the aegis of the cybersecurity executive order that President Biden signed within a few months of taking office, that was clearly partly in response to some of the compromises of federal agencies that are often referred to in shorthand as solar winds, which involved the compromise of that vendor. Obviously, there's some other stuff wrapped up in that too. There was also shortly before issuance of the order, too, there was the colonial pipeline ransomware attack that a lot of people felt and that real people around the country noticed at some in their day-to-day lives. And that executive order asked for a whole bunch of things. And my office, the Office of Management and Budget, has been responsible for a significant number of memoranda and policies to implement pieces of that. So we've also done other memos around increasing logging the data within these different systems, endpoint detection and response. There's some stuff going on now around secure software development. But one of the big things that it called for was moving agencies towards zero trust principles. And zero trust is one of those words that can mean a lot of things to a lot of people, as I I know Thomas was getting at, and it's certainly true. And I think if you read what we were trying to do with this, and if you dig into how we frame it, the goal of this is to really take least privilege and the concept of not having implicit trust seriously and really taking that to its logical conclusion. So there's a big emphasis on moving away from, for example, a network perimeter-based approach. It's not the first time this concept has come up, it's not the first time it's been involved in government documents. It was certainly talked about in some of these terms after the OPM breach six, seven years ago. But I think it is fair to say that it has not been taken to this degree of some significant things are going to change over time. It's very common today in federal enterprises for you to log in to start your day by putting your PIV card into your computer and logging into your intranet. And once you're in there, you can pretty pretty easily start logging into other things and doing work. And one of the things that we try to hammer home there a lot is in mature environments, you are logging into the application layer. You are you're doing there's still single sign-on, but you're doing this at layer seven and not layer three or four. Um and that that what that constitutes is moving that perimeter closer to the information that's being protected and also giving you a much more semantically rich environment to work in. Because ultimately, Zero Trust is about least privilege, about taking combining information about the user, the time, the device, the nature of the application, what of these are anomalous, if anything, be having more information to synthesize at your disposal, then you then many enterprises, especially those that aren't gigantic companies with infinity engineers, have available to them. So those are big changes, and I really taking the concept of like, no, when you're if you're just sending traffic unencrypted around your internal network, you are putting tons of implicit trust in that internal network, and that's not what this is about. So even though some people might say HTTPS is getting a little passe at this point, we know that it's very important to do that everywhere, not just out in public, but like inside your internal organization.
SPEAKER_01I guess like a first question I have, just to lay the groundwork here, is so this is an OMB memo, and the OMB governs the way that all the agencies under the executive branch actually operate. So, what's the force that an OMB memo has? What if is it possible for an agency to read the OMB memo and say, this is all interesting stuff, but here at the Department of Fisheries and Wired Life or whatever, we're just gonna keep doing things our way?
SPEAKER_02So the Office of Management Budget uh published a lot of memos, right? And these memos are binding. There's the statute to back that up. I have had the experience of working before in the government when I was, particularly when I was at GSA, I mentioned the HTTPS policy earlier. That is a policy effort that I think was successful for what it did in the federal government. And my experience was that it was successful, not because it was perfectly crafted, but because years of private, persistent, multi-stakeholder follow-through ensued to have that actually happen. There were people who cared about it who answered as many emails as were necessary to resolve questions. There's public material to help folks do that. There's ample public engagement and private engagement to see this stuff through. And one of the things we're really trying to do here, it's a tricky thing because this is a big enterprise security overhaul. This isn't just like deploying a protocol. This is a major change to how an organization functions. That is a multi-year project. And so, how do you maintain urgency? How do you not let the energy drop out of the room? And so we have taken a few routes to try and do that. Part of that is how the memo itself is structured. So there is a mix of places where there's flexibility and that sort of that multi-year timeline is explicitly contemplated, and we're asking you to come back with a plan for are you gonna do that? Some of those there's actually like more short-term. What it means to be on a path to do this in three years is that you've done this basic thing in the first year. And for example, there's a couple of sort of MVP style things in there where we're saying, like, Zero Trust is about combining a lot of signals, but we will really want to see you get at least one signal relating to your to the devices that your employees have. And we want to see that connected to your authorization authentication decisions in some way in the first year, and we think that is a reason that will get you over some barriers by forcing yourself to do that, and that will be relevant for the rest of the time. Similarly, we have a thing in there about requiring like a significant internal system being prepared for it direct internet accessibility to match the threat model that we are talking about here, where things should effectively be treated as internet connected. And that doesn't mean that an agency is necessarily gonna do that to their entire organization, but that being able to do that and figuring out a security strategy, a set of tooling, a set of processes that like that can get everybody comfortable with doing that is gonna be important. The other area in which we're doing this is in which we're trying to sustain this over a few years, is we're really taking these plans that we have asked for very seriously. So agencies have submitted to OMB and to the DHS Cybersecurity Unit, who CISA, the Cybersecurity and Infrastructure Security Agency that we work very closely with. They've submitted those plans to us, and we're reviewing those plans. We're gonna be talking with everybody about them, and we're gonna be taking those are gonna form the foundation of a lot of our follow-through and oversight work in the years to come. And that is separate from a bunch of other stuff that we're also doing, where we routinely have seven, eight hundred-person strong gatherings of the federal IT community to like talk with us about some of the issues they face, raise questions, and and ultimately provide input into how we go about providing further guidance to them. So there's a whole host of things that are sometimes don't necessarily come across when you just read a fancy PDF with a signature at the end of it. But one of those things about doing policy of any kind is that issuing the policy is the beginning of the work and not the end, even though a ton of work goes into making that policy. And we're really trying to embody that with this.
SPEAKER_01There's a lot of agencies, right? Like you think of the top-level cabinet things, but there's like hundreds of them, right? Like the Susquehanna River Basin Agency or whatever. And these are, for the most part, they're all separate IT organizations.
SPEAKER_02Yeah, there's many agencies. There is the cabinet, and then there's all these small agencies that would not meet some people's definitions of small, but in the federal government are considered small. There's microagencies, and they really run the gamut. There are agencies with single double digits of people working for them, right? There are also agencies that have very specialized functions or may not have even a full-time IT person ranging up to cabinet-sized agencies that have what you and I might consider many almost independently functioning organizations inside them. Like the Department of Commerce contains the entire Census Bureau and NIST and NTIA and all these things that a lot of people engage with as if they were their own agency, but they are in fact part of the Department of Commerce and report up, and they're a real functioning unit too. But these are big sprawling, the federal government is a sprawling enterprise. That's exactly right.
SPEAKER_01Okay, the memo the memo itself, like it's a big memo, right? There's a lot going on. And I don't know if this is a critique of it or just like an observation, but it reads as if somebody took a look at the uh the whole sprawling gamut of different agencies and different IT organizations and like wish cast the ideal kind of enterprise security organization in some detail for all of them. Like it's a whole architecture, right? I think the top line thing that people have taken from it, and I think that the thing that probably made the most news was there's a sort of implicit repudiation of VPNs in it, right? It's an old problem in enterprise security where people are running these kind of these network architectures where the way you get access to things is to get on the VPN. And then once you're on the VPN, you're on an internal network. And in our industry, in our field, if you're on somebody's internal network, that's almost always game over, right? Once you're there, there's nothing that's going to prevent you from escalating privileges all the way to whatever the maximal privilege is. And it seems like the goal of this, one of the primary goals of this, is to make these organizations more survivable so that they're not dependent on simply having you put your PIV card in, you get access to the network, and now you have access to literally everything in there, right? Everything now is authenticated through an IDP, through a single sign-on system, and there's device attestation and all that. But it's pretty ambitious, right? Like most enterprises are not operating at this level of sophistication, isn't even the word I would use. I would use coherence as the term, right? Like it's a very coherent view of how enterprise security should work. That is, I would say it sets the bar past where the enterprise, like where industry sets that bar right now.
SPEAKER_02Uh, I think there's a lot of truth to what you're saying here. So there is absolutely a deliberate strategy here of I described what preceded the cybersecurity executive order, and it's not the first or second time that the US government has been punched in the face by a significant adversary. And there's only so many times that you can do, you can have that happen to you, right? Before you really have to not just focus on triaging incrementally to something that's a little bit ahead of where you are, right? There there it there is a time where you have to lay out like we have a lot of work ahead of us. And this is something like what it looks like. There's definitely lots of room for some flexibility in there into how agencies choose to architect their work. And if you dig into the whole NIST zero trust architecture special publication, it goes into a lot of detail about that. It's a great publication, but you're right that there are some things here that are very different than how federal agencies operate and and in fact how a lot of industry operates. This is something that I and other folks in organization and in the administration have been talking about with this a lot is that this isn't a case of the government running 10, 15 years behind industry in this particular thing. This is something where, to a large degree, like we're all in it together, and there are tools out in the space that are still maturing. There's a lot of friction still to just doing all of this stuff in a big enterprise. And you know, that's understood, but it I think it really can't stop us from action and from some understanding of what we need to do.
SPEAKER_00It's definitely a very ambitious plan. I think that in the three-year timeline, there'd be people at large, nominally call them tech first, tech companies, scroll stage startups, that if they got some of these uh requirements in a three-year timeline, they'd be like, holy crap, I don't know that I can pull that off. And personally, I think that's great.
SPEAKER_01Kind of like we we can summarize what the architecture is, right? There's a kind of a core idea, and Eric, I'm saying this so you can correct me. So we'll jump in when I've got something wrong, right? But there's there's a core idea of all of these applications that like any system that's running inside of an agency presumably exists to support some application that they're actually using to serve their mission, right? So for all of those core applications, the idea is they're no longer gonna be parked on internal networks where you get you know a key to the backstage network and that gives you access to the application. Instead, all these applications are gonna be not like because they need to be on the internet, but like they're gonna be their their security posture is they're internet accessible. And the way you control access to all these applications now is through single sign-on, through an IDP, which is a good a good thing. Everyone should be doing that. That probably is where the industry is heading. Uh because of that architecture, and because I think you're probably working to correct a lot of bad practices that came about from people hiding things behind VPNs, there's an idea that we're not doing any kind of protection or enforcement based on VPNs anymore. That these systems, if you're running it, you know, the VPN can't be what's protecting it. Something else has to be protecting it. In addition to that, there's pretty strong recommendations that in addition to, you know, single sign-on through an IDP with phishing-proof multi-factor access, which is all like motherhood and apple pie, right? That you're gonna get device signals, right? Like that one of the things you're trying to do with this architecture is not just prove that, you know, in theory it's the person themselves that are requesting access to the application, but that they're doing it on a machine that you can somehow trust is, you know, up to spec or whatever. It goes further than that, though, right? Like there are prescriptions in the memo about security testing. This is a big thing where I'm gonna have follow-up questions, right? But there's language in there about moving all of these applications and all the federal agencies to a point where they welcome external security testing. And where they have a published VDP effectively, it's like saying we're gonna have a bug bounty for everything inside of the federal government without the bounty part, right? You know, an overt authorization for people to do security testing. There's stuff in there about if we're keeping secrets, we're gonna have the OMB is gonna come up with a document classification process somehow, right? You guys are gonna figure out what sensitive data is. And then for access to that data, there's gonna there needs to be, you know, KMS or vault style systems where the specific guidance is that even if the application itself is compromised, you'll at least have logging that these, you know, these particular documents were accessed. Which you can see why you would want that, right? Because after solar winds, you're like wondering what the hell our adversaries actually had access to in the first place. But a knock-on effect of that is that the way that you provide that, the way that you do, you know, I have an audit log of people who access something even if the application is compromised, is it's protected by something like a KMS or you know, a vault or something like that. It goes on with logs and things like that. So it's it's pretty comprehensive. There's mandatory encrypted DNS. I was gratified to see that it was DNS over HTT and DNS over TLS, which is awesome. To be honest, like we saw we see a video feed of you when you come on the podcast, right? Like in the days leading up to this, like I was making wise cracks, not about you, but about the US government in general. And Alex Gaynor, a friend of the podcast who also works in the governments. Have you ever considered working for the you know the federal government? And it's there are reasons I wouldn't, like the fact that I don't want to move to DC, but then I saw you pop up on the podcast in a s in a jacket and tie, and I'm like, I'm never working in this environment.
SPEAKER_04It's the American flag behind you.
SPEAKER_02Thomas, you absolutely should go work for the US government. I'm happy to say that to everybody on this podcast. You should, and by the way, I yes, you're right. You have a video feed of me and you're seeing me in a suit and tie, which is not representative of most of the time that I've spent in the U.S. government. And yeah. Anyway, we don't need to go into that.
SPEAKER_01But start start here. Is encrypted email gonna happen?
SPEAKER_04This says encrypted email traffic. This is not the same as be encrypting your emails, right?
SPEAKER_02That's right. It's it's encrypting email traffic in transit. Okay. That's the subject of that whole section, right? So again, if you if you take a look at the cybersecurity executive order, which for the the careful reader is executive order 14028, it it talks about uh just encrypt I mean encryption in transit in general. And and that is a big push right now in the federal government to button up on encryption in transit. And what we are trying to focus on in that part of the memo are some places to really make sure we prioritize and where there are some specific things that we're probably gonna have to resolve in order to do that. Right. So with HTTPS, there's a focus there on browser preloading because that is an as a pretty incredible forcing function for making sure that you're tackling both They essentially, I'll maybe I'll misuse this term, but essentially split-level DNS. You have a ton of agencies who hang internal host names on their public domains, and there is actually a way to externally validate some enforcement of that for browser-based HTTPS at least with preloading. So that's a particular angle that we're we're dealing with. And the US government has been preloading duckov domains as they've been registered for some time, and this is a bigger retroactive push. And with DNS, I I think it's pretty well understood that this is still a thing that is, I mean, there's still work being done to do this. DNS over HTTPS just was built natively into Windows for Windows 11, right? So this is something where we know we need to do this, and we know in fact Sciza has been standing up protective DNS, which supports this, right? There's there's a there is a big push to get this stuff in place and to start using it, and we need to drive to the point where it is the norm and it is encrypted at for as many legs as the US government can is in control of. And then with email, I think there's probably a wide understanding of the folks in this on this podcast and and and among readers that email encryption is pretty fraught and that it's fundamentally still opportunistic in the way that it is deployed on on over the internet. And what you see there is a request of SISA and FedRAMP and others to work on this. It's not it's not spelling out exactly what the solution is, but it's saying these are these are the things that we're gonna have to tackle if we actually want to encrypt this stuff in transit.
SPEAKER_00This is where I feel compelled to plug the paper I wrote back in 2015 or 2016 called Neither Rain nor Snort nor Snow Nor Man in the Middle about Start TLS.
SPEAKER_01So like the external security testing thing, the VDP stuff, is that a marked shift from the way the agencies operated before? Is this a big thing? I think I should be thinking a lot about, or is this just like a like a formalization and a continuation of a kind of a tacit policy that already existed? Like how big a deal is the VDP stuff?
SPEAKER_02I guess it depends on how you look at it because there has been some policy on the books around VDPs from OMB and from CISA as well through their there's a directive to agencies. CISA publishes mining operational directives, and one of the ones that they published, I guess a couple of years ago now, was requiring the instantiation of vulner disclosure policies. So there is something there, but it is very new and to most of the federal government. I think many folks may be familiar with what the Department of Defense did with Hack the Pentagon some years ago and some other bounties since. Proud to say that at GSA, we also had the first civilian bug bounty and vulner disclosure policy some years ago, but it is still quite new. So one of the things that we're trying to dig at with that application security section with vulner disclosure is really presenting this concept of there's multiple rings here of security that need to be brought to bear, all the way from bringing in some seriously expert people, which is you know a heavy overlap with probably the read listeners of this podcast, and right, and having them take on a sort of application by application basis, do their best effort to subvert its expectations, the way that our adversaries go about doing this, all the way towards, you know, there's a pretty huge existing body of security analysis that is part of the federal government's standard operating procedure. And then including even external, making sure that we're able to hear what folks on the outside can see and want to tell us. I'm certainly comfortable talking a little bit about like before this was a thing across the federal government, you know, it was not unheard of for people to report serious vulnerabilities to the federal government by, you know, having somebody they trusted in a random unrelated agency, you know, essentially proxy their report over, you know, using contacts that that person have had inside the government to get that done out of not knowing how to report it, and also a fear of what it means to report that stuff to the federal government and a worry that the conversation will focus more on how you found it than what the severity is. Yep. And and so there, you know, there has been certainly a multi-year effort to change the norms around that and create these open doors. And we really do view it as a critical aspect of what it means to take seriously the concepts and principles that are part of the zero trust strategy and just part of modern enterprise security.
SPEAKER_01So, like brass tacks on this, right? So I'm I'm a vulnerability researcher. That's my background, right? And I interact with kind of government systems every once in a while. And, you know, uh most of the government systems I interact with are like at the state and county level here, right? You can imagine they're like state and county IT systems. They're not like Google didn't build them. And you know, they're they're janky in lots of different ways. As a vulnerability researcher, I would be terrified to poke around any of those systems looking to see just how janky they were because I can just imagine they're like I can imagine the response I might get from you know an admin there or how that might escalate. Is it safe?
SPEAKER_04They yell at you for looking at an HTML.
SPEAKER_01It might yeah, that yeah, exactly, right? So is it is it safe now? And if it's not, will it be safe with within the next, I don't know, year, two years for a vulnerability researcher like me, but you know, good, to if they're like just randomly poking around looking for.gov systems to go look for pre-auth SQL injection, right? Is that a safe thing to do now and would it be safe in the near future based on this memo?
SPEAKER_02So, you know, Thomas, things have changed a lot. And it's not just something in the government, right? Or it's certainly not just something in the federal government. One of the things that I worked on when I was in the Senate in 2019 was vulnerability disclosure for election systems, which involved mostly state and local and county systems, like the kind you're terrified of poking into. So I spent a lot of time dealing with researchers who were in fact doing that and trying to get things reported and fixed. And that did lead some states to some secretaries of state to publish vulnerability disclosure policies for the first time in that space. There's other areas that are historically litigious that have calmed down substantially. The medical device industry comes to mind over the years, where it is a place where if somebody came to me and was like, I have I found something and I want to report it, I'd recommend they do it. And I would absolutely, you know, at this point in the federal government, if somebody came and found a sensitive thing, I would have no trouble pointing towards CIS vulnerability disclosure platform that they set up with it, which a lot of agencies use, or if an agency is running its own thing, I'd recommend they go in that way. It is a time where I can recommend people do that.
SPEAKER_01Do you see the memo as encouraging, like, you know, vulnerability researchers who sometimes do this stuff for sport or for fun? Do you see this as encouraging those people to start like, you know, aiming their scanners at .gov and looking to see what they can find? Like I get there's a there's an element of tolerance and of streamlining reports so they go to the right people and they don't blow up. But is this a thing that you want to have happen? Are you actively encouraging it with the memo?
SPEAKER_02It's not just us. Uh, you know, obviously I just mentioned SIS's uh vulnerability disclosure platform that they set up. There is an active encouragement by the government to to want people to go and find these things because we want to know. It's better to know than to not know. So absolutely. And I think, you know, one of the though this is one of the smaller things in the memo, there is also an element in there where we also really want to make sure that, especially since there's so much of an emphasis on relying on the cloud, which agencies are doing more of, and taking advantage of the security features you can sometimes only get in the commercial cloud. What we wanted to also clear up is that, you know, just vulnerability disclosure authorizations should be applicable even if there is a cloud provider under the hood, right? And we we want to make that extra clear that just because an agency is moving to host their thing on a cloud provider, that that doesn't get in the way from them feeling like they need to have ears to hear what people find on their systems.
SPEAKER_04So does that mean like you host your app on AWS, and there might be a vuln disclosure program for AWS subsystems that you may use, but just because that exists, that does not preclude your app, your software that is running on AWS for also having your own vuln disclosure process, right? Aaron Powell Yeah.
SPEAKER_02That'd be an example. And there there are a lot of cloud providers out there, not just sort of the big ones that come to come to mind offhand, but right? But there's a there's a long tail of that. And and they are all in very different places with how they do vulnerability disclosure.
SPEAKER_00Run your containers on fly.io Anecdotally, you know, my experience with interacting with the federal government, we we ran a scanning program out of the University of Michigan for many years. That was what most of my PhD was about. And in the very early days, even actually prior to when I was in the research group around 2012, when we were doing the internet wide scanning, DOD like bulk opted out all of their IP ranges. And they remained on our you know, scanning deny list for years. And then when we launched Census as like a public research project, some amount of time later, someone from DoD reached out and was like, Why can't I find any dot mil domains in Census? And we were here's this email from 2012 where you told us to cut this crap out. And they were like, that's interesting. And then some amount of time later, someone with a much fancier title from DoD contacted us and was like, please put us back on your list. We would like to be able to take advantage of all the things that that you're doing and get this information. So there definitely, like anecdotally, has been an attitude change, even just within my limited interaction with the federal government.
SPEAKER_02Um, just to talk about the civilian side of my own experience at GSA, I, you know, had absolutely had the experience of making use of scan data. Perhaps it was from some of the stuff you worked on and some and also from some other sources, and putting those to defensive use and sharing that information with civilian agencies so that they could do something about it. And it it was very clearly helpful to be able to take that outside perspective and move it inside because it just doesn't always happen organically.
SPEAKER_01I I have one more question on this vulnerability stuff, and then I'll let up on this. So there's a big section in the middle of the memo about changing the way that you deliver the actual applications themselves. And you guys get into details about how you're gonna do security assessment. You point out explicitly that like you can't get out of these requirements just by running a scanner once, like where you know you're actively interested in human expertise. And you talk a lot about making that human expertise, about making people that can do software security assessments, for instance, much more accessible. I don't think we talk about making people, but or maybe the okay making more accessible, you know, resources for getting things tested. There's there's a point at which you talk about bringing down the amount of time it takes to get a test scheduled from weeks to days, for instance, and about finding, I assume, vendors to do that with. That's a big question I have is there isn't a huge pool of talent to do that kind of work in general. It's it's difficult for people in industry to get that work scheduled. Like people are you know scheduled way out into the future right now. How do you guys think about who the effective vendors are gonna be, who the effective suppliers of that expertise are gonna be? How are you gonna evaluate that? Is there gonna be like a list of trusted vendors? Is that gonna be a program that you guys run?
SPEAKER_02You're asking a whole ton of questions about procurement stuff before it happens. So I I probably won't be giving you as much detail as you might want here, but I think one thing we're we've we've asked GSA and DHS to work together to figure out some of those things and to and to try to advance this, right? So you're absolutely describing why this stuff doesn't happen organically, and that's that's a problem we want to try and walk towards and not away from over the next couple of years, is is probably about the best way I could put it. But you're also you're absolutely correctly describing the kind of skills that we are looking for and that don't that just don't always get applied consistently to some of the sensitive systems that we have, but they are applied relatively consistently by our adversaries.
SPEAKER_00So let's let's pivot some to identity and devices selfishly because that's always the part of zero trust that I personally find the most interesting. I think for the most part people have agreed on the value of like single sign-on at this point. And like when people think zero trust or when companies market zero trust products, a lot of the time it centers around like the single sign-on and then the reverse proxy to put in front of your app to do the single sign-on. And that's a fairly well-trodden path at this point, although individual instantiations of it can get like arbitrarily complicated depending on your setup. The part that I see less sort of consensus on is with devices. Like in general, even if you have really good asset management and you know what all of the computers that are in your employee fleet are, it's not as clear how to integrate that into your authorization and authentication process as it might be to say, you know, all of my employees have a PIV card because they're in the government or or or some other form of security key and they all have an account in some system, and so you know, I I have phishing resistant MFA, but actually being able to say they're on a government device versus they're on you know some other device, I think is a lot harder. And I mean you could debate the value of do they actually need to be on that device or not. But like I'm curious if I were to pick the spot that's like the least broaden path and is going to probably require the most, let's call it engineering work in here, it would be the goals underneath devices of integrating devices into your authorization signal. And so I'm curious what you think the first steps are there, or like maybe like concrete examples of what might satisfy the memo, or if a government agency came to you and was like, what do we do for devices? We understand identity, but we don't know what to do with your devices, what you would tell them?
SPEAKER_02Sure. And so you're you're absolutely right. This is one of those things where we, I would say deliberately chose this because some of these, some of the things that are required to connect these device signals into your SSO or your enterprise identity system, it's about connecting pipelines, connecting systems that aren't organically connected. Like it's it's not easy for enterprises to do that. They don't do it organically, right? They have to make the decision. And so we, you know, and we did not spell out how to do it and left a lot of flexibility. But, you know, there are certainly things that you can see, you can see close parallels to some of these things here, right? There your device signal can be very flexible. And I don't want to accidentally constrain the thinking here of agencies and how they do this, right? But you know, even stuff like, is this device patched, right? I mean, those are those are things that are meaningful to an agency and to and to its fleet. And and in fact, you know, you can see that in some of these VPN-based solutions we've talked about, where you can you can watch the the client app try and and make some, you know, do some ascertaining of the local environment. And we're talking about connecting information together in a little in a fancier way, but some of the concepts under the hood don't have to be that sophisticated.
SPEAKER_00What what about endpoint detection and response? Do you think that we're we should be or that we are heading towards the world where people are running agents on government devices?
SPEAKER_01It was actually, yeah, I was reading that. It was tricky for me just reading that to get a sense of whether EDR was mandated or not. There are parts of that where it reads like every system and you know all the agencies is going to be EDR'd, and then there are other parts where it's like it might be flexible. So yeah, that was a big question I had too.
SPEAKER_02Mm-hmm. Yeah, that's that's fair. I mean, so there there are a lot of agencies that run EDR already in different ways, right? And you know, that section of the Zero Trust memo links over to a whole dedicated memo on endpoint detection response that we also issued to be responsive to the cybersecurity executive order, which calls for EDR throughout the government. So there's a whole bunch of other material. Uh uh honestly, most of the material about EDR in the government right now is actually not in the Zero Trust memo. Though we talk about it some there, it's it's important to understand that that is a part of what we're doing. But some of the stuff that dedicated memo talks about is are we talking about you know one vendor that everybody uses? Are we talking about flexibility across agencies? If so, how does SF interact? And you you'll see that the memo sort of charts out a path where there is more flexibility. And I'd encourage you to dig into it. But I mean, there absolutely is, I mean, there already are agencies with fairly large deployments of EDR of different tools, and the goal is to have there be a consistent baseline of sorts throughout the government to do that. Part of what we also wanted to acknowledge in the memorandum and in the Zero Trust portion, especially since so much about Zero Trust is a least privileged conversation, right? It's about reducing attack surface and constraining impact, right? That we we want to acknowledge that devices or operating systems, other things that are so constrained by design that they might not support the typical big heavy EDR. You know, that that's not some we're not trying to discourage systemic architectures that focus on reducing attack surface. That's that's a recurring theme in a few areas that we wanted to make sure we reinforce there as well.
SPEAKER_04That's nice. You could like have a fleet of Chromebooks or something like it, which are just like fundamentally just you can do less with them. It's just a browser wrapped up with a kernel and then you're done. Something like that.
SPEAKER_02That sort of thing, right? Yeah, of having we're not trying to rule out the concept of thin clients or constrained devices or other innovations that might come in the future that are designed to just you know focus down a particular device for a particular purpose for security reasons, among other things. And we we want to be able to take advantage of that as well.
SPEAKER_04Aaron Powell Such as like a capability-based OS, which is like a fundamentally different model than a lot of the things we already have, including a Chromebook. It could be more powerful than a Chromebook, but fundamentally more constrained than your like a Linux laptop or a Mac OS laptop.
SPEAKER_02Aaron Powell Right. Conversations about this, about least privilege, come up in all sorts of situations, not just with EDR, but with just how monitoring and visibility work gets done in general. Yeah.
SPEAKER_00The other thing you might see about EDR, and and maybe this is getting more into the realm of the EDR recommendations that exist in other parts of the government, is you could broadly classify them into two forms. There's read-only ones, and then there's ones that do what I think Thomas would refer to as RCE as a service. Where where they will uh run commands for you. And and both of these offer interesting security trade-offs. And is that something that is one or the other recommended, or do we see that there's a trend towards one, or is this just the realm of somebody of a different agency's memo and guidelines?
SPEAKER_02Let me try to answer that with an analogy because I think there's a very similar conversation in sort of network monitoring, right? And about how you and also just where you put your monitoring tools in general. Sure, this group has a lot of thoughts about break and inspect and and network decryption. And one of the things we spend a lot of time talking about is the need to wrestle with the, depending on what you're doing and what architecture makes sense, like to wrestle with the real pros and cons of doing that and about how visibility can be intentioned with attack surface. And this this I think maybe even closer to the example you're asking about, you know, when we talk about like how we monitor servers, right? Like you can monitor a server by having a daemon that runs on it with fairly expansive privileges so that it could do a bunch of fancy stuff and and watch deeply into how your server is performing, or you can have that server be set up to push out information into another space, and then something can be watching that space. And you can, by having that level of indirection, adds complexity, gives you more interesting security properties to be able to essentially just reduce the impact of the thing that's doing the monitoring being compromised so that at least the the adversary can only do more monitoring. Right? That what you're talking about in EDR, the you know, the same considerations are present. And I think our our goal is really just to make agencies have that be a conscious thing that is reasoned about and and and done strategically rather than something that we just stumble into.
SPEAKER_01So so like the elephant in the room on all this stuff is the VPN situation, right? So you know, I think if you look at the story of VPNs in the industry leading up to like a couple years ago, I I think everyone would be jumping up and down applauding the OMB memo for saying, you know, that that that VPN strategy is bankrupt, we can't build security, you know, based on this kind of you know, crunchy outside, soft, chewy middle or whatever, right? There's a million different metaphors for this because it's been a problem for 15 years in the industry. VPNs have gotten a lot more interesting over the last couple of years, right? One big question I have about this is if you have a like a hypermodern VPN situation where you get to do application by application that's linked into you know multi-factor off, and you know, you could potentially do device attestation into the VPN as well. Why so explicitly rule out VPNs as opposed to ruling out the architecture?
SPEAKER_02I really don't think that this memo rules out the kind of architecture you're talking about. There's I don't think an agency would be constrained from pursuing that, right? In fact, when it comes to like how folks do whether they choose to do network segmentation, whether they choose to do sort of identity as the boundary, anywhere in between, some combination of them. That's something where the agency needs to, you know, actually come back with a plan for they're gonna do that. The purpose is to credibly isolate your system so that there is not on whatever on whatever layer you're doing it, there's not a uh hard outer shell and chewy center. Now, there's also there are some things in there, and if and specific. About you know making a system internet accessible that is not today. By doing that, that may not be what an agency ends up doing for every system they have, right? But the capability of being able to do that and and operating securely to government standards, the system is going to be very helpful for people. I really do. And I'm I'm totally comfortable saying this publicly, right? It's it's there's all sorts of ways you can go about securing organizations and systems, but it's very important not to overrely on any on something to the point that you have a false sense of security about what you're doing, because you know, it seems to make a lot of the problems in front of you go away. And and there is a degree to which network level security has played that role in the federal government and something where we want to challenge that to to make sure that whatever state we arrive at, and I do think there will be a variety of states, and we we're explicitly litigators pick that flexibility, that it is different from what's there now, and that if somebody breaks into whatever kind of network or hybrid or or you know, whatever system is coming, that we have meaningfully constrained that impact and that we tore ourselves up inside to do that, which is probably what it takes, almost no matter what approach you take.
SPEAKER_01That's very cool. It's it's my happiest possible reading of what you guys are going for with the OMB doc, right? It's pretty subtle what you're doing with the uh the taking a host and making it internet accessible. Like the really blunt force read of that is where we're going, it's every single system is going to be internet exposed. Like the Redis instance that's sitting behind this Rails app will need to be internet exposed and authenticated, which doesn't make a whole lot of sense. And it is a it's an easy pop shot to take at the document. But but but you have to go through the exercise with something just so you have that you've exercised the muscle. There's like a like a Cast Sunstein nudge thing happening with that, where if you're just getting people to think properly about how to do this. I think if you look at like message board conversations about the OMB memo, if you had said modern VPNs are, you know, have identity-based barriers, you could do device attestation, you could do fine-green authorization for applications, and like the the the dumb message board response would have been no, the OMB memo says you can't use VPNs anymore. And it's pretty clear from how you respond to that that that's a misreading of the memo, right? It's a lot more subtle than that.
SPEAKER_02I mean, it is. It is more subtle than that. And to the to the point that we talked about earlier, the government is is quite a big place, and it's very difficult to make binary statements and to set binary rules for things. But it is also very important for something like this. There is a big shift, right? So, for example, there's another part of the document where we talk about reasoning through, as I just mentioned, on network visibility, the tension between attack surface and visibility.
SPEAKER_03Yep.
SPEAKER_02We specifically are not banning break it inspect or requiring break it inspect, right? But because we are choosing to say something on it and to not say that it's one thing or the other, like it's e you know, sometimes people do read it as, you know, saying something black and white, where what it's really trying to do is make sure that the conversation around it and the policy choices and the operational choices people make are more well reasoned. And I realize that's very hard to get across in on the internet in general and in really any particular document, but that you know, that's the mentality that we are trying to take with the with some of these items.
SPEAKER_04One thing I wanted to touch on that's related to like things again with like nudges is the uh facus on moving to non-officiable two-factor for anyone that has to authenticate against these services. And you explicitly you mentioned this before, but you explicitly government people are more used to p uh PIVs, like personal individual identity verification credentials. Like you can stick a card into a thing and that's how you you actually have a key. I'm less familiar with this. Is that is that about right to authenticate?
SPEAKER_02Well, so PIV itself, right, is a protocol, and there's also the card form factor and piv piv is a few things. It's the protocol, it's also related to the the sort of background investigation process that goes into ultimately issuing you that, right? And so there's there's this hard identity, a token that has a whole bunch of interesting cryptographic properties that also tied to something that the government did that it cares about about you. And so, yeah, the way that PIV, the protocol, has you know, functions in practice is through mutual TLS, is through client client certificates.
SPEAKER_04Um we talked about that.
SPEAKER_02You know, and one of the main things that we are emphasizing in this that I've talked about quite a bit is really recognizing that that approach is for one, it is fishing resistant. It is, and it has been for a long time, and it is a way in which the government was a little ahead of its time, but it also has problems in a lot of environments. And there is there is an in there's a bit of a brittleness to it that doesn't survive in a lot of the environments that we work in. And so one of the things we're really trying to do is there is when it comes to MFA, this is one of the more strict and specific parts of the memo, is that we are really trying to, from a policy perspective, close a door on some older MFA methods in enterprise settings, not in public-facing settings, but in enterprise settings. Yep. But we are also really trying to explicitly open a door besides PIV protocols to do this authentication of the government, which is not something agencies have been told before.
SPEAKER_04I see. Yeah, and it explicitly says like we uh support things like Web Auth N, Fido 2, which is if you use your you know Yuba key in a browser to log into Google or other, you know. I think there's like a ID.gov that does this as well.
SPEAKER_02Do you mean login.gov?
SPEAKER_04Yeah, login.gov, which is cool. I logged into login with.gov with my YubiKey, and then I never did anything with it again. It's nice that it exists. But for like contractors and other things like that who don't have a PID, you can explicitly enforce a phishing resistant two-factor to support them logging into your application or something like that. My question is, do you think that enforcing things like web off end and explicitly discouraging more fishable forms of two-factor like codes or that you have to input into your browser will be easier to roll out when you have a workforce that is generally they understand what PIV is and you have to have a physical credential and you have to present that. Do you think that'll be easier to do? Because in the general user base of people who use internet services, it is a like low, less than 1% adoption rate amongst, you know, huge web services that support things like WebOps.
SPEAKER_02I think there I could see it working out both ways. And so yes, people are more used to carrying tokens, but they're also very used to like a specific way in which those work, which is the card format. But I think what you're talking about here is the fact that there is a greater ability like you can't you cannot enforce this uh in the general public, right? That's right. But in an enterprise setting where we expect there to be the ability to to purchase things for people as you need, to train them. There also is just it's a policy, you need to do this, you need to you need to get it's not that that it makes it easy, because it isn't, and there are absolutely all sorts of you know security and usability problems with you know, like in the government. People have issues with their security team all the time when things get too tight. So I'm not saying it's easy, but in the in an enterprise setting, this seems like something that we can get done, particularly when we are when we have piv to start with, which is a just a working, workable option that is deployed widely in the government. And in a lot of ways, in these enterprise settings, we're talking about filling the gaps that PIV is not the right choice or cannot fill. Got it. I also should mention, by the way, in this context, is there's the part of the final memo here as well. There is some references to NIST and what NIST does with the derived PIV spec, which is the derived PIB being how you sort of can create other tokens that are related and derived ultimately from your PIV card. That has often been used in the context of being able to log into things with your phone, right? Of like blessing your phone cryptographically. But they anticipate, and you know, it says this in the in the memo, updating the derived PIV spec to accommodate other kinds of authenticators like this. And so, like over time, a lot of this stuff you should expect it to become more harmonized.
SPEAKER_04Nice.
SPEAKER_00Cool. Other question about the nitty-gritty details of the memo. I noticed on the first page there was a footnote that said agency is defined as in this other memo that was written. And I was curious if that's in all government memos in the same way that like the words must, should, shall, may are defined in RFC 2119 is at the beginning of every RFC. Is this the government starter language?
SPEAKER_02I mean you absolutely will find people referring to other canonical definitions in law, circulars, other memos. That is a common feature of government documents like that.
SPEAKER_00Okay, so we've been all over in terms of topics within the memo. Maybe we've been we've we've been a little combative unintentionally. I want to be clear that I think this memo is like really great, and I think this is a really positive direction to be going in. And I think a lot of other people think that as well. And so for people that want to be a part of this, can you pitch them on working for the government or where in the government they should go to work if they want to work on this type of thing and like how to get started in that? If if someone wants to do technology work in the government.
SPEAKER_02Yeah, absolutely. And look, for one, I don't feel this has been combative at all. I really, I really appreciate actually you digging into this memo. For one, a lot of folks out in sort of the you know hard tech and security space do not read government PDFs about cybersecurity policies or say the word cybersecurity at all. So I really appreciate that you did that and invited me on to talk about it. And in general, there are many places where one can come into government as an engineer, as a security professional, as a technologist, uh, as a designer, as a user experience person, as a content writer. Anything that you, you know, where you just have a craft of wanting to have things be delivered better for people, there are at this point a very healthy number of places around the government to go and that are ready to help you do good work and give you a place where you're like your technology is good, or at least is reasonable, and where you know where you're around other people like yourself who have, you know, an understanding of how to connect where technology and sort of design and and development practices are going and bring those and put them to bear in a public service context. And you know, that wasn't as much the case a decade ago, but a lot has happened in 10 years. And I came into the government through a team called 18F, 18F, that started up in 2014, started up right around the same time as the U.S. Digital Service started up. They're a little bit like sister organizations, lots of cross-pollination and collaboration between them. U.S. Digital Service is actually a neighbor of us in OMB and is sort of you know engages on a whole bunch of different initiatives for a whole bunch of different reasons. And then 18F is a consultancy that operates out of the General Services Administration that ultimately led to a bunch of interesting services. You mentioned login.gov earlier, which was a collaboration between 18F and USCS some years ago, which has matured over time, has you know tens of millions of user accounts doing Web Authane and MFA and you know doing cool stuff. And I really, you know, one of the experiences for me that was really profound, because I was somebody who found the government, it just I just assumed it was stoltifyingly boring inside. And when I came in, and also the you know, I had never worked for an organization larger than 50 people. So also the idea of bureaucracy in and of itself, no matter where I was also very intimidating. And one of the things that was really affecting was for one, there's a much bigger community feel than you would expect once you're inside the government. There are communities of practice that span across all these different agencies that operate very fluidly. There's actually a surprising amount of trust at the working level, and all sorts of interesting collaborations like emerge from that organically. It's not all top-down at all. And I know my my experience there, not just on the projects that I saw turn into things like login.gov or the US web design system or all these other interesting things, I found not just me, but many people around me were able to actually work on and influence policy. And just also gaining an understanding for what the government is on the inside, which is of course like just a bunch of people doing stuff and trying to do the best they can in a in a bunch of different interesting situations. You know, that was the a lot of that was surprising to me and something I really come to feel like government is is sort of, you know, the default thing for me now. And I definitely would not have expected that at all. But it doesn't even matter whether you it'll become something that you want to spend tons and tons of time in. Really just any kind of experience you get in the government at all, I think is a profoundly eye-opening thing. Helps you not only read the news better, but helps you reason about how organizations function, about how human incentives work, and just about like how complex every single thing is under the hood. And especially an appreciation for what it means when you're not allowed to say no to a class of users. That is one of the things that makes the government profoundly different than anything out in the private sector, is you don't get to pick and choose what segment of the market you go after in a lot of cases. In almost every case, you are if you're doing something interesting in the federal government that you have no about today, it's because it serves the general public. And you have a particular obligation to make sure that you do that. And obviously, there are all sorts of policy things that get tied in. Like people have expectations that the government will be excellent on privacy, excellent on civil liberties, excellent on security, excellent on usability, and also assume that these are big powerful organizations that have everything tied up nicely in a bow and run just like a well-oiled ship all the time, despite all evidence that uh that you could see that it, you know, this stuff is hard. So I really do encourage folks to get some of that experience. And there's a lot of people that have done that over the last decade, whether for six months, two years, or the whole time, and it is a really positive experience. And almost nobody in that is doing technology in the government that that I have come across wears a suit and tie like I happen to be doing at this moment. Do not be fooled.
SPEAKER_00So where should so GSA 18F good places to actually apply if you're just looking to do technology and government? Like where should you be sending your resumes?
SPEAKER_02So look at GSA. You know, if you go to join.tts.gsa.gov, that's the technology wing of GSA. They do fantastic work. USDS is hiring our office uh hires as well. And we, you know, we and a lot of other places also just publish stuff on USA jobs, which is the government's hiring place. Some of those postings will look very intimidating and government-y, and sometimes you might need somebody to like explain to you what is what does that mean? But that a lot of really great, interesting jobs get posted there. And I would also say at this point, too, like if you look around, I I bet you somebody you know in your circles in the technology world has done work in or around the government. You probably don't have to make too many hops to find somebody who can, you know, point you to something and help help explain what's going on. I guess I'll also I will make a brief plug for the Consumer Financial Protection Bureau, in part because they started a decade ago and actually really prioritized technology up front in a way that was very impressive. And, you know, people don't always expect regulators to be prioritizing that in the same way. And and they're a good living example that you can. So those are those are a few places. But uh there's good work all over.
SPEAKER_00Awesome. Thank you for coming on our silly little podcast. To all the listeners out there, we encourage you to both go to USA jobs and find a way to work directly for Eric. And and then once you finish doing that, we're gonna want to go to merch.securitycryptography whatever.com and buy a mug.
SPEAKER_04Or a hoodie or a t-shirt or a sticker with your hard earned government money.
SPEAKER_01Eric, thank you so much for doing this. This is super interesting.
SPEAKER_04Yes, thank you so much.
SPEAKER_01Thanks so much for having me. This is great.