Security Cryptography Whatever
Security Cryptography Whatever
Alex Gaynor
We chat with friend of the pod and special guest Alex Gaynor, former deputy chief technologist at the FTC and all around good Security Person™. Join for nerdery about WebAuthn, stay for accidentally melting down GitHub APIs around November 2020!
Watch on YouTube: https://www.youtube.com/watch?v=gBoGvyvsSi4
Transcript: https://securitycryptographywhatever.com/2025/08/16/alex-gaynor
Links:
- https://knowyourmeme.com/memes/no-take-only-throw
- https://alexgaynor.net/2025/jan/13/challenges-funding-open-source/
- https://alexgaynor.net/2025/apr/08/putting-a-price-tag-on-open-source/
- https://dadrian.io/blog/posts/corporate-support-xz/
- https://alex.github.io/nyt-2020-election-scraper/battleground-state-changes.html
- https://github.com/alex/nyt-2020-election-scraper
"Security Cryptography Whatever" is hosted by Deirdre Connolly (@durumcrustulum), Thomas Ptacek (@tqbf), and David Adrian (@davidcadrian)
Hello and welcome to Security Cryptography. Whatever, my name is David.
Speaker 2:I'm Deirdre.
Speaker 1:I'm Thomas.
Speaker 2:Yeah.
Speaker 1:And with us today we have special guest Alex Gaynor. How are you doing, alex?
Speaker 3:I'm doing great. I'm excited to be here.
Speaker 1:Thanks for having me, woo alex, I would say, has been um a long time listener, but I'm pretty sure he's too smart to actually listen to us. But we have wanted to get him on the podcast for a while to um talk about a number of things um, and we conveniently find him at a break in his career, and so I thought now would be a good time to talk about why in the world, ftc would want to hire some sort of technologist.
Speaker 2:So why don't you?
Speaker 1:say maybe a little bit about yourself and how you ended up at FTC.
Speaker 3:Yeah, so I am by background kind of we'll call it mixed specialization software engineer, and from 2022 to mid 2024, I worked at the US Federal Trade Commission as the Deputy Chief Technologist, ncs. The shortest answer, maybe, is that I ran into somebody who'd been a colleague in my first stint in government back in 2016 or so, and she had just become the chief technologist at the FTC and, perhaps knowing me better than I know myself, said you've got to come here, like I think you will enjoy this. This is like an opportunity to do like very interesting things. You'll be good at this and I had kind of just wrapped up a project I was working on, so I needed to find something new to do anyways, and I jumped in.
Speaker 4:So your first time in government.
Speaker 3:No, so my first stint in government was 2015,. 2016,. Right kind of at the founding of the US Digital Service, which was the group created kind of after the founding of the US Digital Service, which was the group created kind of after the healthcaregov crisis and recovery efforts with basically the mission of bringing more folks in who had kind of hands-on, particularly private sector technology experience to help the government deliver and the services it delivers and the kind of services it operates better. I took a couple of years away from the federal government in between and then came back in 2021.
Speaker 2:And the digital service did like logingov and things like that. They did other things, but that's one of the things I'm like ooh, this is nice.
Speaker 3:Yeah, logingov was a joint project between USDS and sister agency 18F.
Speaker 1:What is the difference between USDS and 18F, because I see them both on GitHub.
Speaker 3:Yeah, so I mean you use the present tense is, which I think is much harder to speak to.
Speaker 4:Okay, Because DS has a new name now.
Speaker 3:Yeah, in the past tense. The difference I would have said is 18F overwhelmingly uh works on kind of greenfield projects that they are brought in and asked to work on, whereas usds gets sent to places to make sure kind of in progress projects become successful.
Speaker 2:So this is sometimes described as 18f is the vampire and usds is kool-aid man okay, vampire okay vampires have to be invited into your home I see I was like they suck the blood out of some executive branch place.
Speaker 3:Oh, that's very they have to be invited in, whereas, like the kool-aid man slams down the wall.
Speaker 4:Oh, okay, am I crazy or did ds happen because of the obamacare rollout? Yes, yes, yes.
Speaker 3:There's this kind of healthcaregov catastrophe, a bunch of kind of folks with private sector SRE experience come out to help put that together and then, at least in the pop culture, telling the president says why did none of these people work for me? And this is kind of the opportunity for folks who were already in the government, who had been trying to get this thing off the ground, like Jen Polka, to say yep, we've got the plans ready, to go Like just give us permission to start hiring.
Speaker 2:And was that like the? We can talk more about this. Is that like the root cause of like? Why healthcardgov, the like first version was basically a shit show because it was had bad like site reliability, engineering and load balancing and handling of traffic and all that sort of stuff.
Speaker 3:I mean root cause is maybe it's hard to like. You know, if you did like the five whys exercise, you'd be like 15 whys deep and probably still not a singular root cause. But to the extent there is a single cause. It was that it was a system designed by dozens of contractors which had never processed so much as like a test transaction before it went live. It had never been load tested, it had not really been. It had been developed the way you might develop like the landing page for some subcomponent of the government that like posts a website with like news announcements and like bios for its executives, not like the way you would architect like a consumer service that like does many transactions per second this, this hates, this hurts my soul yeah, just given the timing, I would have assumed that that would have been like one of the first times the you knowS government was ever in a position to like stand up, you know, a continental scale SAZ, basically.
Speaker 3:Yes, in many respects it's kind of the first one, and particularly the first one, that has like that degree of public attention on it, right, despite Obamacare becoming law in, I want to say, 2010 or 2011,. This website doesn't go live until 2013. So, like you know, it's had this really extended rollout period in which healthcare you know the ACA is becoming more controversial. There's just a lot more interest on it. Incidentally, the website launches kind of during Ted Cruz's filibuster that shuts down the government. So first people don't notice that the website doesn't work and then the filibuster ends and, like, the website is still not working.
Speaker 4:The chronology of this is kind of breaking my brain, and like I forget that healthcaregov didn't roll out until after the reelection. Yeah, anyways, ftc versus US Digital Service what was the?
Speaker 3:working environment. Yeah, so I mean at USDS, the thing you were being tasked to do was go work with different government agencies to help them deliver a software project successfully. You know, taking kind of the combination of what you learned about what the government is trying to execute on what the policy goal is and what you know about delivering successful technology projects, mostly from a private sector background, and which you know about delivering successful technology projects mostly from a private sector background. In contrast, the role of a technical person at the FTC is not to help the FTC build software. It is to help the FTC's leadership and attorneys and economists understand how technology works and sometimes kind of the dynamics of technology markets, in order to execute on the agency's mission, which is consumer protection and competition law enforcement.
Speaker 2:Can we talk Oracle v Google? Were you there for Oracle v Google?
Speaker 3:I was not at the FTC for that. I was very much following Oracle v Google and in fact I organized an amicus brief of former government software engineers and for context.
Speaker 2:This was all about Android and Java APIs and we all I think we technologists became very much fans of the main judge, who taught himself Java and programming and had informed opinions about, you know, being able to copyright and sue over. You know the API for an ad function or something like that. But yeah, not that one, but other cases. Is there a favorite FTC-related technology case or thing that you worked on while you were there?
Speaker 3:So I mean, it's always hard to kind of say favorite. I could pick maybe three favorites. One that I particularly enjoyed that is maybe on topic for this podcast is I worked on a case that the FTC ultimately settled against MasterCard, which was about the enforcement of something called the Durbin Amendment, which basically says if you have a debit card in the US, your issuer has to make it possible to process transactions on multiple networks.
Speaker 3:So like you have, a MasterCard brand debit card, but there's also a second network on that card, which is often referred to as the back of card network, and it has to be able to process transactions as well. And what the FTC alleged is that, basically, mastercard had made it more difficult to have transactions from a digital wallet like Android Pay or Apple Pay be processed on the backup card network because of how tokenization worked. You know, your Android or Apple Pay wallet doesn't actually store the credit card number. It stores a temporary token and there is effectively no way for a backup card network to translate the token into the actual card number to send to your bank.
Speaker 2:Oh, that's fascinating. So something that almost from my perspective, seems like a way to like separate responsibilities and make sure that something is not compromised, like if this was broken on your phone, someone could steal your credit card number, and so the way that this kind of works is like it's sort of ephemeral use, right, but that ended up being breaking the law.
Speaker 3:Yeah, I mean, it didn't have to be. So like MasterCard already had systems where for other forms of transactions, including tokenized transactions, backup card networks could call out to them and, basically, you know, get this transaction verified and get the actual card number sent to the issuer. The issue was, for certain Apple Pay, android Pay transactions. They did not, which you know. In addition to this law saying there has to be multiple networks, it said if you're a network, you're not allowed to make it harder for somebody to have multiple networks.
Speaker 3:And so I mean this is, you know, one of those cases that's on my short list of things were fun because you know figuring out, you know putting figuring out. You know did this violate the law? Involved a lot of learning about how debit card transactions are processed, going to kind of each step in the process and like asking people, you know, please explain to me merchant. You know stripe, uh, uh bank. Like explain to me how you understand how the data is flowing through the systems, and like you know finding write-ups of, like the cryptography involved in these systems, like what is a token? So like, uh, you know, it was just kind of a lot of fun for that reason I I didn't even know there was a back of card thing.
Speaker 3:No like so I mean, there's no reason to know, unless you are basically a merchant, and merchants like this because, uh, it gives competition between the debit card networks and that gets them lower rates yeah, and like everyone, everyone who has gone to a place and they, like, they're like no, we do not take american express because the fees are too high.
Speaker 2:Uh, can understand why, that you want multiple options to to execute a transaction because of you don't want to get locked in or you don't want to get locked out. Yeah, do we have an inkling of like how multiple implementations of this sort of solution, which is Apple and Google, arrived at the same solution, which is not well supporting the back-of-card network? Is it just like oh, we didn't realize and we copied each other? Or was there something more like let's see if we can get away with this until we don't have to?
Speaker 3:Yeah, so I mean, I think Apple and Google were ambivalent about this. Or was there something more like let's see if we can get away with this until we don't have to? Yeah, so I mean, I think Apple and Google were like ambivalent about this. They don't have a stake in this one way. But like my recollection slash understanding is Apple rolled out Apple Pay first and kind of got a handful of credit and debit card networks on and they kind of you know, they designed the interface, kind of got a handful of credit and debit card networks on and they kind of, you know, they designed the interface kind of between themselves and not kind of what MasterCard or Visa's relationship with other networks would be.
Speaker 3:I mean, it's also the case that these systems all look slightly different in different countries. Like it's there's just kind of different network layers in different countries. Like it's there's just kind of different network layers in different countries. Like I believe Canada has like approximately one network and like everything just goes directly to your bank, more or less. Okay, don't quote me on that, but that's my recollection.
Speaker 2:Cool, all right, so that was one. Do you have another? The other two of your top three?
Speaker 3:Yeah.
Speaker 3:So the other thing that I really enjoyed working on was the FTC does consumer protection cases, which include data security, which is largely, though not exclusively, kind of data breaches, and I got to work on kind of a series of cases there, primarily focused on kind of what the terms of settlements were so that when companies settle these cases, they basically agree to injunctive relief, that is to say, like relief that changes companies settle these cases, they basically agree to injunctive relief, that is to say like relief that changes their behavior, and that they agree that they will from now on run a reasonable security program that kind of effectively protects consumers' information and has, you know, they will study the risks and like implement safeguards they find necessary.
Speaker 3:But also the ftc will enumerate, you know, generally half a dozen or so particular safeguards, and I worked basically across a series of those to introduce kind of more modern best practices. Like you will not just have mfa, you will have fishing resilient mfa. Right, you're going to use something like uh, you know we don't write like web, WebAuthn or you know if you wanted X509 certificates, but like it has to be phishing resilient. There's a case that requires the company to manage their infrastructure with infrastructure as code basically, and you know, actually get code review on infrastructure changes, changes. It's kind of like a whole series of these kind of modernization things. I worked with attorneys to kind of not just get into individual cases but really get into the kind of toolkit for, like, when this is on point for the case. This is kind of the best in class thing we're looking for.
Speaker 4:These are like the commercial equivalents of consent decrees. Then they're like negotiated settlements, enforced by.
Speaker 3:Yes, you know the FTC occasionally litigates these cases cases, but the vast vast majority of them are settled between the ftc and the company do you feel strongly that those are being well enforced and well well followed in practice?
Speaker 3:uh, I think so that there's kind of a non-trivial lag time between when a thing ends up in a consent decree and when, um, when, uh, uh, you know you would actually find out if it's uh, you out if they're not actually following their obligations. But there was some good anecdotal evidence that people were paying attention. Just in the course of negotiating one of these, before the ink had been signed, one company was explaining to us that they'd purchased tens of thousands of Yubikeys. Yeah, one company was explaining to us. So they'd purchased, you know, tens of thousands of yubikeys. So, like you know, I think I the very least, at least that company got the message purchasing the yubikey is one well hey, like come on this is exciting, this is nice.
Speaker 3:Dare to dream. They actually deployed them.
Speaker 2:Yeah, okay, and what's number three?
Speaker 3:Number three was probably the FTC's lawsuit against Meta alleging unlawful monopolization, and that was particularly interesting. So that case is kind of a. There's like a ton of like deeply like technical economic questions about like what market like Meta's products are in. Yeah, but one of what are called, uh, if you're defending one of these cases, you can assert what are called pro-competitive benefits or justifications, basically saying, notwithstanding the bad stuff you accused me of, actually this was good for um, this was good for competition and in this case in particular, meta alleges that their acquisitions of instagram and whatsapp were necessary for them to be able to scale because of the kind of infrastructure that meta provided to them. Um, and so that was just like a ton of fun to like dive into.
Speaker 3:Okay, well, like what were what were these companies you know not now kind of a while ago? What were their pre-acquisition infrastructure like, what was their migration to Meta's infrastructure? Like you know what kinds of metrics exist and like. I'm a little limited in how much I can talk about this. It's still in trial now, so there will be more things in the news in the weeks to come about this. But yeah, that was just like a blast and like particularly Instagram's stack um at the time of the acquisition, and I mean through to today, as far as I know, is like python and django, which are really yeah holy shit which are like.
Speaker 3:Those are like I I'm mostly emeritus, but like I'm a I was a core developer of both of those at the relevant time. So like, yeah, it was like it's cool to like kind of be looking at the you know, probably the largest django website on the internet yeah I was on a team that did an assessment of that code base a long long time ago of the instagram code base
Speaker 4:yeah, this is like a thing I don't understand about. It's the thing I don't understand about the discourse about the, uh, the, the. The trial here is just like Discourse Instagram when Meta bought them is not Instagram now. Like they're different products, like there was like a nascent kind of social, as I remember there was like a nascent, you know, social network thing going on with Instagram when they got bought, but they are a social network now, right, sure, and to the extent that happened, meta made them that.
Speaker 3:I mean. So the exact facts of these are just like a huge question of case, like I think. What I would tell you is I've been sitting in court every day for like the past three weeks, kind of just listening to the arguments.
Speaker 2:Because you're out of the FTC now. So you're just a regular dude hanging out in court.
Speaker 3:As one does.
Speaker 3:As one does as one does, but I think who knows who you might see in that courtroom uh, I think the through line is basically meta was very much struggling with getting facebook to be popular on mobile, kind of in roughly this like 2011 2012 period. Like may recall, like they had initially rolled out an HTML5 mobile app that was like slow and like not a positive user experience, and so this kind of provided a window for Instagram to build an alternative social platform on mobile that was like really mobile first, that used kind of photos as the centerpiece of communicating with your friends and family, and meta acquired them in an effort to prevent them from um from doing that. I mean, it's interesting that you say like obviously instagram is a social network today, because a key thing meta argues in this case is that, yeah, people maybe think of it that way, but actually what Instagram is mostly is it's like a video platform. Like you go there to watch videos the way you might on TikTok or YouTube shorts.
Speaker 4:Well, that's not true.
Speaker 3:All right, well, you can let them know.
Speaker 2:Yeah, one of the things that I find interesting about the Instagram wouldn't be what it is today without Meta argument is that the advertising solutions that Meta brought to them, or then Facebook into Instagram, the platform seems to actually have like really mattered because, like you could make an argument that like Instagram would have been pretty big even if they hadn't been acquired, if they figured out their infra and they were able to scale and all that sort of stuff. But it does seem that that is one of the things that really seems to make a difference, at least from the business side and the user experience of the advertising solutions and stuff like that. So I find that kind of plausible. The other stuff I'm just sort of like, yeah, maybe.
Speaker 3:So kind of the ability to monetize via advertising is definitely also another thing meta has claimed as a benefit of the transaction. Um, it's, it's interesting. Kevin sistrum, who's kind of the one of the two founders of instagram was the ceo, testified the other week that, like they basically had like shown designs to the board for advertising on instagram pre-acquisition that look remarkably like instagram's ads today. Um, so it's it. My impression was, at least in his mind like they, they would have been able to execute on this stuff the same, uh, from a kind of consumer perspective, one of uh I think the ftTC has been pushing is that by kind of being owned by Meta and by being kind of together forming a monopoly in this what the FTC calls the personal social networking services market, what's called the ad load you know basically the percentage of items in your feed that are ads that higher than they would be able to, kind of independent services that have to compete for users more.
Speaker 2:What's interesting is that they also acquired WhatsApp and WhatsApp is part of this case as well and, like it seems from the testimony so far, like the founders of WhatsApp were like no ads, no ad injections ever. And like, so far that seems, the founders of WhatsApp were like no ads, no ad injections ever. And like so far that seems to have been the case, but not without, like the parent company of Meta, really trying to figure out a way to make that work, and it still hasn't worked yet, and so it's hard to like argue what the benefit has been for WhatsApp by being acquired by Meta.
Speaker 3:Yeah, that's a great point, because WhatsApp is a slightly different case.
Speaker 3:The FTC does not say that WhatsApp is a social network in the same way.
Speaker 3:What they say is a thing that we had seen happen at this time period is several mobile messengers in Asia had pivoted from being just mobile messengers to also being social networks so Kakao and Line and WeChat, all in different countries and basically Meta was afraid that WhatsApp was positioned to do the exact same thing because of their kind of fast-growing giant and fast-growing user base, and so they acquired them to prevent them from executing this pivot.
Speaker 3:And yeah, whatsapp founders I think the testimony is pretty clear would not have wanted to user base, and so they acquired them to prevent them from executing, uh, this pivot. And yeah, though, whatsapp founders I think the testimony is pretty clear like would not have wanted to like kind of do any features. They did not want ads, and then so part of the case is just like, well, it would. As an independent company, it would have had to make money eventually. Like, would investors have basically pushed whatsapp to monetize in some way? Right, develop new features, sell stickers, you know, probably roll out more social features, because that gets you more kind of time and app.
Speaker 2:Yeah, and like didn't they used to have like $1.99 a year or something like that. They had like a really really low service fee.
Speaker 3:Whatsapp in for indefinite messaging or data or whatever, and then they got rid of that yeah, it varied uh a bit by country. Like they, my impression is they were kind of constantly experimenting with it, but at least you know, in countries like the us they were charging, I believe, a dollar a year uh, and then you know, meta got uh kind of rid of that fee kind of across the board, which you know they claim is a benefit to consumers.
Speaker 2:So it is, and I wonder if this is also a function of the zero interest rate era that all of this happened in and if this were another time and place, that just would not be a thing anymore.
Speaker 2:But on the argument that the everything app future, that WhatsApp could have been a la WeChat and the others future, that WhatsApp could have been a la WeChat and the others, I've heard an argument, independent of the arguments made in this case, that the American tech app culture just completely splintered into you have an app for calling a car, you have your Uber, you have an app for ordering your food, you have an app for uploading your photos, you have an app for messaging your friends, you have an app for all the things, and that had already happened.
Speaker 2:And so, like, the American tech user on mobile platforms was just sort of like, didn't care about an everything app, like you might in another country, another market, and therefore, you know, maybe if you tried to make WhatsApp the everything app and tried to make X the everything app, it just wasn't going to fly because everyone already has their app for that thing and they don't care that much. Maybe you get some usage. But now I'm hearing about the argument is like well, it might have been, and Meta bought it and stopped it in its cradle. And now I'm wondering what could have been.
Speaker 3:Yeah, I mean, what could have been is kind of definitely at the heart of this case. It's interesting in a separate lawsuit the Department of Justice has against Apple another antitrust lawsuit. The Department of Justice alleges that like Apple has basically like worked hard to prevent everything apps because those would kind of compete with Apple's control of the platform. Fascinating, yeah, I am like less conversant in the exact market dynamics of WhatsApp than I am kind of the pure tech infrastructure pieces, but yeah.
Speaker 4:What is like, what is the FTC? What is the deputy? The deputy, you know technologist role at the FTC. With regards to these cases Like, what's that work like?
Speaker 3:Yeah, so I mean, as a deputy CTO, my job was kind of partially, like you know, management, leading the team, hiring kind of all that set of stuff, and partially being an individual contributor on these cases. And basically, you know, I'll focus on the IC part because the manager part's probably not so different than being a manager anywhere else. But as an IC on one of these cases, really it's your job to kind of understand what is the case the lawyers are trying to investigate or litigate, and how technology interplays with that and how you can use your understanding of the technology to better inform what the lawyers are attempting to do. So you know kind of you know, if we think about just the full life cycle of a case, kind of before an investigation has begun, you've got kind of what we'd call target spotting, identifying. You know things, you know whether it's in the news or in security research that may indicate possible violations of laws. So like you're kind of you know, consuming your tech media and seeing, hey, from what I know about the law, what are these might be of interest to lawyers. Then, as kind of a case is going on, you're kind of the first step is you, or an early step is. It's called a CID civil investigative demand, it's a subpoena Right, you're helping attorneys craft what are the types of documents requesting.
Speaker 3:What are the questions we're looking for answers to. And you know helping everybody use technically precise language so you're getting exactly what you want. Hey, we want this data from your issue tracker, not just your email inboxes, for example. You're helping review those documents. You know as you're getting. You know you've made a decision that you think this does violate the law. You know you're helping draft the complaint, making sure the kind of technical points are accurate.
Speaker 3:In a settled case, you're helping kind of identify what are the remedies you want. You know what are you asking for in the settlement. You're often kind of getting on the phone with the opposing counsel. Maybe in a security case you might be on the phone with the company CISO where they're saying, hey, you asked for XYZ, but we can't really do that for this reason. And you're saying, okay, if we tweak the language like this, would that work for you? Work for you? I mean you're just working alongside the lawyers really through the life cycle of the case to make sure it's informed by what the technology is, how the technology works For the meta case?
Speaker 4:what does the FTC's technical staffing of that look like? Not lawyers, but technologists that are actually guiding and reviewing discovery and things like that?
Speaker 3:Yeah, so there were at peak two or three of us on the case and you know every technologist on our team was on multiple cases. So and then in the meta case in particular, the FTC also has kind of two outside experts. They retained to expert. You know, testifying court will draft a report explaining their findings and kind of all that. You've written about going through this process recently, thomas. And so in that case there was kind of outside folks as well and so in META we were also supporting kind of the experts through the process.
Speaker 2:So do you get to like be one of the first people to like get discovery or to see some of the technical discovery? Do you get to see source code and other cool juice possible juicy things?
Speaker 3:I mean there was a ton of interesting things. I don't I mean, occasionally you see source code, like in an email or in a chat message. There's mostly, mostly not a lot of, uh, source code. Yeah, I mean, there's just tons of fascinating emails, um, you know, debating the pros and cons of migrating between different internal databases, and like, just like I don't, lots of like, just like I don't know if you're nerd about this stuff like lots of cool stuff nice.
Speaker 1:A lot of those like emails and stuff end up being public eventually.
Speaker 3:And then well, because there's that like that twitter account.
Speaker 1:That's like tech emails or whatever that just posts like random emails that have come produced by both the parties and third parties.
Speaker 3:In antitrust cases you have other companies that may or may not be kind of in the market also producing documents.
Speaker 2:Right.
Speaker 3:So you know you go to kind of the funnel of all right, which of these are at all interesting, which of these you know maybe support one side's case, and then which will actually get used with a witness. A witness, you know maybe will see like a dozen documents. So like you know it's a tight funnel. And then you know the public ones are obviously redacted. You know if there's competitively sensitive things or you know names and stuff like that. But yeah, so there's a funnel but like in some sense, the most interesting or most important ones, yeah, you actually get to see.
Speaker 1:And then was there anything that, like you worked on, and then it determined oh, I like, uh, you know, we want, like a technologist, to look at this. And then actually, you know what, like this, this behavior is fine. There's, there's nothing here yeah, I mean it's.
Speaker 3:It's difficult to to.
Speaker 1:It's much more difficult to talk about those because you know, you might want to change your mind later well, no, not even that's.
Speaker 3:I mean you think about this the same way, like a law enforcement agency does, like you don't talk about uncharged conduct because that's that's prejudicial, Right, if you talk about charged conduct because the other party gets the chance to defend themselves in court, they get the chance to clear their name, but if you talk about uncharged conduct, then like they never get to go to court and prove, like that I didn't do that bad thing. It's like you know I can't really talk about all the times we said we look, you know we looked at a thing and we said, you know, this doesn't seem bad to us. Or you know more, like working, collaboration with lawyers, we didn't seem bad. But yeah, you know, there's absolutely cases where, like a better understanding of technology or the norms or the market was like you know, we would not be able to bring this case.
Speaker 3:And when All right Is there anything else we should ask you about the FTC. No, I mean, I would just say like it was a fantastic job.
Speaker 1:I you know if you were the kind of person who's like interested in the law and also is interested in kind of like diving into different areas of technology, like is fantastic job nice so let's talk about open source, because I think there's a lot of things to talk about there starting with how you have at alex on github, which I think is impressive, because I I thought I was like somewhat early on GitHub and that I created at Daydream in 2010, I think. But I believe you have me beat there as well with just your first name.
Speaker 3:Yeah, I mean, I guess all of my stories start with, you know, having friends who like pointed out good things. But I had a friend who like very much pushed me. I mean I've been programming like a year at this point. It, who like very much pushed me I mean I, I've been programming like a year at this point it was like very much pushed me to like apply for the github beta. Oh so, like there was a beta. That was yeah, I didn't know, I think. So there was definitely a point where, like you had to apply to like get an account. Like you know, there's a wait list.
Speaker 1:So that's, that's how I got alex and I think most people know you or, if they don't know, you have interacted with you indirectly when they pip installed cryptography um, or perhaps pip installed certify um, or actually you know. I'm kind of remiss to say pip, because every time I look at the Python ecosystem the packaging story has changed. Maybe they're PyEnving something. It's all UV now right, I've never even heard of that.
Speaker 2:I've heard of that.
Speaker 3:UV is kind of the. He looks so alarmed.
Speaker 1:I just feel like every day I spend as a product manager I get a little bit dumber yeah.
Speaker 3:UV is good stuff. It's from the same folks who made Ruff, which is kind of the new, much faster linting engine. Uv is kind of a reimagined Python packaging experience. It's implemented in Rust. It's very fast. I like it yeah. I don't know what Ruff is.
Speaker 2:I don't write Python, no more. That's not true.
Speaker 1:I wrote some Python today let's loop back to Russ later, but first. So like, what is the cryptography package and why are you maintaining it? And are you a cryptographer?
Speaker 3:I am not a cryptographer. I am, you know, somewhere on the spectrum of like cryptography engineer, which is I am you know somewhere on the spectrum of like cryptography engineer, which is to say, like you know, I think mostly about you know, how are these primitives used? Like what kind of APIs make it more likely for users to use them successfully? Like kind of the ecosystem and API design side of things. Like I'm very fortunate that my co-maintainer, paul carer, has like a much stronger cryptography background and like he does understand what the lattice is, I'm pretty sure, whereas I do not, um, but so what is it? Do you really not know what a lattice?
Speaker 4:is no.
Speaker 2:No, he should know what a lattice is no, it's fine, it's like secretly, secretly the plan son yes, it's like here's a secret All the lattice-based cryptography that we're using does not actually implement any lattices. So you're fine, you're good.
Speaker 1:Outstanding. I think we've had people who've literally published at Crypto on this show who have been like I'm not a cryptographer.
Speaker 3:I am less of a cryptographer than they are, I'm prepared to say I am less of a cryptographer than they are, I'm prepared to say. But anyway, we aim to be kind of a general purpose cryptography library for Python. So the place you would go if you want to encrypt something with AES or do an ECDH key exchange or also kind of a lot of the file formats that exist in the cryptography cinematic universe, kind of a lot of the kind of file formats that exist in the cryptography cinematic universe, like X, five, oh, nine certificates, or like PKCS 12, like bundles, that sort of thing. And then we also, kind of under the same, we have this Python cryptographic authority brand. Under that brand we also maintain pineapple, which is a Python bindings to well, nowadays it's libsodium, but the library formerly known as DJB's NACL, not pronounced salt and we also maintain PyOpenSSL for our eternal shame.
Speaker 2:I didn't know this.
Speaker 3:Yeah, and as well as the Bcrypt Python library, but like cryptography is kind of like the flagship library for no other reason than it's got the obvious name.
Speaker 2:You're doing more than like I would take on like as a software maintainer, especially all the x509 stuff, and I think, like you, have some more stories about being motivated to re-implement how these things are implemented, partly because of some of that sort of stuff. Um, you're, this is a great service, because this is non. This is stuff that people doing software they're just trying to get the cryptography part working as part of the rest of their software project they have to interact with. And all of us highfalutin crypto people who just want to do some nice math and just make it go brr on the computer. They don't want to deal with the X.509 part. But the real people just trying to deploy some shit that uses cryptography they need to deal with. Deal with the x509 part. But the real people just try to deploy some shit that uses cryptography. They need to deal with things like x509. So thank you for helping do that securely pre-cryptography.
Speaker 1:Like I remember it was like circa 2011, like I don't know if you guys had started cryptography yet or if it was still in its early days, but like trying to just like make an https connection in Python was like fucking impossible and you had to like the wheels didn't yet or barely existed and so you like had to like apt-get install OpenSSL and like hope that PyOpenSSL worked and like probably got VCC errors and that was a mess.
Speaker 4:I have a question Much better now. Yes, I've been seeing this for like 20 years now. What's a wheel?
Speaker 3:Okay, so a wheel is. I've asked. Alex this before it's an. The name itself is an inside joke. It is a binary distribution of a Python package that is pre-compiled for some particular platform.
Speaker 2:Okay.
Speaker 3:So, like you know, you pip install cryptography on Mac OS Windows. You know you pip install cryptography on, you know, macos, windows or Linux. You will get like some so that we have already compiled for you and you will like not have to build it on your computer.
Speaker 4:It just means you don't need, like the depths to build the library.
Speaker 3:Yeah, particularly. You don't need the system dependencies, you don't need GCC Gotcha. And yeah, historically, compiling Python extension modules is a very fragile thing, and so it's a huge performance and user experience boon to the vast majority of users Separately. I think there's a big security question in shipping, so's it's effectively unauditable that they match up with the original source, but uh was there.
Speaker 2:Was there a I might be confusing a debate about this and create stud io and rust land and uh, in python land? Was there a big debate about this recently, or am I just I think, was it for cryptography, for python cryptography?
Speaker 3:uh I don't I don't think there's been a big debate in the python, like the python community has been pretty accepting of these. Uh, there was a fracas in the rust ecosystem yeah, I think that's a year or two ago because, uh, I mean, rust does not have pre-built uh binary distribution as part of, like, the cratesio ecosystem. But there was a package that like, basically like in their source distribution was like I have just put a dot so in here and I will use it if it looks like the platform is correct for it.
Speaker 2:Yeah, as a performance optimization.
Speaker 3:Yes, there's much gnashing of teeth.
Speaker 1:Well, it violates their security model that if all CPU cycles are spent on the compiler, then none of it can be spent on an unsafe program. Penny.
Speaker 2:I have lots of other formal methods languages that spend way more on CPU than Rust C. I will not take the slander but wait a minute.
Speaker 1:What was the inside joke about the name wheel?
Speaker 3:So most Python naming is downstream of like Monty Python, so PyPI the. Python Package Index, which was originally known as the Cheese Shop, and you get wheels of cheese from the Cheese Shop.
Speaker 2:Okay, I thought everything was. Someone just named something Python, and then everyone went with the snake motif. But no, apparently not. No, it's Monty Python.
Speaker 4:What's funny is I was exactly this many minutes old when I got the Monty Python to Python thing just now. Oh, that's why they're doing everything with Monty Python.
Speaker 1:Yeah, oh, me too.
Speaker 4:It all makes so much more sense. It's the worst.
Speaker 3:Happy to be of service. Happy to be of service.
Speaker 2:Okay, so we've touched on this, but when you started PyCryptography and took over some of this stuff, almost all of this was Python wrapping C right or calling out to C, and now a lot of that has evolved away to a lot of Rust right.
Speaker 3:Yeah, so when we started cryptography back in 2013 or so, it was basically all wrapping what we called backends. At the time, we had this idea that we'd support multiple backends. So for a while we supported OpenSSL as well as CommonCrypto, the Apple system cryptography libraries, and yeah, those were all kind of binding between Python and C, so it was technically all Python code, but it was the the kind of Python code where, like, you've got pointers and can have use after free vulnerabilities. And then I think it was back in roughly 2021, we kind of took our first steps of diving into porting some of that kind of binding code from kind of Python C hybrid to Rust. And nowadays we've kind of we dropped the common crypto, so we only have OpenSSL as a backend and we but now the chain looks like you've got Python calling into a Rust extension module which uses the Rust OpenSSL Rust library, which you know still binds to OpenSSL.
Speaker 3:At the end of the day, for all of the kind of things, we think of it kind of as cryptography. So AES, rsa coming soon, mlchem, you know that family of stuff, but a lot of the things that we think of as more like file format, parsing, like parsing X509 certificates, a lot of that stuff we used to use OpenSSL for, and now we just have our own Rust implementations of that stuff we've we used to use open ssl for, and now we just have our own rust implementations of that we've taken it, including like parsing, you know, keys from pen piles and things like that.
Speaker 2:We've taken that away from open ssl nice, okay, because for some reason I thought you had like literally like we're shipping all of it and not just linking out to to open ssl for those parts and rewritten all of but a pile of C with a pile of Rust. It's like, okay, well, we wrote, like some of the like most gnarly surface area that we control with some Rust. So okay, I get it.
Speaker 3:Yeah. So I mean we have a lot of you know, tens of thousands of lines or maybe 20,000 lines of Rust code these days, so it's non-trivial. But we're still reliant on kind of open SSL and we will also support LibreSSL and BoringSSL and AWSLC, but for now kind of all of our official distributions, and I'm sure 99% of our users are kind of using open SSL?
Speaker 2:What about our Russells?
Speaker 3:Well, yeah, so what about something else? So I think we have a lot of interest in what a post OpenSSL world looks like for us. You know, rustles itself is, I mean, it's mostly a TLS library, so we need to use kind of the underlying cryptography library, whether it's, you know, it's, a Rust crypto or AWS LC or something like that. We have a lot of interest in that, in what something like that looks like. We're not super happy consumers of OpenSSL. There's a lot of logistics between here and there and particularly figuring out how to manage the conflicting needs of many of our downstreams. So using OpenSSL as the lowest common denominator means, you know, aws can just drop in AWS LC and Google will drop in boring a cell and red hat can use it with the system open SSL. They've like done FIP certification on, and so you know, kind of any move we make away from this is going to be disruptive to somebody. So we have kind of the question before us of how do we come up with a plan that mitigates that.
Speaker 3:Yeah, is there any notion of upstreaming the 20,000 lines of rust? That you guys wrote Upstreaming to who. Yeah To OpenSSL.
Speaker 4:Right. The path that you guys are cutting through OpenSSL is disproportionately the stuff that people care about, right Like. The reason you're doing this is because people make TLS connections from. It's what everyone uses OpenSSL for.
Speaker 3:Yeah, well, I want to hang like a little asterisk on that, like in the sense like I do not think we have a good understanding what OpenSSL is. We have a good understanding what OpenSSL uses. Sorry, I'm sure by volume. Like the number one thing people do with it is like they use it with Apache or Curl to like make or like receive HTTPS connections, like I'm sure that just has to be right. I think by volume of participants in like the OpenSSL ecosystem. There's actually a huge amount of weight on like other use cases, just in terms of who participates in the OpenSSL development process. It's actually weighted, I think, probably far from people who just want to make TLS connections. But if the OpenSSL community were interested in upstreaming some of this, that's absolutely a conversation we'd add.
Speaker 3:Part of it is like we've made pretty different design decisions than them and it's not like it's not super clear how you map like a backwards compatibility path, like a fact of their X.509 and really ASN1 APIs in general is there's a lot of small allocations, like you have all of these subfields on an X.5.9 certificate and many different tiny strings. Each one gets its own heap allocation, whereas the approach we've taken is we just keep around a reference to the original byte string and everything is just pointers back into that byte string. We have a lot of confidence that's safe because Rust, but it's not super clear how you would navigate a backwards compatible path between those Got it. But we've structured most of these things as the standalone Rust crates in our source tree, including our X. We have an X.509 path builder that's implemented in Rust on top of these structures. Nice, if there was an interest, we could make these things reusable and designed for somebody else. They're kind of minimally structured for that now.
Speaker 2:I will say that the fact that you have your own, you have your separate X.509 parsing set and all these sort of things will be good, that the new MLChem and MLDSA certificate stuff that is coming out of IETF you get to make your own choices, independent of OpenSSL, about how you parse those, and that's a good thing.
Speaker 3:Yeah, I mean if you look at the MLChem design discussion issue we have open, there's pretty strong consensus. Obviously the only like formats you should part or parse are like seed only, yeah, no like expanded form, uh thank god, yeah a chunk of this code is not like all just buried in the internals of the cryptography repo.
Speaker 1:It's like alex slash, my super fun ASN1 parser, or something like that.
Speaker 2:It is a.
Speaker 1:Rust library. It's just the stuff to jam it, wrap C around it or wrap Python around it as well.
Speaker 3:Yeah, the kind of baseline dir library is its own Rust ASN1 library. It's on cratesio. That is designed for reuse and Signal uses it, which is one of the coolest things I get to say, oh nice, so yeah that part. But the part where we then use that to define X.509 structures is not really. It's not on cratesio, it's not structured for reuse at the moment.
Speaker 2:Okay.
Speaker 1:Are you getting paid for any of this?
Speaker 3:Early on, paul and I got a bunch of time for our employers to basically work on this, but it's actually been quite a while since either of us so much has used cryptography for our day jobs, much less got paid to work on it. So this has been for a while now, a labor of love. Well, a labor of love and a labor of ideology, in the sense that we've become a very popular thing and we take that seriously, both as an obligation to users but also as an opportunity to improve things. So we want to be the library that pushes forward what's possible and by being a very popular thing, we think we've made it materially easier for, like other projects, to use Rust and Python extensions, for example, by like being willing to take some of the user frustration and, you know, push through that.
Speaker 2:And I think you experienced a lot of user frustration. Was it when you imported a bunch of stuff to Rust and you had to like basically drop support on certain platforms or something like that, with a major version or some version upgrade and they were like this doesn't work anymore on my extremely obscure platform that hasn't updated Python in 15 years, and you had to just sort of be like we're sticking to our guns, folks, it's going to be better for everybody just sort of be like we're sticking to our guns, folks, like it's going to be better for everybody.
Speaker 3:Yeah, when we shipped the first release that contained any kind of Rust build integration at all, that was definitely kind of a point of frustration for a lot of users and like we get why, like people had the experience of breaking overnight, there were also like subtle things like an assumption that our versioning scheme matched semantic versioning, which it didn't. Our versioning scheme matched semantic versioning, which it didn't. So there's kind of a lot under the surface there. Since we worked through that feedback, in particular, alpine Linux distributions got wheel support.
Speaker 3:Oh, that did not exist at the time, so that was probably the biggest cohort of users who went from this pip installs correctly to what is this nonsense about? I don't have rusty on my path, yeah uh. So like, since support's uh been introduced for alpine wheels, um, you know, I think experience overall has been better and I I think there is a recognition that kind of more marginal platforms. Your ai xXs, your PA risks, you know, to a certain extent, like if the maintainer of your platform is not willing to invest in it.
Speaker 3:There's just, you know you can't really be expecting open source maintainers, who have you know, never so much as seen a system running on your CPU or your operating system, to be the ones to do the work that you know, IBM or whoever isn't?
Speaker 2:Yeah, what's your like? It's just the two of you right Like, what's your CI infra? Like to try to maintain support on such a wide platform as or does PyPy take care of a lot of that for you?
Speaker 3:Yeah, so we are huge believers in CI. We often joke that we're mostly a CI engineering project that happens to have a cryptography library. So we run nowadays we're kind of exclusively on GitHub Actions, entirely on their hosted runners. But we have I think it's roughly 60 builds in the matrix of different Python versions, different OpenSSL versions, different Linux distributions, windows, mac OS. It's mostly just kind of a big matrix and the worst part is we have lots of things that we would like to do that we haven't done. It would be very helpful to run builds in Intel SDE, which basically lets you simulate different CPU feature sets and make sure everything is correct on. Do you have AVX 512 or not, all of those variations. Part of this is we just think we're responsible for the end product, so we don't take for granted that OpenSSL has implemented the primitives correctly. We take witchproof with this collection of test factors and we run it against every one of these build matrices to make sure we're trying to do our best to do right by our users.
Speaker 4:One of very few projects ever to have operationalized witchproof.
Speaker 2:That's not true. There's a lot of projects that have operationalized that you just might not hear about, and it might be just one or two of the test vectors. That's actually interesting, because what happens if your connection to your dependency, which is mostly OpenSSL, breaks something and your tests against it don't pass? Tests against it don't pass and, like you, just turn off a feature or you, you, you ship, uh, you know it's like this has got broken or you don't like, what do you do?
Speaker 3:yeah, so you know, say new release, open ssl comes out, it has something broken. This happens significantly less frequently these days, if for no other reason. We we have kind of, uh, a bot that has us testing nightly against, like whatever the latest uh kind of head of open SSL is oh good, okay, okay.
Speaker 1:So like are you basically running CI for open SSL?
Speaker 3:I mean they have, you know, a non-trivial matrix of their own. I believe we are, we're much, I think we're very disciplined in particularly right Adding regression tests, which I don't think open SSL always is. But and it's you know, there's a little bit of a snake eating its own tail, because they also run our tests in their own CI, but only on one platform. But yeah, you know, in a situation where, like you know, we identify breakage, like first thing is, we're not going to ship a release. Our wheels statically link a copy of OpenSL, so we're not going to ship a release with that, we're going to report the bug to them, we're going to try to get some action taken. If a bug fix release is not forthcoming, or something like that, we might take the step of, you know, kind of pulling, you know, can we detect the circumstance of this bug, kind of add our API layer and, you know, take some sort of corrective action or error out early.
Speaker 2:That's good. So you have a early detection and constructive relationship with your dependency and somewhat vice versa, so you're able to avoid this as much as possible. That's great, and now I'm looking at your CI and your custom runners and all of your infrastructure on GitHub, because now I'm Historically it was a lot more complicated.
Speaker 3:Before GitHub had hosted ARM64 runners or before they had M1 CPUs, we ran our own. For a while we were running our own Kubernetes, cluster of ARM64 Linux to provide runners.
Speaker 2:And a Mac mini in the closet or something.
Speaker 3:Yeah, and then at various other points we've had you know kind of other hosted CI providers, but for the last couple of years we've settled on GitHub Actions.
Speaker 2:Yeah, they pretty much have everything you need. It's pretty awesome.
Speaker 3:Thank you, github in front, yeah, yeah, and they graciously provide us, you know, an increase on the standard, uh, amount of like free concurrency. So we're really we're very yeah, otherwise I think, running kind of 60 concurrent builds on every um it would cost you a lot of money.
Speaker 3:Yeah, well, less, so the money and more, just like it would give us such a long feedback cycle on pull requests. Okay that I think it would. Um, give us such a long feedback cycle on pull requests that I think it would really hamper kind of motivation and productivity.
Speaker 2:Yeah, so instead of you getting rate limited on the regular rate limits, they're just sort of like letting you not get rate limited.
Speaker 3:Yeah, I don't know what the actual threshold is, but we have higher than normal pre-concurrency.
Speaker 2:That's awesome, so that sounds like a nice perk that you don't have to pay for yourselves to run your open source project. Yes, do you have opinions on paying open source maintainers to make sure that their open source projects stay maintained?
Speaker 1:I in fact heard that open source added over a trillion dollars of value to the economy from Harvard Business Review recently, yeah.
Speaker 3:So yes, I mean I do have some opinions on this. I guess I have far fewer opinions on what if somebody would like to pay an open source maintainer. By all means, I'm not here to tell you. I am much more interested in thinking about this from the open source maintainers because I've been involved in open source for 15 years at this point and more. For as long as I've been involved in open source for I don't know, 15 years at this point and more.
Speaker 3:For as long as I've been involved in open source, there's been this like open question of why are more maintainers not paid by the companies that rely on them? Right, there's, there's so much value creation. Why, why is so much open source happening, kind of in people's free times? You know, to some extent, you know, I think it would like time stolen away from like their other hobbies or their family and like so. That's kind of been the question for as long as I've been involved in open source. I think you've seen different kind of generations of thoughts on this. I wrote a blog post recently that tries to. I actually wrote two blog posts one that kind of tries to take a look at it from an individual maintainer perspective and the other that tries to take a look at it from more kind of systems economics perspective. But I think it's really important to recognize that very frequently we have trouble articulating what it is that paying maintainers would achieve for the payees paying maintainers would achieve for, like the payees, um, uh. And as a result of that it's hard to get people to give you money if, uh, you don't want uh, or if you can't articulate what it is.
Speaker 3:And I think sometimes this is desirable from an open source maintainer's perspective in this, in the sense that one of the things that comes with somebody giving you money is like probably a certain degree of expectations about what you will do. And for me at least, part of my enjoyment for open source maintenance is precisely that. It's much less constrained. If I am not feeling motivated, I don't have to look at my project for weeks on end. If there is a feature request that I'm not happy with the API for, I can sit on that till we find an API that we're happy with. Right, I have a lot of flexibility to prioritize how I would like, and for me that makes it more sustainable, like I think I'll be maintaining open source software a heck of a lot longer as a thing that you know competes with my other hobbies than I would if I was getting paid for it and feeling a sense of obligation. So I think there's a trade-off there that is sometimes unacknowledged, okay.
Speaker 2:I like. I like this perspective because I definitely feel those feelings of like, this is my project and I don't have I'm not obligated to, you know someone contracting me to do X, y, z or or whatever. I have more control and I have it on my time, um, and I'm working on it because I like it. But then we kind of fall into a little bit of the tragedy of the commons of like, you know, it's no one's responsibility to like make sure that this project that we all benefit from stays maintained and well kept. Do you have suggestions about that?
Speaker 3:I I I feel like I do not have a lot of suggestions about how to address that. I I think the biggest gap is to me is that I think we do not have a great sense of how to predict which projects have a risk to their maintenance right, which projects would an injection of you know, money into actually materially impact, how they're maintained? I think there's a tendency to kind of treat all open source projects as if they're kind of all very similar to each other, and I think it's mostly not correct. Like open source projects vary a lot. There's another huge problem which is for maintainers who have full-time jobs, an injection of small amounts of money. Well, like it may be, like very appreciative and like you can buy things with money. Like actually you can't. It's difficult to buy a lot of time with like less than full-time salary replacement levels of funding.
Speaker 3:And I think there is a huge unresolved question of like what should you do for an open source project that like does not you would never say this subject matter justifies a full-time person on it. Like maybe like a stable compression format, like that's probably not 40 hours a week of work, like what is the appropriate way to fund that? Um, like even like totally blue sky, you know no kind of ordinary commercial constraints, like what's the right way to to fund something where, like you wouldn't say this is a full-time person's uh work and you can't buy half their time because they have like a 40 hour a week job, like that. I think that pushes you towards uh kind of some of the all or nothing challenges that we have and I don't have a solution to that. But I think identifying the dynamics is is helpful, because I think too much of the discourse is like companies are greedy or companies are mean and less on, like what are these systemic factors? And like you know what? What are the actual incentives?
Speaker 1:Well on, like what are these systemic factors? And like you know what are the actual incentives. Well, we do have a solution to that that we can steal from paul graham, which is to fund open source and patches right. Like you can, instead of paying one person to maintain one project, you pay one person to maintain 10 projects oh um yeah like, and then, all of a sudden, you're running a consulting business which is why like I feel like this whole like conversation about open source is.
Speaker 1:It's like the no take, only throw dog meme. It's like you can't get paid for open source. You just need to find, like the open source projects that like are valuable about some amount of people and then like work on them in exchange for money with the people that they're valuable to. This is an option that is available right now to everybody and like we have friends who do this in various ways.
Speaker 3:It's just, yes, it's not every project all the time yes, and then you have, like you mentioned, someone asking shit from you and that kind of sucks yeah, well, I mean, I think it's important to distinguish between getting paid to work on the open source project itself versus getting paid to work in and around the project, like if you were the maintainer of a project and like you would like to get paid by somebody who, like, is a big user of it. Yeah, and you know, maybe part of your time will be like upstream bug fixes as relevant, but a lot of it will be like helping people at the company use it or, like you know, making the company more efficient at using it. You know, either kind of in-house or as a consulting like. There tend to be like lots of opportunities to like help somebody use python better, help somebody use django better, like you can definitely get paid to do that. I think of that as being like fairly different from like get paid to work on the open source project itself.
Speaker 2:But the help, the help use it side, as opposed to the like push commits to the project side.
Speaker 3:Okay, yeah, and like there are people who, like you know, square the circle by like they they charge people to like teach them how to use the project or whatever, and then they, you know, take their 30 hours a week of income on that and like, and they use their other 10 hours a week to to work on the project itself, like you certainly kind of do, kind of in between solutions like that, but um, yeah, I've I've seen, uh, what david mentioned, which is like, uh, basically a consultancy.
Speaker 2:But you like pay, you care about the Go cryptography standard library or these other maybe OpenSSL, but another library that people really care about. We don't just work on any one of those things. We work on an ecosystem of things so that it's not just one person getting a full-time salary or whatever to work on a single thing that doesn't need a lot of work every single week by people who care about the maintenance and that may care about like hey, we need this, we need MLChem in Go. Who's going to do it? Who's going to prioritize it? Who's going to propose it? Who's going to get it merged and code reviewed and stuff like that. Like that seems to be a at least functional paradigm and it's definitely not like give $10,000 every quarter to the you know, the go implementation of ML cam that lives on GitHub and like maybe the maintainer will have five hours every quarter to like I don't know upgrade dependencies or something like that every quarter to like I don't know upgrade dependencies or something like that.
Speaker 3:But yeah, it's like there's no one real paradigm that seems to be like well established, of the open source ecosystem. Yeah, and I think Filippo is doing particularly with Go, is doing a lot of work to kind of figure out the practical realities of doing I think what he calls professional, professional maintainership.
Speaker 3:So, like I mean, uh, his is the uh, I think experiment model that, um, you know, if it's successful, other people will see if it's applicable to their ecosystems as well. For for myself, like I said, I I enjoy these things being kind of unencumbered.
Speaker 2:Yes, me too, because I like my little repos that are just me and maybe some other people. And then all the other stuff involves setting up a consultancy or a company and all that crap, thomas.
Speaker 4:I don't know, Just speaking of encumbrance, right. So obviously one of the reasons we would be interested in talking to you besides the fact that you're one of our spiritual advisors is that four odd years from now, when President Pritzker is sworn in, you could be the chief technologist at FTC. Right, you could go back and do that for real. And my question is if you were in that role, if you were leading tech policy at FTC, right, Like you could go back and like, do that for real. And my question is if you were in that role, if you were, like, if you were leading tech policy at FTC, would you support a rule that says that you can't upcharge for security features?
Speaker 3:Would I? I mean, that's a great question. I love that question because I think it pits kind of two strong intuitions against each other. One is like the strong intuition that like we like security features and like we would like more people to use them and like charging for them reduces their usage, so like getting more people to have like SSO, for example, more of that will happen.
Speaker 3:Yeah, yeah, we will get more people using single sign-on if we do not charge for it. Against the second intuition, which is that there are lots of. So like SSO is like kind of mandatory in some sense, like if you were running like a SaaS business that has enterprise customers, like you probably have to have it, you will have to develop it, but there are lots of security features that are like basically discretionary, like you will basically add them based on whether you think you can get new sales based on them, and like they will simply not be developed if you cannot charge for them. So like I do not think like I don't.
Speaker 3:I don't think a blanket like you can't charge for security features rule could possibly be right. I do wonder if there is not something for certain baseline features like MFA and SSO, and maybe literally only those two where you want to push, just like if you think about what a national cyber God I can't believe.
Speaker 3:I just said national cyber a national information security goal was, you would say, we want basically all businesses to be using single sign-on uh, whether they're small businesses or like whatever, and that will only happen if we drive these costs way down. Um, and so I I would love less interventionary like policy ideas for, like, how do you make it so every small business' Quicken account has single sign-on? I would like something less interventionary, for how do you solve that problem?
Speaker 1:You kill SAML. That's how you do it. While IDC is cheap, SAML is expensive.
Speaker 4:I don't think that actually solves any of the problems. Now I think that smart organizations already are killing off SAML themselves. I don't know, I find most of the discourse about this topic to be kind of blinkered. So like one way to look at it is that people are charging for SSO and that retards uptake of SSO. The other way to look at it, though, is that you know, people paying for SSO is subsidizing smaller customers. Like there's really not much of a difference, right? Especially when I think you yourself kind of acknowledge it's the reason there's a SAML upcharge or not a SAML like an SSO upcharge.
Speaker 4:There should be a SAML upcharge, I agree, but the reason there's an upcharge for SSO it has nothing to do with support or cost basis or anything like that. It's just it's a really pure customer segmentation thing, right, like the cohort of customers that you want as an enterprise, you know company. Right. Like that cohort is almost uniformly required to have SSO, right. So like the right way to look at it is just like the account that has SSO is what the service actually costs, right, and every other account tier is not the real service.
Speaker 3:Yeah, I think that's like just descriptively correct, like sorry, like not just just it's like clearly descriptively correct that there's like this is a customer segmentation mechanism and they're like three tier users who have effectively the same product are being subsidized. I think it is. If you're just like looking at it as like a policy outcome, like does it produce the outcomes we want? Like I think you get suboptimal outcomes um is there evidence for the suboptimal outcome?
Speaker 4:I get what I I think I understand like if you look at the last 10 years. Given the sso tax right, have we seen decreased uptake of sso as a result of it? Is there evidence for that?
Speaker 3:uh, I can I can't point you to a study, but I have a fair number of friends who work with small businesses and a lot of their customers are roughly in the position of we are not going to spend $3 extra a month for each account, for each user. That's the optimistic case, right? There's a lot of SSO taxes that are, you know, significantly, we'll say more segmented than that, and so it would be utterly shocking to me if you know, not just on the margin but like in the whole, like there was not significantly reduced single sign-on. Like I said, I would like something less interventionary than like it's illegal to charge for this Because, like I think, if you make it illegal to charge for this, you get less of it. That's kind of like economics 101.
Speaker 3:So I am far more interested in what are holistic policy changes one could pursue on just how we price the externality of bad data security than we have right now and like kind of let the chips fall where they may on terms of like what, what protections companies choose to like implement. Like I think right now you have this problem of what is the cost of bad security is maybe very difficult to predict. It is the through line between, like I don't do this, I will have this a breach of the severity, and like what will my punishment for that breach be? Is is insufficiently predictable for people to invest intelligently, yeah and I hate that, but it's definitely true, it's, it's.
Speaker 2:You can make an argument that this could be very, very bad, like a breach that happened over there, to a similar person in your market share, with a similar size customer base, with similar kind of data, but you can't deterministically, with any kind of confidence, actually say that.
Speaker 3:Yeah, sorry, I was going to say, even if we stipulate like I will have you know, if I don't do this I will have a breach, like what? What will the consequences for my breach be? Are are, I think, far too unpredictable. So I think policy that betterly better forces companies to internalize costs of bad security are like much better. And then, like we, bad security are like much better. And then, like we, we let the chips fall where they're right, like let that incentive guide how companies choose to invest. If they still don't want to buy sso, they don't buy sso.
Speaker 2:um, what's an example of a policy that internalizes those costs as opposed to externalizes them out to users?
Speaker 3:yeah, so the policy I'm uh very interested in is uh basically a strict liability regime, okay for data breaches.
Speaker 3:So like we've got like a basically a law that says kind of strict formula If you have a data breach, like we look at how much data was breached, like what types of data elements, you know how many rows, how many rows of data, and then like here is the price for this. You know it's close to trivial to go into court and get that penalty. There's some really cool economics paper from, I believe, george Mason University in the Michigan University Law and Technology Law Review for David.
Speaker 3:Bill Blue, of course, and they run through kind of the economics of it, of course, and like they run through kind of the economics of it but like basically the thesis is that if the company knows what it will cost when they have a breach, they know better than anyone else what kinds of safeguards are likely to impact their risk, like they know way better than some, like external regulator does, like what will be good for their company. And if they know what the price is, they can act efficiently.
Speaker 2:It's like basically the core thesis. What if imposing an SSO tax is like a multiplier on a data breach or something like that? Or the lack of phishing-resistant authentication?
Speaker 3:Yeah, I think there's probably lots of interesting variants like from you didn't have the security mechanism. I've also heard people talk about you know if the data is really old and like you didn't have a good reason for retaining it. Like that's some sort of multiplier. I think lots of these are like potentially very interesting. I think about two things. One is just like how administrable is the system right? Like is it predictable to the end user and is it straightforward for like a government agency to actually bring cases based on this right? Like you don't want the situation where, like bringing these cases like takes a lot of staff time and so like you kind of only do it sometimes, like that is inefficient from like a company's pricing their risk perspective.
Speaker 3:The second factor I think about is like I think probably all of us in this call and like probably all of the listeners have like an incredibly strong intuition that like phishing resistant MFA is like off the charts good like ROI in terms of reducing your breach risk. The concern I have is like if you put that into law somehow, is you get kind of a distortion effect right, companies have a stronger incentive to patch that risk. They do some other risk. So, like, if you, let's say, we put a five times multiple huge multiplier on uh no, no, phishing resistant mfa, then somebody has an incentive to do that, even if there's something else that is two and a half times higher risk of actually happening. Yeah, so you get these distortions and maybe sometimes you kind of want that distortion because it causes the entire market to coalesce. Everyone always has WebAuthn and the ecosystem effect is worth that.
Speaker 2:It smooths out. The consumers are more used to it, Right employees.
Speaker 3:Yeah, right, like, clearly, maybe, I don't know, clearly, sometimes it may be the value of the ecosystem effect may be worth the distortion, but I guess I am nervous about you know reaching for that thing always, right, sure, I think the best incentive is, like when companies are, you know they care about the on the ground realities and not kind of the 50,000 foot view of like across everyone.
Speaker 3:Right, I think we've all probably been in the situation of some regulatory requirement or some policy says you need protection X and you're like this protection makes zero sense in my context, like why are you wasting my time? And like the answer is like that person is like in the best case, that person's like, yeah, in general, like again, it doesn't apply to you, but in general we want everyone to have this because it's easier to manage our across the board risk that way. But like, as a person in that circumstance, you're incredibly frustrated that, like you're basically being asked to spend time on things Everyone agrees for you are not a good use of time, even if for somebody else they are and like you don't want. I think that's a very bad kind of I don't think people really talk. This is a bad user experience for a regulation Right that is a form of that is an experience that will breed resentment to the regulation that makes sense.
Speaker 3:Rx bad Rx yeah.
Speaker 1:So when you look at SSOtax, do you see a wall of shame or do you see a leaderboard?
Speaker 3:I do not see a leaderboard. I mean, I have also been the guy responsible for security at like a small to medium-sized company, like frustrated that like I. You know it's gonna have to justify you know some budget difference because like we were trying to take security seriously but we were also like small. Uh, so like I by I think by far the best outcome is for companies to give a little bit, right, have the free tier, that's you know the, you know your octa and your g suite, oidc, right, like fine, fine, you know, find, find the slice that like still gets you most of your customer segmentation, but like, give a little bit for free, like that. I think that's just such a pressure release valve for, uh, any sort of policy intervention. That's basically what we do is like google and github for free, and then, if you wanted, to do pressure release valve for any sort of policy intervention.
Speaker 3:That's basically what we do is like Google and GitHub for free, and then if you wanted to do something else, yeah, yeah, and, like you know, when we talk to a competition to start, I worry about like encouraging only the largest OIDC providers, but, like you know, that's the best compromise I've got standing here today.
Speaker 4:I like discouraging people.
Speaker 2:I like discouraging people from using small OADC providers. Yeah, because I'm like you don't support Okta and.
Speaker 3:I'm like, yeah, I support.
Speaker 2:Okta. Okay, last thing Can you tell us about this little website that was very popular in late 2020 that scraped the New York Times poll results?
Speaker 3:data the Alex Newton Network yes, so people with long memories will remember, the 2020 election did not end on election night, as is kind of traditional for almost every American election.
Speaker 2:Or at least abstracted that way. Yeah.
Speaker 3:So a thing that was created I think several of you contributed was, uh, we basically made an alternate viewer for when we we scraped the New York times data. Uh, every couple of minutes, we put kind of the raw, uh kind of election result data in a GitHub repo and then we had a you know html page that that rendered it out. Um, it had what, at the time, were some novel statistics. I think was the important thing, the most important of which was the hurdle, which was a measure of, um, what of the estimated remaining votes? What percent, yeah, the remaining votes to be counted? What percent would would the current, as counted, the second place candidate need to capture in order to win?
Speaker 3:And an important fact about the 2020 election is, because of different state laws, there was a really high correlation between vote counting order and partisanship. You had states that counted their in-person ballots and their vote by mail ballots, kind of basically in different orders, in-person ballots and their vote-by-mail ballots kind of basically in different orders, and so this hurdle was like a really effective way to see who was you know how close the race actually was, because you'd have races that were like 50% counted and, like you know, basically like where they were at 50% was very. If you just looked at the raw numbers it was very unpredictable of who was likely to win in that state. So we kind of put this thing together over the course of election week. Many folks contributed it I think kind of went viral on the internet. It was, I mean, it was a useful experience. I think we got a lot of positive feedback. My personal favorite was like a local Georgia news station emailed me to say like this was like incredibly useful to our like local election coverage.
Speaker 2:Yeah, and that was great. Um, and of course, georgia was one of the ones that we let like turn blue and it was like a whole big deal and then and then eventually they had like senate runoffs and a bunch of other shit yeah, yeah, georgia was one of those closed states that took, you know, quite a few days to count.
Speaker 3:Um, my, we had kind of, you know, github issue tracker where people were filing issues. My personal favorite is you know, somebody files an issue and they're like, you know, I don't really know how to program, but here is this like latex math paper explaining why, like your hurdle calculation is wrong in the face of third parties and it's, like you know, fantastically worked through like math problem. So it was a very kind of positive experience. It was a much better thing to like throw yourself into, like building this thing than like actually refreshing. You know, the new york times website.
Speaker 2:Like this was much healthier yeah, and I think I think new york times and maybe other you know election tracking sites like borrowed the whole hurdle notion or they might've called it something else, I think it. Whatever they called it, but it literally showed up and it was not anywhere before your thing.
Speaker 3:Yeah, yeah. When the New York Times had their pages for those Georgia Senate runoffs, they also had kind of the hurdle metric. I don't know if we've seen it in elections since. Now that there's, we're not kind of in peak COVID. There's not that kind of correlation with voted by mail. I believe the hurdle was your invention, thomas.
Speaker 4:I remember. I don't have much more to say. I was wondering. I sometimes wonder if I invented it or if we were just talking about it all at the same time, but I remember it and I like the herd.
Speaker 1:Yeah, yeah, I also seem to remember that like, at some point we got some like a screenshot of like a MySQL database at GitHub that showed that like the. Github pages for the Alex News Network was like far and away the most popular GitHub pages like ever. I do not remember that. I do remember at what point we built a system or the Alex News Network was like far and away the most popular GitHub pages, like ever thing.
Speaker 3:I do not remember that. I do remember at one point we built a system where, like the page would like hit the GitHub public API to like know if there was new data and automatically refresh. And like we definitely got somebody who showed up and was like hello, I am a GitHub SRE. Like you are melting this API, like you are some outrageous percentage of the traffic to this, can you please hit this other endpoint that's cached" and we were like, oh yes, we can do that.
Speaker 2:Yes, we will gladly hit the read-through cache.
Speaker 3:Thank, you, we didn't know about that one.
Speaker 2:No, I remember that and they were just sort of like hey, because the first versions of the website were really really dumb. It was literally just like hit, it would scrape it and it would reload and hit GitHub every time and we were using Git as the index for the database or whatever the database. And then it became smarter and better architected as it became a little bit more popular.
Speaker 3:Yeah, and I was satisfied with the product experience of. There is a TXT file and I run Git log to see the revisions. But wiser heads prevailed and there was an HTML with a table element.
Speaker 2:And that was better. Well, we just wanted to let the people know that, if they ever use that website in the stressful days of November 2020, that was basically you that made that work, and we all just jumped in and scurried around.
Speaker 3:I think by lines of code. I wrote a tiny tiny fraction of that. I get credit for starting the github repo and maybe helping project manage a little, but I think a lot of other folks did almost all of the work.
Speaker 1:Yeah well, we call it the alex news network for a reason, so all right well, alex, thank you very much for for joining us um and well, we'll not talk about what you're doing next, but we'll assure all of our viewers that you're employed and you've missed your opportunity to hire Alex.
Speaker 3:Well, thank you for having me. This has been a lot of fun. Thank you very much, Alex.
Speaker 2:Security Cartography, whatever is a side project from Deirdre Connolly, thomas Tachuk and David Adrian. Our editor is Nettie Smith. You can find the podcast online at scwpod and the host online at durhamcrustalum, at tqbf and at david c adrian. You can buy merch online at merch, at security cryptography, whatevercom. We're also available on youtube now, if you like the pod. Give us a five-star review wherever you rate your favorite podcast. Thank you for listening bye.