Python Cryptography Breaks Up with OpenSSL with Paul Kehrer and Alex Gaynor

The Python cryptography module, pyca/cryptography, has mostly been a sane wrapper around a pile of C, so that users get performant cryptography on the many, many platforms Python targets. Therefore its maintainers, Alex Gaynor and Paul Kehrer, have become intimately familiar with OpenSSL. Recently, they declared that after many years of trying to make it work, they announced pyca/cryptography would be moving away from OpenSSL when supporting new functionality and exploring adding other backends instead. We invited them on to tell us about what has happened to OpenSSL, even after the investments and improvements following Heartbleed. No guests on this pod represent anyone besides themselves.

Watch on YouTube: https://www.youtube.com/watch?v=dEKBHI3rodY

Transcript: https://securitycryptographywhatever.com/2026/02/01/python-cryptography-breaks-up-with-openssl

Links:
- https://cryptography.io/en/latest/statements/state-of-openssl/
- Py Cryptography: https://cryptography.io
- https://archive.openssl-conference.org/2025/presentations/Alex_Gaynor_Paul_Kehrer_The_Python_Cryptographic_Authoritys_OpenSSL_Experience.pdf
- https://securitycryptographywhatever.com/2025/08/16/alex-gaynor/
- https://packages.gentoo.org/packages/media-libs/libsdl
- https://www.youtube.com/watch?v=RUIguklWwx0
- https://datatracker.ietf.org/doc/rfc9180/
- https://docs.openssl.org/3.3/man3/OSSL_PARAM/
- https://openssl.foundation/
- https://github.com/openssl/openssl/issues/17064
- https://www.feistyduck.com/newsletter/issue_132_openssl_performance_still_under_scrutiny
- https://github.com/topazproject/topaz
- https://github.com/actions/runner/issues/1069
- https://crystalhotsauce.com/
- https://openssl-library.org/news/vulnerabilities/#CVE-2025-15467
- https://en.wikipedia.org/wiki/Ship_of_Theseus
- https://boringssl.googlesource.com/boringssl/+/aa202db1d7091b88b80f0a58c630c5c1aefc817d
- https://www.ibm.com/products/open-sdk-for-rust-aix
- https://dadrian.io/blog/posts/corporate-support-xz/
- https://peps.python.org/
- https://cryptography.io/en/latest/hazmat/primitives/asymmetric/ed448/
- https://go.dev/blog/fips140
- https://dadrian.io/blog/posts/roll-your-own-crypto/

"Security Cryptography Whatever" is hosted by Deirdre Connolly (@durumcrustulum), Thomas Ptacek (@tqbf), and David Adrian (@davidcadrian)

Chapters

• 0:00 intro
• 1:03 The Python Cryptographic Authority
• 4:38 fckin OpenSSL
• 9:50 OpenSSL 3
• 23:17 OSSL_PARAM
• 31:16 testing and continuous integration
• 34:48 GitHub Actions, target architectures
• 35:16 flappy tests
• 35:47 Intel SDE, rigorous, stable architecture-dependent impls
• 38:21 Community Maintained Assembly and Bug Handling
• 39:07 Memory Safety and Rust in Cryptography
• 39:59 Recent OpenSSL Vulnerabilities
• 40:09 having the judgment to not write large bits of C code
• 41:18 BoringSSL vs OpenSSL
• 42:32 SSL of Theseus-ying OpenSSL with Rust code
• 43:38 PKCS and P7
• 47:56 there's Rust in my reptile eggs
• 57:20 The Python Cryptographic Authority, 10 Years In
• 1:02:55 Moving Away from OpenSSL
• 1:09:38 FIPS validating non-C code
• 1:11:50 👋

Transcript

0:00

It's 2026. It's like the most scrutinized C code on the planet. All of like the low hanging fruit, memory, corruption, and open. An OpenSSL gone now, right? Like how likely one today, so I was gonna say, like we, we were literally today. Hello. Welcome to Security Cryptography, whatever. I'm Deirdre. I am David. I am a national spokesperson for the Crystal Hot Sauce company. Awesome. That's Thomas. and we have two. Special guest today we have returning champion, Alex Gaynor. Hi Alex. Hello. Hello. Thanks for having me a second time. Yes, thanks for coming back. We didn't scare you off. And, uh, our other collaborator on the Python Cryptography Library, Paul Kehrer. Hi Paul. Hi everybody. I'm just here on Alex's shirt tails. Yes, we invited Paul and Alex today, on today because they wrote a little blog post about OpenSSL and they gave a talk about OpenSSL a couple of weeks ago, on the same topics. And they have a lot of experience trying to operate around OpenSSL as maintainers of the, the Python Cryptography Library. We say the, because it's literally called cryptography. And so if you wanna redo anything with cryptography in Python, theirs is usually the one that you go for. Alex. Te tell us what prompted you to write this statement, which is also the first statement from the Python Cryptographic authority. AKA you two. Before you do that, can I ask one or both of you to explain to, you know, the world, why Pike cryptography is, you know, like why you guys have standing to make these complaints. Like, who do you guys think you are? Hi. Sure. I, I, I, I'll start with that and Alex can jump in as he sees, uh, necessary. So, uh, the Python Cryptographic Authority is a self-proclaimed authority. It's important to note that inside the context of Python, there, there used to be like, like a running gag where you would create a, an authority to define like the, basically the, the GitHub name space that you were going to use. So the Python Packaging Authority, which became a very official concept in Python, was originally actually the creation of a single person who wanted to go and do some work. similarly, the Python Cryptographic Authority was founded in kind of an aspirational manner back in 2013. like Alex and I were working for the same employer, back then, and we looked around and discovered that there was not really a good solution for cryptography and Python. In Alex's case, he was also interested in Pi, PI, support a thing that he regrets to this day. Uh, but, uh, The outcome of that was ultimately that we, you know, we've embarked on a 13 year and counting, uh, adventure of taking over the, the, the world of cryptography and python. and largely we've been successful, mostly because we have a, I mean, I'm sure we're gonna get into this, but we, we have a somewhat maniacal focus on the way in which we deliver the software and the way in which we, that the expectations we set for ourselves, such that we can and do have higher expectations for the things that we depend on. Yeah, I can't emphasize enough how in like pre 13 years ago, so like 2011, if you tried to make an H-T-T-P-S connection from Python, it was effectively impossible. you at best like had to install Py OpenSSL, which didn't have wheels, so it had to build OpenSSL from source every time. And so it, and that, and of course, OpenSSL would fail to compile because it was 2011 so it was just bad. And for those who don't ship a lot of python, libraries, wheels is a technical term of art. It's not just ha ha wheels included, like batteries included. What are wheels? Yeah. Wheels are, uh, python's, binary package artifacts. So you can have an sist which is sourced, that you compile yourself, or a wheel that's pre-compiled by somebody else. So we, we build wheels of cryptography for many, many operating systems and CPU architectures. This is a, a language called Python. Why isn't, why isn't it called like eggs or, or like, were called eggs. oh God. Damnit. Okay. And now they're called wheels because that makes sense. All the, the. Alex is the first person to be asked the same question twice on this podcast in two different episodes. Yeah, you get your wheels at the cheese shop. Which The cheese shop was the original name of the Python package index. What? Okay. Okay. It's cause Python was named after a Monty Pi after Monty Python. And the Cheese Shop is a famous sketch, Now I'm remembering our whole episode that we, we did this whole rigamarole with like, you know, we've been running, going strong for several years now. I, for, forgive me for forgetting, but, okay. So, I was gonna, I was gonna say, right, so like the, the situation with like, like your biggest dependency is openness, L right. correct. Yes. Yeah. Right, and like in the universe of things that use OpenSSL, like that's gotten a lot more complicated over the last 10 years as Chrome has shifted to BoringSSL and AWS now uses their own kind of formally verified SSL and open BSD did LibreSSL, like in the universe of OpenSSL consumers, where do you guys fit in? Like, what's your rank? I, I don't know that there's an official leaderboard of like, OpenSSL consumers, but, I, I guess we have a lot of standing, at the very least in the Python ecosystem, we're consistently one of the most downloaded packages, on the Python package index. Many millions of downloads, like if you use a Python thing, it's like quite likely you rely on us. all, all of the major clouds command line interfaces include us. because then like the, the cert bot, lets encrypt client includes us. Yeah. In the same sense as like if you were already a GO program and you use Net HTTP, you'd be pulling in go's, TLS libraries. In Python world, if you're doing the equivalent of if you're doing requests or whatever, you're effectively to do TLS pulling you guys No T So TLS is the one thing that is, I think in the usual cryptography toolkit that you probably would not pull in, cryptography for, we don't have TLS APIs. You're much more likely to either use the Python, standard library SSL module or PI OpenSSL, which, which we also maintain, but is a separate library. Oh wow. And just for the record, why aren't you just writing your Python cryptographic library in Python and shipping Python? That would be a very convenient, unfortunately, it would have both security and performance implications that are, effectively insoluble inside Python, the language. I, I will also say when we started cryptography, you know, 12 or 13 years ago, we made, put amounts to like a very ironic deal, which is like, I was persuaded that like I would help coordinate this library as long as the one thing we were doing was not implementing cryptography. We wanted to think hard about APIs. We wanted to think hard about what made a safe default and what shouldn't have a default at all. We wanted to think hard about testing, but we were not trying to do something where we felt like the relevant expertise was like really outside of, you know, what a person might have if they didn't have like a PhD in cryptography. and you're talking like low level primitives because like we can have a, a, a wonderful debate about what counts as implementing cryptography, but I, I know what you mean. Yeah. yeah, I, I would say about six months after we, you know, I had this conversation where we're not implementing cryptography. One of our maintainers at the time, I guess this would've been right after HeartBleed, came up to me and said, we should implement TLS. And I said, what happened to our deal? And he said, TLS is a network protocol. It's not cryptography. Uhhuh. Uh, and I think that's one of the, and I think it's like a very true statement that like really rides the line. Yeah. Yeah, and we've enclo we've definitely increasingly blurred it over time. I mean, I think one of the first times we kind of said, oh, this isn't really cryptography, was when we did like a AES key wrap support, back in the day when like OpenSSLs implementations of it weren't very good. and like we still strive not to do it unless we have to, but there are scenarios where we choose to hoist it into ourselves because we believe we can do it better. I think the most recent example of that is actually something that we haven't released yet, which is the, uh, HPKE support. OpenSSL does support HPKE, but we have chosen to implement it ourselves using OpenSSL as primitive. And why did you choose to do that? So I think there were a, a couple motivations to come into it. One, it means we're, you're gonna get a really consistent HPKE implementation no matter where you get your cryptography from. So like the OpenSSL HPKE implementation is not supported on LibresSL BoringSSL, aws-lc, which I, I guess we'll probably mention a lot in this, conversation. So I'll just call the forks. So like, implementing it ourselves means you'll get a consistent experience 'cause they'll all have the underlying cryptography. it means we have a lot more control over kind of the compatibility surface. We're not accidentally pulling in lax parsing behavior or other kind of unintended behavior from OpenSSL and. It gives us the kind of high level of confidence that we're, we're handling all the edge cases from both the security and correctness and just like not crashing perspective. Paul, I don't know if there were more motivation for you. Well, I would say that the, the, the one that's kind of the elephant in the room, that's the, the core component of, part of our criticism that ac occurred in our statement is that the HPKE APIs are only accessible through the OSSL_PARAM api. So, okay. I wanna stop you guys there, right? So I, I, so we're kind of beating around the bush and like, I think trying to like let you guys introduce the, the, the talk that you guys did yourself, but I kind of don't wanna do that. Let's just like cat outta the bag. You guys gave, you're, you're not super happy with the current state of OpenSSL but it's like, not for the normal reason that people usually bring up, like, you know, that's why there's Libra SL 'cause we don't tRust the code or whatever. Although we can get into that later too, but for other reasons. Right. So, and your, your reasoning is mostly about, or is entirely about the design of OpenSSL 3. The, that version of the library. So I have not paid close enough attention to OpenSSL to know what the fuck OpenSSL 3 is. What's roughly like the timeline here? What happened? Like I knew OpenSSL back on the Heartbleed days, and now there's multiple OpenSSLs. So, you know, Heartbleed happened in 2013, and at that time it became like a very well known fact that OpenSSL was kind of a project on life support that didn't have the right resourcing and had various problems because of it. Uh, in the wake of that OpenSSL got a surge of investment, both monetarily and also in, in, uh, human time. Like there were a bunch of folks from Google, like Emelia Kaspar and others who went over there and did a bunch of work. Um, there were folks who, who later founded some of the forks that were involved heavily, and then there was just like a large organization formed that had a bunch of full-time employees to do things. As a component of that, the business realized that, part of the way in which you sustain in the cryptographic library is by catering to business interests around things like FIPS. The structure of FIPS is such that you want to isolate it off and that. Created a concept that they were, uh, that was supposed to replace engines. Uh, engines are a method of plugging things into OpenSSL. Providers became the new thing. Providers are intended to be a superset that can do all that stuff. And in the abstract it's a fine concept. However, in practice, as they went down the path of this OpenSSL 3, which was an intent, a deliberate, um, ABI breakage, they skipped 2 because they were worried about version numbering based on the fact that Libre had forked themselves and they called themselves 2 at the time. they went to three and the consequence, like the actions of three were basically rewrite the entire internals were without understanding the surface area of your own project. Uh, this led to an 18 month alpha beta phase before release, and ultimately well over two years of delays from when they expected to be able to release it. Uh, did that, when did that land, like when did OpenSSL 3 become? I assume that right now OpenSSL 3 is like the mainstream version everyone's using. Yeah, if you get your OpenSSL from like a Ubuntu or Debian, a Red Hat, you're gonna get an OpenSSL 3 from like a recent Linux distribution and then that landed in 2021. Okay, that makes sense to me. So now I know what you're about to say, but let's bring our audience up to speed on the suite of concerns that you guys have about the situation. Yeah. So we've got a couple, and I'm gonna say them in the order that we wrote the post in. as we'll get into, I think we put them in the wrong order in the post. so the, the first concern, it's, it's the most easy to quantify is performance. Uh, OpenSSL 3 had some really, really significant performance regressions. Things like loading, elliptic curve, public key from like a subject public key info format. Really simple format to parse. Got something like, eight times slower between OpenSSL 1, 1 1 and OpenSSL 3 Was it doing the, on curve math? Was it checking that No, no. It like, yeah. Yeah. It's like anything you can come up with it that's like the cryptography, the math that might make this time is not it. It was that the abstraction for how providers worked and like the interplay between it and the DER parser and just like what the public, a pi, like, all that back and forth had so many indirect calls, so much allocation, so much locking, OpenSSL had a whole like format, auto detection thing that happened. It was like, you know, you can find the issues in the OpenSSL bug tracker where folks are like breaking down, like, where is the time going? And it's, it's truly just that the, the parsing itself had become so convoluted, that, that it was just. You know, it time was spent in nonsense places. we'll say since then, OpenSSL has made some improvements and now it is only three times slower than it used to be as of, the last time I measured. So, like, that's, that's quite significant. Like, I, I wanna just give people like a data point for like how extreme this was. We, we have, our own X.509, like path validation code, what people would call like X.509 verification, and, doing our own public key parsing that is moving the public key parsing from OpenSSL to our own Rust code, no other changes was a 60% performance improvement on end-to-end X.509 validation, like that, that's just like how extreme this overhead was to like, what is empirically possible. So this was kind of the first and most easily quantifiable, the performance, was insane. And I, I think. Well, maybe we get into this more. A lot of people think, think that this is like maybe our biggest complaint'cause it's the easiest to quantify. But, but it, it's not, it's, we shouldn't have put it first.'cause our real complaint is, is the complexity that led to the performance regressions. The fact that the provider APIs were designed or evolved in such a way that like the abstraction boundaries were unclear. And, you, you had. Really extreme performance regressions that came down to things like, if you load a hash algorithm, through like a function like EVP SHA-256, which is like get the SHA-256 like hash, I, object identifier like that is slower.'cause what that API is doing is getting like an object that represents like a future promise that I will call the provider API later to like actually find the SHA-256 API, from your provider. And like, that can change at any time in theory.'cause like the provider APIs don't like say like, you know, once your program's initialized, like you can't change where the cryptography comes from. So there's just like tons of back and forth, lots of indirect calls, lots of allocations, lots of locking, lots of caches to try to compensate for that. Like the degree of complexity in the internals got like really extreme, but it also got pretty extreme in the public APIs. Yeah, I mean, I, I won't add a lot here other than to say like the, the ultimate, uh, outcome of these like attempted fixes for performance progressions based on this, is that like we, we have like full on RCU code, which has had bugs in OpenSSL since they landed it to just to try and resolve some of these issues. and like the nature of the DER parsers are actually that it chains things together. Like it'll, it'll progressively try different things, uh, which led to a bunch of bugs DERing the OpenSSL 3 betas where they were leaving errors on the error stack because they, they didn't know they were doing these chain, chain attempts to parse things. What they were doing different. I like different implementations of dear person and they would like try one, and then if it didn't throw, we would try another So, so OpenSSL has like these auto key loader APIs where it's like, give us some DER and we'll figure it out and give you back what you want. but that like necessarily means that it's going to try a bunch of things rather than simply reading the, the, the o out of it and then dispatching correctly. so you get a bunch of performance issues from that was that, I have two questions, right? So first of all was the, kind of the YOLO mode. Just try all the different formats, key parsing thing. Was that an OpenSSL one thing or did they come up with that in, in OpenSSL 3. one did have the same pri uh, behavior on some limited APIs, like it's like D two i auto private key or something of that nature. Uh, so that, that behavior was there, but it was a much faster one in that it dispatched based on the OID at the time, as opposed to this one, which needs the providers. And like every provider doesn't, there's not like a mechanism for providers to necessarily, um. Like necessarily declare exactly what they support. So you have to just incrementally try them until one works. I'm, I'm a little familiar with the OpenSSL code or the OpenSSL code of Old, right. I don't remember Concurrency being a huge thing in that code base. So you, you said that there's like a full line implementation of RCUs now in s like what is the, you guys would know way better than we would. What is the concurrency situation? This, if this is a, a crypto library. Yeah, it's a crypto library, but like you have like various APIs for manipulating like global states, like, adding a new provider to like the global context for example. And so like if you haven't done anything to preclude like that happening at arbitrary points in the runtime, wait, wait, wait, wait, wait, wait, wait. Why am I, why am I adding new providers to the runtime State of a running program, I mean, that's roughly the question. Like Paul and I would ask, like we think that's just like a design mistake that you can support that. But like if you've chosen to support that and are now like going back to like, you know, you don't, you're not quite sure like what your API contract with users is. Like, and not now you have to like deal with like concurrency bugs. People are reporting like tsan issues or like performance issues. Like not, not like, yeah, like, hold on, hold on, hold on. Rcu are what you use when you're locking so much and so many hot paths that you can't actually do mutexes, that you actually need an optimized concurrency primitive because you're, I, I get like, you need to lock because you can add a provider, can you like add and remove providers in a hot path? Nothing precludes you from doing that. Yeah. Yeah. There, there's no, there was, there was not necessarily like, I, I wanna be charitable here. So like, maybe the, the answer may not be that they didn't have a design ethos around it, but like, for whatever reason, as they went down the path of implementing this, the answer of support, everything at all times became the correct answer. And like that led inexorably to these types of choices. Because, because engines in OpenSSL that was there for like people doing like crypto card stuff. Right. And at the time it was for like people with co-processor accelerators. Right? So the idea here is like you're supporting a use case where somebody has like, you know, a card or something like that. They're inserting and removing like several times per millisecond is why you would need rcu. I, I think the, I mean the RCU component is just the, the component. What comes out of the fact that they have to do locking in so many places just to check the, the actual reality is like they, as best I can understand it, again, steel manning the concepts of the provider, there is a design goal in OpenSSL that it is an abstracted substrate upon which anyone can do anything, including adding features that were never considered by the OpenSSL people. Like basically linked into any arbitrary program, right? So like, oh, in the future, 10 years from now, I can load arbitrary providers into some old piece of, uh, kit and get new stuff. Yeah, and like you do see examples this like before OpenSSL had post quantum crypto algorithms in it. There were third party providers that would provide them. And so like you could get an ML KEM inside OpenSSL before OpenSSL supported it. Like I, I think we would say like, this is just like not the correct allocation of resources to like towards your problems, but it, it makes me so nervous. Like I, you know, I don't have, I don't maintain a project like this. Like I've maintained large Rust projects, but like, no, like, is like, lib. SDL works kind of like this. It's designed to both be statically, which it's a, it's a like cross platform graphics library that's designed to be statically linked, but also like, sometimes it gets swapped out and replaced with a different one because, you know, maybe valve like made it run on Linux, but wants it to look like it's windows to trick some game from 10 years ago to run on Linux. And you can kind of see how you end up there. I don't quite understand what the, like surrounding ecosystem is where you need to do this for cryptography, um, as opposed to like games which are, I was about to say notorious for being developed once and then left on for years. But then I just realized I was describing HSMs. Yeah, I mean, I, I think what I would say is that like if you really sat down with a, we want to make more things pluggable, like I think there is probably a design you could get for providers where you put state in the right places. You have like the indirect function call in kind of the right place in the stack that like balances the complexity and the performance and the, like, maintainability of the system. Like I, you know, I, I think there are useful points on the like, trade off curve that are not just like everything is static, but I, I think the point OpenSSL ended up on where, like the SHA-256 or like the result of EVP SHA-256 is like an abstract object which will like call something on the provider later. And that's allowed to change at like arbitrary points is like not a particularly useful point on that trade off Yeah. No, God. I was, I was, I was set off on the whole RCU thing. So I was wondering, it kind of made sense to me that if they did a whole bunch of new concurrency stuff, like that's a way I could see you getting like a six x performance regression. Right. Because Okay. You're just like, you know, you're contending on locks or something like that. Mm-hmm. I, I mean, and there certainly are a lot of locks, particularly in the earliest profiles. Okay. But it's not, it's not your sense that it's, I'm, the vibe I get from you is that it's literally just calling through bullshit indirection code. It's not waiting on locks. I, I, I don't wanna say there's no locks anymore. Like the point of RCU is like, now you're spending less time waiting on the locks. but like, like I said, it's still three x lower than we think. It's like a reasonable baseline for like parsing DER. And, and I, I will note that I think we've, we've fallen to the same trap because of the performance stuff is so easy to talk about. Uh, but like, again, the performance is not our critical concern here. It is the complexity that led to that issue. Yeah. Which I, I think it's worth building on kind of the. Complexity in the public APIs is like a really important other, thing that is, is like honestly like maybe the most pressing thing for us, Yeah, you mentioned OSLs, O-S-L-O-S-S-L, param, and like, you know, I'm, I'm poking around in there, in, in OpenSSL for, you know, post quantum stuff, and I'm seeing this all over the place. And like, tell what, where does OSL param come from and why, why is it there? Yeah. So OSL pram is, is another one of the kind of new APIs from OpenSSL 3. And like it is effectively, like many, many public API functions. Instead of like taking like a list of like arguments to a function, take, a an array of OSL paras, which is basically an array of key value pairs. So like, instead of passing, I don't know if you, you're calling like argon, two, like you've got a key derivation function. Like you've got, uh, you've got your key material. You've got like your salt and your salt le maybe a length.'cause you know it's c so you got your pointer and your length fine, uh, and your like desired output. And, the RN and I can't remember the third parameters name, like e instead of like, you pass each one of those to function to a function as like arguments, the way this works in OpenSSL 3 is that you create an OSL parameter. and it's got, you know, you know, string key pointer to the key string, uh, salt pointer to the past salt len and like. That is like how it works. And you know, you've got types for each of the values.'cause like you wouldn't wanna Yeah, I was, I'm looking at that now. I'm looking at the, the Claude summary on this now, and it's like integer, unset integer, UTF string octe. Does it really have octet string? it does. you wanna pass like arbitrary bites, like what you want. You think your salt should be like UTF eight? I mean, it sounds like they've created C-J-S-O-N Oh goodness. Yeah. But like. C JSON TypeScript interface. Yeah. And like, you know, a again, to like try to steel man, this, like our understanding of like, this is part of just like the make things very abstract theory of like, well you could write a program that like, I don't know, has a configuration file and like read some parameters from it. And like that would mean that like somebody could bring a new algorithm and new types of parameters and like, you'd never have to update your program 'cause like it all flows through like this abstract OSSL_PARAM. In practice, our experience for like the things we are trying to do is, it means, it's very difficult to tell what arguments the function takes. It's very difficult to tell, you're passing 'em correctly. You're, you are losing a whole bunch of like static like. Static type checking that you would normally get from a computer program. It makes things slow and it makes the OpenSSL code like much more complex, right? Like if you think like, oh, I have the name of a variable. That's an argument to my function. Like that's just like clearly much simpler than I'm gonna go root around in the array. And in fact, many C Source files in OpenSSL's source now have a custom Perl pre-processor to like make dealing wait, Oh, no, no. wait, wait, So wait, wait. Hold on, hold on, hold on. In the new OSSL_PARAM world and OpenSSL 3, none of the cryptography interfaces are type safe anymore. They all just take abstract arrays of parameters. All the checking is, none of the new ones. Like they're still old APIs it's all run, it's all runtime checking. Now they're just like, okay. Correct. Yeah. So we're going backwards. yeah, any new interface like Eeb, E-V-P-K-D-F, and E-V-P-A-E-A-D, those are all interfaces that now require OSSL Perra, uh, and almost any new feature added, does require it. have spent a lot of time and energy like trying to not use OSSL_PARAM except where necessary, and I think we currently have two places we consume it, but we actually abstracted away by pushing it into Rust OpenSSL. Let me, let me steal man. This design. All right. What if the idea here is that no normal person is ever supposed to use this interface, which is the only way I can think. The only way I can think to describe a, A cryptography. A cryptography interface where like the IV isn't type safe at, at compile time. So you're saying me and my collaborators are not normal 'cause we did just that the other day. on, hold on. I'm saying only two people in the world are ever supposed to consume OSSL_PARAM, and it's Alex and Paul, and what you guys are supposed to do is take that and then turn it into something reasonable. So I, I mean, I think what I would say is like, if you, you like, you'd come to the idea that like your, internal APIs needed to have this for whatever reason, right. You, you needed, you know, more flexibility and like the provider API, 'cause like that a BI had to be the same for forever. Like, if you, if you had like a theory of that, like this needed to be kind of your internals. I like, I think the, like I, I'm not sure I would ever reach that conclusion, but like if I did, it seems like what I would want to do is like, have public APIs that are like entirely like type, safe and like construct these things internally. And it's just like not the case. Like if, for example, if the thing you would like to do is configure, OpenSSL to do elliptic curve, signatures that use, uh, deterministic nonces like specified in RFC, like the way you do that. Yeah. on, hold on. Hold on a second. You're telling me that in the new system. If I wanna like do GCM or something and I'm passing a nonce, it has to do a string compared to find the nonce key. Correct. like I, I mean, I think in practice there's like old APIs that were type safe for like things like just like doing a symmetric encryption, but I would say, Alex, they've been significantly re-implementing some of the old APIs, using the new APIs so that like underneath the hood, they generate OSSL pers, and then there's definitely a string comparison. So like, that's better than making me know about them.'cause like literally, if you want to do an elliptic curve, private key signature, and you wanna use a deterministic nonce RFC 69 79, the way you do that is you create an OSSL paras with two entries, one of which is a bool indicating true. And the other is like the marker for like, this is the end of the params. And like you pass that to like EVPP key, like set paras or whatever the function's named. Like, you can't just pass bull to like, you know, enable deterministic nonce. Like you create It is like, it's like they're implementing Ruby, I, yes. So this like, I, so like I used to be a programming languages person. Like I, like Paul mentioned, like I got my start working on cryptography stuff.'cause I, I was working on pi pi, the Python implementation and like, yes, this is what it looks like if you're like writing an interpreter and like, you know, a user can define arbitrary length functions. Like, so you have to have an array of parameters. Like, yeah, this one interpreter looks like, It It you don't get to get out of this discussion without noting. You also worked on pie Ruby. it was not called pyro, but it was called Topaz, but yes, yes, I created a Ruby interpreter. Yeah. I will say every time I end up doing OSSL_PARAM things, what it feels like to me is actually like, Apple's design aesthetic from next step era. Like Ns, mutable dictionaries everywhere with all the same sort of challenges where like maybe the, like maybe the, principle like golden path has been tested, but God help you if you pass anything outta the ordinary, who knows what's gonna happen, which I think I gotta say, to testing. I gotta say, passing in its parameters to a function, a dictionary of random string keys and values does feel very python to me. Hmm, but it's, but this is the part that's in OpenSSL, supposed to be C library, but apparently it's getting OSL with param to just rename OSL per to qua, and suddenly you guys would be all over it. I mean, it's not really our design aesthetic for Python. Like, you know, I think we have like pretty strongly typed APIs within the Python world, but like y yeah, like that would be a recognizable like di dynamic programming language aesthetic. That is like definitely true. I just don't see why you would bring that aesthetic to C like, that's not Steve's problem. So all of this dynamic stuff and lack of static validation and verification seems like it would make it, hard to test, or at least harder to test. So we definitely, it's, it's definitely a difficult thing to test. it is also the case that the OpenSSL project was founded in the nineties when aesthetics were different. however, over the course of the, the decades the Open SS l project was been around and including the time now where they are a very well-funded organization with more full-time engineers than work on any of the forks, to our knowledge. They, yeah. And a foundation. And a corporation and, 17 different, like interest groups at this point. they. they still struggle with testing. So like Alex and I consistently joke that like the Python Cryptographic Authority is a CI engineering project that incidentally produces a cryptography library. and part of the reason we make that joke is that like, it, it, it reflects our real belief that like, that type of infa investment in continuous, in integration and testing pays dividends in terms of like software engineering velocity and the quality of, of the product we deliver. we spend so much time on it that like it can almost make the other work we do seem trivial. unfortunately the OpenSSL project, I mean, we've worked with a lot of these folks. We, we, we, we like these people. But like it, it's important to, to note where, where struggles continue. And one of those things is that despite all this time the OpenSSL project does not prize testing in the way that we prize testing. we have seen. There are, there are many ways, we, in which you can judge this, but one of the ways which is fairly prominent is go and look at any bug fix or new feature set that lands on, uh, OpenSSL and look to see whether or not there are tests, now new features typically, yes, bug fixes frequently no. and if you ask about them, they won't say they shouldn't be tests, but they may not happen. They're not prioritized in the way that you would expect a project like this. Similarly, you have a large CI matrix. Alex and I spend a lot of time and energy making sure our CI matrix is clean and fast because not otherwise is very painful. In fact, uh, as, as a, as a related note, OpenSSL ship, the bug fix release today, which meant we ship the release because we statically link OpenSSL in our binary artifacts, the wheels. And because we've spent two weeks with Windows arm 64 builders failing, we removed Windows, arm 64 support. That's how serious we are about this sort of thing. Failing. Not like, 'cause we landed a regression, but due to like a platform issue in GitHub's, uh, windows arm 64 runners. Yeah, yeah. This, it wa it was because of the, the runners. Yeah. The runners themselves have platform and it's not the first time we've had issues like this. This is not a, a conversation about how Microsoft owns GitHub and yet somehow they don't prioritize Windows Arm 64 at all. But like, we, we've had enough issues and that like be, we gave them a lot of time, but we care very deeply about our CI working and being performant. And so we were willing to remove it even though it, it, it is painful. that's indefinitely until those things can work again. Yeah. When, and they work and we have confidence they'll stay working. Exactly. I need a track record behind it.'cause fixing it is fine, but like I need to see it working for an extended period of time and that like, they actually respond in a timely fashion to future issues. a failure is just flapping all the time, then it's, it's a noisy signal and it's not a useful signal, and then it's yeah. Flip flapping at the all the time is like a, a useful segue into like, that is a big problem with Yeah. really the apex of this was OpenSSL 3.0 0.4 had a buffer overflow in the RSA implementation, for a VX five 12 capable CPUs. And in fact, this failed in CI sometimes when the CI runner happened to be allocated on a machine with AVX five 12. But it didn't but, but it was doing a different architecture, like. it was randomly GitHub action. Well, so GitHub actions doesn't guarantee like which CPU class you'll get. And so like sometimes you built on a machine with AVX five 12, sometimes you didn't. And so like when you were on AVX five 12, your tests probably failed, like as a buffer overflow. So there's always an element of luck, but it was not noticed.'cause tests were kind of always flaky. And, you know, I think this reflects two issues. One tests being flaky all the time, which is a really persistent issue. Like the day we wrote our slides, for the original talk version of this, five of the 10 recent commits had failing, ci checks. And when we checked again the day before we get delivered, our talk, every single commit, had failing builds for cross compilation. This is like the first issue, just like lack of really prioritizing stability. The second issue is like lack of investment in the kind of infrastructure. for like open a sales, a project with lots of like per platform assembler, per platform assembly. And Intel in fact, offers a tool called Intel, SDE, which basically lets you dynamically toggle, uh, CPU features on and off. So you can, you know, simulate A CPU without a VX five 12 A CPU without SSE three. And so you can in fact test all of the combinations of assembly you have. And there are forks that use this in their CI to verify all of their assembly. And it, you know, this is not present in OpenSSL ci. And like, we think that's a real miss. Like that's just a missed opportunity. Like I'm, look, I'm, I was, I was stunned that you're like, you can't make sure that you get like an AVX backend runner in GitHub actions. And like the answer is okay, maybe you can't, but that means you put your A VX specific stuff behind a flag. You put, online your own custom runner. You, you rack your own hardware or you pay someone to rack the hardware, and then you have the flag so that when you do allocate to your custom runner that always runs on AV X, then you unflag your flag and then you test your AV X specific implementation. I know lots of projects that do this. Yeah. now a word from our sponsor, fly.io, provider of a VA escapable. And from the official hot sauce of security, cryptography, whatever, crystal hot sauce. that a salt brand? It's also you're thinking of a diamond. Diamond crystal. Yes. Yes. I am gonna be very disappointed if that doesn't make it into the co podcast. Leaving the, uh, so like, I mean there's, as you noted, Deirdre, there's a bunch of ways where you can, you can slice this such that you get the type of testing coverage you want. It's also the case that OpenSSL is at this point, a big tent with a wide variety of supported things. And they actually do have leverage in this ecosystem. So like if and when someone comes and says, I would like you to land. Like architecture specific assembly for my pet architecture, it would not be out of the bounds for them to say. Supporting that in our, in our system looks like the following and it look would pro, frankly, it would probably look similar to what Alex and I have said in the past around Power PC 64, little endian support or Windows arm 64, which is you will provide, you know, ephemeral runners that can maintain no state and integrate into our CI such that we do not have the responsibility of, of like managing them and they will work in the following fashion, And if they do not, we drop you. yeah, yeah. I mean, I give a real concrete example there. There's an open bug right now against OpenSSL for the assembly, for Spark for doing, I don't even know what, and it's, it's got some bug in an optimization pass. And you know, the OpenSSL folks have basically said like, look, SPARC assembly is not maintained by the OpenSSL core team. Like this is community maintained. Like what I would encourage them to do is say like. This is a bug. Therefore, like we're disabling this optimization If like the maintainer of the SPARC platform wants to like, contribute CI or contribute fixes, like we will accept those. But like we don't wanna ship this buggy thing and we don't wanna give users like the impression that like, you know, give them like, we, we don't wanna carry all of this performance sensitive and buggy code. We would rather slip the ship, the slower thing that's guaranteed to work and put really the onus on like, if the spark owner wants that to be a fast OpenSSL supported thing, like they should do the work. And so like, you know, if the OpenSSL, project like pushed on things like that, we'd be very supportive. we haven't even touched on all the work that the, the Python Cryptography Authority has done on moving towards doing a lot of the riskiest stuff in, a memory safe language like Rust. But like you, just, the two of you and your project did it a ton of stuff on your own, and we're not, it's 20, it's 2025 at this it's All, all of it's 20. I'm sorry, it's 2026. It's 2026. It's like the most scrutinized C code on the planet. All of like the low hanging fruit, memory, corruption, and open. An OpenSSL gone now, right? Like how likely one today, so I was gonna say, like we, we were literally today. we were, or Tuesday the 27th. We were originally scheduled to record this podcast yesterday, and if we had recorded this podcast yesterday, we would not have been able to discuss it. There were several pieces of memory corruption in OpenSSL that were disclosed today. And like Goodie If we had recorded yesterday, I would've had you with that argument. I. I'm kind of like with you, Thomas, like adults understand how to write small bits of C code without like totally screwing it up. Part of that is having the judgment to not write large bits of c code, parsers for like length, type value. And, you know, deterministically bites in, bites out functions. Like we should be able to write in Rust or, or excuse me, in assembly in sea without making huge mistakes. So, the people that are working on OpenSSL don't seem to be able to do this. Many people exist who are capable of doing this. Many of them have been guests on this podcast. Huh. So I I, I'll agree partially with that. Like f folks who are familiar with my work know that, like I talk a lot about memory safety as a language level concern and about how in the long term we need to be looking at CNC plus plus replacements or, or to the extent the CNC plus plus standards committees have any openness to make their languages memory safe. But it is absolutely true that you can write small, where small is like probably less than a few thousand lines of like c code for well-defined tasks that don't change very often and maintain memory safety. Like that's, that's like an observable fact that like boring SSL is a code base like this that has like a very low rate of vulnerabilities because it's maintainers are like very diligent maintainers. They take testing seriously. They just like think hard about the changes they're making. But it is also just true that complexity and velocity are real world phenomena and, Part of how BoringSSL is able to be that diligent is by having a very narrow scope that is like roughly the set of things Google exclusively cares about. And open cell does intend to cater to a larger audience. It has more features, supports more functionality. And that's like reasonable and like very useful. Like when we compare the set of features, open cell has that boring, a cell does not, like, there are things missing that we would like, for example. so like complexity and velocity are real world phenomena. Scope is a real world phenomena. And so you need approaches to security that are responsive to the real world, to those real world constraints. And so, we think you just have to have a design approach and like a memory, safe programming language is by far the strongest one to not have certain classes of vulnerabilities. Like formal verification is another thing from that bucket for like right. How do you write certain types of programs? Very, very safely. I wanna, I wanna talk a little bit more about how you guys are SSL of Theseus-ying, the, uh, OpenSSL library with your own Rust code. But before we do that, like, the thing today was if you give open LAP seven file that, um, is encrypted with an A 80 cipher is encrypted with GCM or whatever the EVP code, the high level OpenSSL library when it goes and tries to parse the P seven file and like pull the nons out of the it's DERight? Or it's BER, is it BER or DER in P seven. It can be either. Wonderful. Okay. When it goes to, when it goes to pull out the, the, you know, the, the nons slot of the goo right. it, it, it spills that goo all over the stack, right? So, like, I guess the, the first question I have is, does that, we were, we were talking about this earlier, but I'm still wondering if this does or does not hit you guys. this doesn't hit us. while we do have, some, uh. Pkcs, seven APIs that do still use OpenSSL for parsing, for reasons we can talk about our pkcs, decryption APIs use our own, Rust, DER library. And it's a, you know, totally memory safe parsing. are like, this is a little, this is a little off topic, but like for somebody who's asking what the attack surface is for P seven files, what does that look like? so Pkcs seven is like a pretty widely used container format, like it pops up in all sorts of places. Like Mime is A-P-K-C-S. So like, if you're doing like, encrypted email, like not pg, not like pg p, but like, you know what, like what, what Microsoft Exchange will like, encrypt your emails with, like, that's a PKC S seven format. I think Microsoft's like code signing format does it, like it pops up in all sorts of like places like this. Like if the thing you were shipping is like roughly a signature and encrypted blob and like some metadata and like. Particularly if it's like maybe a slightly older standard, like I, I think modern es, modern cryptographic aesthetics or like container formats like this are not particularly useful. But like in, in older things they were like super common. Yeah, it's actually, uh. I forget what the underlying weird name for the like, uh, standard is, but Apple Pay also actually uses Pkcs seven signatures, uh, in the backend. But not with OpenSSLI. I looked, I looked today, and Libra SSL didn't have the bug. Libra ss l in the code where they pull the, the, the non Sout, is doing an explicit length check. Does boring SSL have the bug? I believe so. No. I, I. Pretty so BoringSSL actually published today. A whole bunch of like notes from past OpenSSL, vulnerabilities on, you know, what, whether they were infected. Like what, what customers of BoringSSL need to know. And I'm almost positive that this is one of the ones that marked like this bug was introduced after we forked Mm-hmm. Okay. Ah, So like, you guys missed this bug. You, you missed out on all the fun because you rewrote part of OpenSSL I wouldn't say they're missing it, Yeah, we, we missed this bug there, there were a handful of other bugs, in today's release that, that did impact us. I, I'd have to go back and like, look, look at the full list to tell you which ones, but How, how much of OpenSSL are you guys gonna rewrite in Rust? Hmm. Anything that's not cryptographic crypto cryptography itself. The core crypto like routines are that one before we opened with how that's false. Yeah. so like, we're, the things I would say is like, we will do anything that is like parsing that. It, it's, you know, serialization, deserialization that is like orchestrating cryptography, like kind of the HPKE that we mentioned. But encodings. yeah, we're pretty close to saturated on this stuff. Like almost all parsing at this point, whether it's, you know, public keys, private keys, whether it's like X.509 certificates and CRLs, all, all of that stuff is in Rust. Yeah, I think the 5 0 9 path building. Yeah. the only exception to this point is actually what Alex alluded to earlier, which is that because, p Pkcs seven does support BER in addition to DER we do have in one code path a fallback where if we can't parse it using our DER parser, we hand it to OpenSSL because we have not implemented BER and Rust. For, for and for but you're going to. no, we probably won't for, for those Yeah. For those who are in this like alphabet soup of like BER and DER just as like causing their eyes to glaze over, they're, the, the really short version is BER and DER are two different ways of serializing, kind of the same as n one data. And DER is a subset of BER that like basically takes away all of the flexibility that BER has. Like BER will let you do kind of like very bizarre things that makes it like much more complicated to parse and like, we basically decided like BER is a bad idea for all modern cryptographic standards and like DER is just like a mu much more compact from a, like a surface perspective. So like we, we restrict ourselves to DR and like, that's the only thing you need to care about for things like X.509 certificates, kind of standards that are less well tended than X.509 is, maybe I'll say it like PKCS seven still have a lot of BER in their ecosystems, and. And when we refuse to support BER, and so we'll use OpenSSL in the places we have a compatibility need to parse BER. So has that, how has that gone for you guys? How has that I, I'm, I keep stepping on Deirdre and I'm sorry, but I, I, I want to hear, like, I, I want to hear the story about how it's gone taking PICA cryptography and making it a Rust project. I think it's gone. that's like a whole podcast of its own, but. yeah. Yeah. And, and So it was, it was complicated and dramatic, and not a great experience is what I'm hearing. No, no. I mean, so we, we gave a talk at, about this at Pycon a couple of years ago that really focuses on like the initial release and like what that migration looks like. but and, and to be clear, there, there were challenges and some drama early on where like, we, we, is early on, like how many years ago was this? 2022, I wanna say not that long ago. Oh. 2021. No, it's 2021. Alex, I'm, I'm Okay. 20. Yeah. It's like February, 2021. Yeah. Yeah. February, 2021, we do our initial release. So it's like five years ago now. and the initial release has some, has some drama. Like we, we were pretty aggressive in like pushing Rust into the ecosystem. Users who were getting not getting wheels, they had to compile themselves, like were, and if they weren't per pinning their versions, like they woke up one morning and were like, why is my, Ansible CI pipeline failing with like no Rust sea on my path? Like, what is this garbage? but like we have with lots of help from like other folks in the community, we have like pushed past that. Like we now we ship wheels for a great many platforms like that is no longer a problem. And so like I, you know, I think your question is mostly about like, how has migrating our own code been? And like, I think it's pretty. Much just been like an across the board win. We have much better performance on like all of our parsing APIs. As we kind of alluded to earlier. We have a much clearer compatibility surface. Like, because we own the parsing, we understand exactly what are the places we're being lax because like, you know, the, the specification is like kind of in an ht ML style, like diverged from like common practice. And we know where we're being strict. we have a much, just much clearer, abstraction layers between things. So like, I'll give you a concrete example of, I think that's just like better. so we used to, we, we have a bunch of Acts 5 0 9, like certificate APIs. we used to implement those on top of OpenSSL X.509. APIs. And so when you did something like sign an X.509 certificate, right? Creating a new X five nine certificate and you do a signature, and you'd pass in a private key to like do that signature, but actually that private key had to be an OpenSSL private key. Like we, we nominally had these like abstract APIs that like you could implement, for a private key, but like if you didn't pass an OpenSSL private key, we didn't have a private key to pass to the OpenSSL signature API. So like there's this real like abstraction failure and it's, it's not an uncommon one. Like you, you see this in a bunch of things that like try to do, abstraction layers like this. Like I believe the Java crypto APIs have like a whole bunch of like fast paths and slow paths depending on what kind of, pri private key you've got. But now that we own X five nine parsing top to bottom. What happens when you try to do a, a sign a new certificate is that we use the public sign, API and like any private key you've got, whether it's one of the ones we provide or you have like a third party implementation of, of our private key APIs that uses that, I don't know, talks like A-W-S-K-M-S for example, or GCP or like any, any cloud providers key management. Now that just works. So like we have a much cleaner compatibility surface. We have much more coherent story for things like third party keys. Like, it's just like, like I don't have any bad to say about the migration besides like the initial like stumbling with like, the kind of pain of lots of people having it to adopt to Rust. But Paul, I don't know if you have like a different reflection on this. Yeah, I mean, I, I think I generally agree like pa past the initial teething, like one of the components of that, that entire project is Alex and I decided that. This was worth the breakage budget. We knew our position in the ecosystem was important. We knew that it was going to cause us, us pain as well as some user pain. But we wanted to, you know, both manage, manage our, uh, manage our long-term pain as maintainers and also drive down the pain for the adopters as quickly as possible. So we were able to work across the ecosystem, basically blaze the trail, such that like future Rust, Python projects have effectively the ability to deploy with no fear. Where we, five years ago, obviously had a lot of work to do. on the actual Rust development side, I would say that like. The only piece of pain that we've really experienced, I'm not, not even MSRV, although MSRV has its own like that's minimum supported Rust version for, for folks who are not deep in the Rust ecosystem. Uh, there was some work there we had to do, but like the only real thing was that in our ci Rust compilation is slower than what we had before. And so we ended up spending a lot of time and energy looking at what it meant to cache intermediate artifacts and make sure the caches don't do bad things.'cause we had a few incidents where our caches were pretty bad. but once we got past that, we were in really good shape. And like, I, I mean, even right now there's, there's a current, feature Alex and I are working on where we were unhappy with the Python APIs, uh, that would allow us to express what we needed. And so we're likely to rewrite this piece in Rust simply so we can have the visibility control that we want. Yeah. Yeah, I, so like, I don't know if I had to register a complaint, like my number one complaint might be like, the Rust coverage support is, not quite as stable as like pythons coverage pie. Like, th this is like the level of like complaint we have, you know, about five years of like maintaining this Rust code. so you, you ate the pain of supporting Rust in your extremely widely supported, in terms of platforms, a diversity of platforms where it is, is used and it needs to be supported. Did you have to drop any platforms that PI Crypto was supported on in order to ship the Rust stuff? We dropped no platforms we officially supported is a good way to say that. one of the tricky bits of a migration like this, and I'm sorry Alex, I'll, I'll no, you we're gonna say the same thing. but yeah, one of the tricky bits about. Having a project that's just C and Python is that C and like Python is not a compiled language and c is a compiler that exists. Like there's a compiler that exists for every platform under the sun, including things that are weird, like 31 bit architectures. and so like implicitly, when you ship software like this, you end up with consumers at some level who maybe they only want to compile it to prove they can, but there's a set of folks who are like, we, like I was able to compile this and therefore it should be supported. And so one of the things that did come out of our Rust migration was a much more obvious, like a much more clear and obvious statement of like, we support architectures that have enough support that LLVM has been ported to it. Uh, and one of the, one of the, perhaps most prominent, and Alex and I might be overly patting ourselves on the back for it, but also it really feels like we, we might be responsible is that IBM recent, I guess now it's about a year ago, but, uh, IBM ported Rust to AIX. And one of their headline messages on the blog post where they announced it was Python Cryptography will work. Oh, that's pretty nice. Yeah. And like this is, You're perpetuating A IX. It's not nice. I know, but like, the fact that they, like they are that good at This is a positive sum interaction, is the way I would say it. Like I, IBM like gets better support for whatever customers A IX has. We can point users at a IX who like show up at our issue tracker and like IBM, like they maintain your stuff and like, I think this is like a good message to like projects, to like when you get requests for like weirdo, particularly commercial operating systems or architectures, like push back and like try, try to make people go to like the company they have a support contract with and not like pawn it off on you. You know, if you, if you're not interested in maintaining that. Let's, let's talk about like that aspect a little more before we get back to like broader points about OpenSSL. both you, Alex and Paul have been doing like, open, open source, I say in quotes, for, you know, like 13 years here, you have more than that, but this specific project, it's very widely used. but, well I, I don't know Paul as well, but Alex seems like a fairly emotionally stable person. and, Like you see a lot of discourse around open source of like, oh, there's all these like, people freeloading off me. I don't have enough time now. I hate this project. but at the same time and kinda look over at you guys and you've been able to both kind of keep some amount of support for like a lot of, or, or, or keep a lot of support for a lot of platforms you've been able to get Intel to do. or excuse me, or who, whoever to do AIX. and like you're, you're still chugging along and to some extent, like the Rust thing, like you chose to do that in Rust in part because of your like personal preferences around memory safety, which I agree with. But like you could have also just been like, well, we'll control our own parer by writing c code that doesn't like suck. so like you're able to like kind of push your personal opinions into the project and. Kind of have fun without dying. Like what's your approach to this that like, lets you both kind of keep doing this, but also like doesn't necessarily result in either the drama or BERnout that you hear a lot about in open source? And how do you feel about, like, like are you being funded sufficiently for this? Or like, like how does money fit into this, if at So very early on in the project's history, actually, like Paul and my employer, like, gave us both time to work on it. That's close to 10 years ago at this point for me. Maybe it's more than 10 years ago. So it's, it's, you know, for the last 10 years at least, this has been a, you know, labor of, you know, my, my personal time. as for like, I guess how we think about it, I think. You know, I'm not, I'm not sure we have like a documented philosophy of this or something, but like, we would like working on this project to be, sustainable and enjoyable and a, product that is like, you know, advances things we care about in the world, right? Like we think, you know, crypto cryptography, the ability to build secure systems is important. Like we're, this is like a positive contribution. If we can make that easier in the Python ecosystem, if we can advance memory safety like that is, that is a, like, good thing for the world. And so, like part of what you hear when you hear us talking about like the importance of like really robust testing is that. For a thing you work on in your spare time, it's like really, really valuable for the way things come to you to be predictable and not emergencies, for example. So like, it is a thousand times more preferable to like spend some time, you know, getting a PR degree, because like I control when I'm working on that than to have like a vulnerability reported to us. Like I, you know, if you're like a company, you talk about this in terms of like shift left and like your developer productivity, but like for in your a volunteer project, like what I'm saying is like, it's really good to like not have a vulnerability get reported. Like when I'm busy with life, like I can, you know, I can take a week or two off this project because it is stable and like, that's, that's really good. and the stuff that you're working on, the time is not like. Just trying to debug a flap, ECI or like trying to fig, like wasting time just trying to make something functional and workable as opposed to like, Hey, here's a new feature that people want. I need to go ship it. Or a new primitive or making something better, like we put a lot of time to ci like, you know, it, I don't know if it's literally the majority of our time, but it, it's really plausible. It's the majority of our time, but it's almost just overwhelmingly like proactive time, like improving it so that like when there's a feature work site about like, we're just working on that feature, that's the thing you can schedule on. I mean, did David, to your like, question about like funding, I mean the, the way I think I would say it, like particularly for the last 10 years is like, I think working on this project has been incredibly, like, positive some, and I, I think it's a concept that like doesn't get, it's due these days just in the sense of like. We think we provided things that's like valuable to many companies. and even though it's the case, like at least I have not been paid for the, to work on this in quite a while. Like, I'm like very confident. I've gotten lots of opportunities, whether it's to travel and give talks or meet people or just professional opportunities because like I work on this and like that, that is for me, like an exchange of like, you know, my time and like, for like remuneration that like really seems fair and like very positive sums. So like, I, I don't know. I don't know. I don't have any complaints. How you feeling Paul? I mean, I think I found that like over the course of 13 years, like again, yeah, I got time for my employer when we first started. I have not gotten time since I left that employer, so it has been 10 years roughly since I, I was, I was quote, paid to work on this project. We, I mean, I think a lot of the sustainability for this is that, like Alex and I are a small team, right? It's, it's two primary contributors to this project, which means that when we wanna make major decisions, this is not like a large scale, long-term effort. It is a conversation between two people who largely think very similarly. that means we can, we can make big decisions, we can execute against them, without a lot of, BEReaucratic time or a lot of like, politicking to determine what's good or bad. Now, that has its disadvantages too, but so far inside this project that they've been largely advantages. it is also the case that like we have built the project such that like when people come and they ask for something unreasonable, we feel comfortable pushing back. Uh, we feel comfortable telling people they should not be, you know, they shouldn't behave certain ways in an issue tracker when they behave in ways that are inappropriate. Uh, and we. Built the system so that it's enjoyable to use, right? Like, uh, one of the unofficial rules of thumb Alex and I have used for a long, long time now is anytime the CI exceeds 10 minutes, we spend time on it because it annoys us. and so like those types of things just make it continuously pleasurable to work on. it is also the case that it is a large bully pulpit, right? Like we, we have a lot of influence and, and we are cognizant and respectful of the fact that we can use that bully pulpit, but we do want to be able to use it to like, pursue things we consider, worthwhile in the ecosystem. Y yeah. May maybe to really tie these two together. I, I think for Paul and I, a really big indicator, what, one of the things that, at least for me, more than anything, led to like us giving this talk and writing this, statement about where OpenSSL was, is that we found that working with these new SSL APIs, looking at things like argon two or ML KEM, it, it was becoming like really unpleasant. Like it, like if you had like a log of all of our chats, you would see like the profanity really went up. Like we were experiencing a lot of fRustration and like. That, that's like a pretty marked difference from, what came before, like that, that was just like, kind of not the emotional valence we had about, you know, adding new features before. And so like that, that was a big signal to us. Like, Hey, something has gone wrong. Like we, we need to take stock of like, what's, what's happening here? and so that basically leads you to the two of you. You're generally on the same page about everything, and it's not that difficult for you to finally de declare and come to the conclusion that you basically have to move away from OpenSSL for at least some things for Python cryptography. Yeah, so the, the, the core thesis of the statement we, we issued is that OpenSSL as a project has. Slowly diverged toward, from, from paths that we find aesthetically appealing, technically acceptable, et cetera, uh, to the point where we are now actively seeking mechanisms by which we could end our dependence. Now, that is a difficult thing to do. There are a variety of reasons why, like compliance reasons. We have downstream consumers that care deeply about the support of OpenSSL. We have feature gaps based on like what, what the different forks support. There's a bunch of things that make this challenging, but like while OpenSSL continues on its current trajectory, and to be clear, it is possible for them to course correct. but it is a difficult thing, based on the fact that we've spent years advocating for it. but if they fail to course correct, if they fail to, to, provide material improvement in the, on the axes that we've defined, then we will be trying to at the very least, remove OpenSSL as our default wheel configuration. And potentially based on the, what Alex and I consider sustainable for ourselves in the long term OpenSSL entirely. Do you have any other wheel configurations besides OpenSSL right now? we support build against, LibreSSL, BoringSSL or aws-lc. But you have to bring your own. We don't, we don't, uh, ship, prebuilt, wheels for those. But the, it, it's not difficult for y'all to just sort of start, experiment with just building those in.'cause you already have the work to, to link, to hook them yeah, so we're perfectly capable of doing it right now. The challenge is that like there's no, there's no concept of like wheel variance that would say like, oh, I want cryptography, but with a different backend. and even if there was that concept, which there's actually some PEPs that might make that possible, even if that concept did exist, I think that's not a thing I, I can expect a consumer to understand the consequences of, Alex and I believe pretty deeply in having this library be the sort of like drop in, in just work and it has secure defaults. And so I think like what we wanna do is get to a point where we would just swap the default. Right. So everyone is just like in, in to a first approximation. Everyone's just gonna use the default, and if the default has feature gaps, then you're randomly gonna blow things up for people. Exactly right. So like for example, the current state of the state of the world is that we sh support and ship Argon2, ID and scrypt, both which are APIs that live natively inside of OpenSSL. Not all the forks support either of those, and that meant if we, if we shipped a wheel with those, we would lose the support for those, algorithms immediately. But like it's more likely that you'll like, this is kind of a normal engineering problem, right? Like a flag day problem kind of deal, right? It's more likely that you'll resolve that problem than that. OpenSSL 3 is gonna get rid of O SSL param. I won't try and speculate, but I, I, you might be right. Yeah. a betting person, but like, it, it seems plausible. It seems plausible. yeah. It seems like one of us could just sign up for Poly Market and set this up. That sounds like a long, long, but I mean, it seems like all you have to do is violate your first rule, which you also opened by saying you've already repeatedly violated, which is just like write a little bit of your own cryptography. Uh, then, then I gotta get involved in FIPS more heavily, and that's already too much of my life. Like you, you end up with things like if you actually like go inside the Phipps. Not yet, hopefully sometime soon. i, I If, and when NIST finishes their, starts, I, I lost track of the status of their, key derivation function work, but like in theory, maybe someday. but like another example like Ed 4, 4 8 is I think not supported by any of the forks it is do you need it? do We, whether or not it was wild wise, we, we did expose it at one point. Oh Lord. And like this is one of those things where like. We could, we can make, we in fact have made the list of like, what are the things that, we ship that like we think are not stupid? Like there, there's some stuff that like, we expose that like we don't care, like I, I don't know, SEC t whatever. Like there's weird elliptic curves that like, if we dropped them, we don't think many people would care. We, we would be okay with it. But there's things like X ED 4, 4 8 or like, different kds or like, I think various like a EAD modes. Like we've got a list somewhere that like any one of them, like you might look at and you might say, ah, how likely do we think it is? People use this. We can make an assessment of like, how much breakage would it cost to do this? You know, what's the likelihood? Are there standards that implement this? Like, you know, if I search GitHub, does it like, seem like there's some projects that use it? but. For any of the given forks, we're looking at like, yeah. Decently sized list of these like, I don't know, 10, 10 or so, algorithms. And that's like, I don't know, 10 times the breakage budget of the, you know, mean of them. So it's, it's potentially non-trivial. So like Hmm. we, we would like, we wanna spend our breakage budget. Well, is I think the way, I would say it like we we're prepared to break things when we think like, the ecosystem, our users get like substantial benefits, but, we have a lot more ability to make changes like avoiding OpenSSL if we were reduce, uh, the level of breakage, that, that we incur in doing so. Mm-hmm. That does sound like, like you're incentivized to implement some of this stuff in Rust to paper over the migration, away from OpenSSL to another default backend. That that really does, that does sound like what you're it's not impossible. But there are, there, there are challenges there too because depending upon what we need to re-implement, like we don't want our downstream consumers to be silently surprised when, when the thing they thought was FIPS no longer can be, similarly, like there are, there are various like dependency requirements in, in the Rust cinematic universe of, of cryptography that are somewhat challenging for us in some scenarios. and so like one of the things we mentioned in our, in our statement was, graviola a pure Rust cryptography library. that is interesting to us in the abstract. It's nowhere near where we would need it right now. It's like we've spoken briefly with the author and like they're, that's not currently their focus for like, its adoption, but it's something that we're watching with interest as like maybe a long-term solution, as we go forward. And when I say long-term, I mean like on the 5, 10, 15 year time horizon, Alex and I are genuinely thinking in the long term here. because, and, and it's also why we spoke up when we did, because we gave it several years, but also knowing that it will take us many years to, to migrate off. We want people to be aware of the problems rather than springing this on people at the last second. Yeah. Yeah, I, I mean, just in that space of like, you know, compliance questions and libraries, I, I do, I am very hopeful that, Filippo Valsorda's work on FIPS support for Go is going to prove as a model that is very valuable. So, like, I would say in the, at least in the open source cryptography world, OpenSSL has been one of, it is like the most default choice, like maybe even more of a default choice than like OpenSSL, just as like open source cryptography is in general. Like they, they had, you know, done the work to like build the, like back when it was called like the FIPS canister. and I think Felipo has really demonstrated that it is possible if you were a diligent and knowledgeable cryptography maintainer maintainer to like take on this project. is not, you know, impossible. It's not all consuming. It doesn't even require changing your library in that like, disturbing of ways, Go's cryptography modules are still kind of like a model of like clarity. Yeah. so like, I, I am hopeful that serves as like, a model that, like organizations that have felt like I only have one choice will look and be like, oh, it is actually possible. There are other directions. I, I really hope the way that he was able to get that working for Golan will be a, a model for like a future possible Rust, like alternative, like Rust does not have a standard library. The way that Golan has a standard library, including all the cryptography that comes, including TLS, that comes with go. But I could totally see a. Library project in Rust, trying to be validated as a FIPS module the same way that the the Go Lang one did. It just would be its own project as opposed to a piece of the standard library of a language or something like that. So I've always said, instead of rolling your own crypto, you should build it for someone else and then charge them for the FIP certificate. Yeah. cool. Um, good luck in Godspeed. You've come 10 years. What's another 10? To actually get away from the fundamental, underlying foundation of your project. I'm sure Alex and I will, will be, complaining about this in our retirement, next to each other in the old folks home. Looking forward to it. Aw, that, that's sweet. Tha thank you guys very much for introducing me to OSSL_PARAM. we're, we're glad to help and thank you guys for having us. No. Absolutely. Paul, Alex, thank you so much. Security, cryptography, whatever is a side project from Dear De Connolly, Thomas Patrick and David Adrian. Our editor is Netty Smith. oh my God. What's the rest of it? I haven't done this in so long. You can find the podcast online@scwpod and the hosts online@durumcrustulum, @tqbf, and @davidcadrian. You can buy merchandise at merch dot security cryptography whatever dot com. If you like the pod, give us a five star review wherever you rate your favorite podcast. Thank you for listening.