SOC2 with Sarah Harvey

We have Sarah Harvey (@worldwise001 on Twitter) to talk about SOC2, what it means, how to get it, and if it's important or not. The discussion centers around two blog posts written by Thomas:

• SOC2 Starting Seven: https://latacora.micro.blog/2020/03/12/the-soc-starting.html
• SOC2 at Fly: https://fly.io/blog/soc2-the-screenshots-will-continue-until-security-improves/

Transcript:
https://securitycryptographywhatever.com/2022/10/16/SOC2-with-Sarah-Harvey/

Links:

• Tailscale recent post on getting SOC2’d: https://tailscale.com/blog/soc2-type2/
• SSO Tax: https://sso.tax
• David’s previous job: https://getnametag.com
• David's other startup: https://censys.io
• Thomas works at https://fly.io

"Security Cryptography Whatever" is hosted by Deirdre Connolly (@durumcrustulum), Thomas Ptacek (@tqbf), and David Adrian (@davidcadrian)

Transcript

Deirdre: 0:12

Hello! Welcome to Security, Cryptography, Whatever! I'm Deirdre.

David: 0:17

I'm David.

Thomas: 0:19

I'm Thomas.

Deirdre: 0:21

And we have a special guest today, which is Sarah. Hi Sarah. Hi. And David. I think we're talking about security audits today,

Thomas: 0:33

We're going hard on the Whatever

David: 0:36

Yeah. Since it's audits, this is clearly the Whatever and not the Security or Cryptography parts. Uh, but before we get into that, I just wanna take one opportunity to say that, um, a year ago on this podcast, I was right cuz there was a Matrix bug and I was like,

Thomas: 0:50

One time.

David: 0:51

I was like, Yeah, just one time I've been right! I was like,"Wow, this protocol seems like it's really weird if this type of bug can exist in it". And I think I said it was not just a straight like sender ID to AEAD key lookup and, uh, low and behold, There's some bugs, um, that we're not gonna talk about today, but I just, I just wanted to do a little self-validation here. Make it feel like I know what I'm talking about,

Thomas: 1:14

You wanna call it on Matrix being a steaming crater

David: 1:19

Um, on the matrix protocol being, being not what you want,

Thomas: 1:23

Not what you want.

Deirdre: 1:24

What was the bug a year ago?

David: 1:26

eh, some sort of like, you could change out the key from underneath people. I don't remember. But anyway, that's a topic for a future podcast.

Thomas: 1:34

What's the topic for this podcast?

David: 1:36

The topic for this podcast is SOC2, um, to set, or security, security through rules, we'll call it. So set the stage for this. Thomas has done a bunch of writing about SOC2, some of which I've kind of followed because I've done aspects of SOC2 at two different startups of various stages in the last three years. And Thomas, you did SOC2, I assume, for a plethora of startups when you were doing, I don't know if I'm, it's COOs to call it consulting at LED Cora, but consulting. And so let's talk about the LARAs two starting seven that you wrote a few years ago.

Thomas: 2:17

Sure. So in late 2016, me and my wife Erin, and Jeremy Rouche from Matasano started a company called Latacora, and Latacora runs whole security teams for startups. Um, so me and Aaron were at Latacora for about four years. And during that time when I was, when, when we were there running security teams for startups, when people would call in, you know, as prospects, as people we potentially work with, we'd asked them like, what are the reasons they're calling in? And we heard SOC2 a lot. It was like a major driver of people reaching out for kind of consulting help, which is kind of what put it on my radar to begin with. We're running a consultancy, we're trying to like get people to, you know, let us work with them to build a security team and one of the big things that, that big things they wanna SOC2. So I should probably pay attention to what SOC2 is. And then like over time, over the four years that I was there, I had the experience of working with a couple of different companies as they went through their SOC2 process. I'm like, like peripherally familiar with kind of formal audit stuff. Like PCI is probably the best example of something I had exposure to before. And like, like everybody else that is familiar with kind of like standard PCI security audits, I have contempt for them. And I have like roughly the same kind of take on SOC2. Like I kind of figured SOC2 is gonna be roughly the same sort of variable thing and it's, it turns out it's a different terrible thing. It's, it's not the precise, terrible thing that I thought it was gonna be. So like, the weird thing is like, you know, we have this first client that we're gonna do, you know, an actual SOC2 project with, we're gonna actually be the technical support for our real SOC2. And we're all nervous about like, you know, what we have to be ready for and all the ducks that we have to, you know, have in a row for the SOC2 auditors. And then like, we go through their SOC2 and like, we have these calls with these auditors and they're just taking screenshots, which is like, I think, I think everybody notices and like, I, like, I, I noticed that like we were explaining things to our auditors that I wouldn't have expected to explain to a security auditor. Like, like what is an IP address or what, what is a URL? Um, like, literally like, like they were going through log lines. This is not us. This is not like, this is like one of my clients, um, who shall remain nameless, but like their auditors, like, they take a screenshot of all these like log lines, and the log lines are obviously like their HTTP logs, right? Like, here's the URL somebody was hitting and, and we're like, Okay, well if you look there at the URL you can see, and they're like, Well, I'm gonna stop you right there. What do you mean by the URL? Right? It's

Deirdre: 4:43

Uh, so, Okay. Okay. Okay. I, I know shit about fuck about security audits of this sort. I know about a person who writes code, looks at like a cryptographic specification and they look at the code you wrote and they tell you where you either did or did not implement the thing you're supposed to or where you made an error when you're doing it. I know about that kind of audit. What is this kind of audit and who are the people doing the audits if they don't know what these things are that you're telling me?

Thomas: 5:13

Here's the mistake you have made, The thing you're describing where somebody looks at a piece of code and finds, you know, looks and sees if there's problems in piece of code, that's not an audit, right? Like in the rest of the world, like in the rest of the universe. That is like that thing you're describing is not called an audit. An audit is by definition, according to the official rules of accounting. You know, all the accounting standards like, like the actual textbook definition of an audit. Is the process conducted by somebody who doesn't know what a URL is.

Sarah: 5:41

I I mean, you, you can also like make the analogies like the same as getting your taxes like audited, right? Or like whatever situation audit, your

Deirdre: 5:49

Okay. Okay, So

Sarah: 5:50

So like if you think of it from like the bookkeeping perspective, it's like bookkeeping and we have taken it to security. And actually, you can see from a lot of the auditors too, right? Like you, you see that bookkeeping like background coming in, which is why they end up being like, Hold on, what's a URL?

Thomas: 6:08

And I be a little careful, right? Like, um, it's funny to put it the way I, I think it's funny, I think it's funny to put it the way I just put it, but, um, I, I like our auditors at my current company a lot. I would have to say that, wouldn't I? But, I, I do, I do really like our auditors, right? They're not super technical and I think that's like the most important thing. If you're not familiar with SOC2, like if I was gonna like take a random security person and bring them up to speed on SOC2 and pick just a couple of things to say, right? The very first thing I would say to them is, get rid of whatever notion you had of what kind of person is doing this audit and how much like, Background they'll have on your engineering replace that completely. Right? Like these are really more like accountants. Their practice kind of arms them with like a list of kind of high level things. Like they have a playbook on things they're looking for, and those can get into some engineering detail, but like they don't have first principles understanding of that detail. Right? They have a first principal's understanding of accounting for sure, of what an actual audit looks like and how to like, you know, reconcile documentation, how to do samples and things like that. Like the audit part of it. They've got down right. But the engineering staff, they'll have a lot of engineering detail, but none of it is first principle stuff. So like we had that experience and it's like you do a couple of those and then you hear people talking about SOC2 and what you have to do to be ready for SOC2, and it's like, well I have an answer for that, right? Like I've been through that a couple times. So we wrote a blog post, it was called the SOC2 starting seven, which was like our kind of like Joel test type post. Like do these seven things to get ready for SOC2. And it was a really short list. And I think the moral of that post was the things you need to do to get ready. The technical things you need to do to get ready for SOC2 are things you should generally just be doing anyways. You know, set up protected branches and get so that no one's, you know, pushing directly to master, right? Like that seems like a really small thing. Like not a major security thing. Like maybe you're not doing it, but like turning it on is not that like it's not a huge project or whatever, right? But it's like most of SOC2 is that kind of stuff. In fact, of all the things we've done at my current company for SOC2, Protected branches is the thing that was most disruptive. It's between that and single sign on and you're pretty much there. I would bet that any company, any company trying to get a SOC2 audit that has SSO enabled for most of their applications and protected get branches, no matter what other decisions they made. They could get a SOC2 no problem. And you know, if I'm picking a person at random and trying to bring them up to speed on, SOC2, there are other things that I would tell them. I have to think like, probably my second thing would be to question why they're getting SOC2, and talk about when you should get, SOC2, cuz I feel like people do it way too early. I think the point at which you do, SOC2 is the point at which you're literally forced to do it, where the cost of doing SOC2 is substantially less than the deal. You will close immediately when you can give somebody a SOC2 report. that's, and that's deceptive, right? Because you can get a long ways just by promising people I'll eventually do SOC2. So it's not the point at which you're doing conditional pos on SOC2. It's the point at which you literally have to have, SOC2 done is the point at which you should do it, right? So you hear about like, you know, five person companies that are, SOC2, because it's getting easier to do SOC2 or whatever, right? And like, you know, if I have to pick if I have to pick a second thing, yeah. If I'm picking a second thing, it's, you know, finding those people and shaking them by the shoulders and saying, Stop. Don't do it.

David: 9:19

well as, as somebody who got a SOC2 certification, um, or did most of it at like a five person company before, uh, before I went to Google,

Deirdre: 9:28

Why did

David: 9:28

the.

Deirdre: 9:29

do that?

David: 9:30

The cause of that is you, like we were in the like, consumer identity space and we had like PII, so people were more willing to ask us about it. And then the other problem is because of, well, I don't know about problem, but because of companies like Secureframe, companies like Vanta that automate a lot of the checking of compliance with SOC2 in terms of like how your cloud and, and uh, SaaS products are configured. The bar to do it is, is arguably much lower than it was before. I'm sure that y'all at Latacora were very efficient at doing, at it, doing SOC2. But in terms of like people that hadn't been doing primarily security for startups or hadn't specifically done SOC2 before, you know, it's still, it was still a bit of a lift even if you were trying to follow best practices and then the existence of those companies makes checking a lot of the boxes and coming up with the evidence that you would provide to auditors. Um, a lot. And so, because it's easier to do than people expected earlier.

Thomas: 10:35

Yeah. It's just, let's be clear, right? It was easy for us when I was consulting to do SOC2 technical support for our clients, right? That was very simple. Doing SOC2 was a lift, right? It's just that we weren't the people that had to do the list. So like there's a, there's a shitload of like just stupid paperwork that you have to do of just like business fundamentals. Remember, it's an audit in like the conventional sense of an audit, so like your financials and your company documentation comes into it, right? We didn't do any of the real SOC2 work. We just did the almost non-existent security technical work that supports SOC2.

David: 11:09

Yes. And you need to do things like make sure everybody has acknowledged and signed the, their like employee handbook and the update process for the employee handbook and send everyone through security awareness training, which, you know, depending on how you read it could just be asking every employee if they're aware of security, uh, and, and all that kind of stuff.

Thomas: 11:28

So I mean, I guess like, you know, we have Sarah on the podcast with us and we brought Sarah in because we like Sarah and also because David went around looking for people that had opinions on SOC2 that may or may not be completely in conflict with mine. So that was like a shotgun blast of random SOC2 stuff for me. But I'm curious what Sarah thinks about what I've said so far.

Sarah: 11:54

So a lot of the elements of what you said that I agree with in that there are probably a bunch of companies that are getting into SOC2 more, too early and maybe they don't need to do it. And is it really like a good indicator for what you should be doing from a security perspective? For a company, especially for a startup, I can provide a different perspective of how I experience SOC2, which is primarily through like doing vendor security evaluation, of which I did quite a ton in a previous role. It turns out, yes, if you are SOC2, you want a lot of your vendors

12:29

to also be SOC2. But we found

Sarah: 12:31

in, in terms of giving out as part of, So I'll go in and take a step back and explain like the whole vendor security process. You know, you go in the process for engaging with a, a third party vendor, right? For some kind of SaaS thing or service or software or what have you, right? You have a few different elements. You have the financial element as like, okay, all the invoicing, what is the pricing, whatever. You have the legal perspective, like, uh, what is the contract? Any certain kind of NDAs do you want to do? Especially now with GDPR things do you, do you need to have some kind of protection or data privacy agreement as. Um, and then the third part is like, do we have some certainty that this vendor is just not gonna fuck up with all of our data once we send it and as soon as we send the data, it's all gonna go on the wide open internet? And it turns out when you are engaging with a wide variety of vendors, right, you have this like extreme, broad like spectrum of the type of companies that you're dealing with. You have on the one hand Google where you say, Hey, fill out a vendor security questionnaire. And they're like, Are you fucking serious? Sure. This is, and like every single like comment in that vendor security form is, has just this light amount of snark that is just like, I can't believe you're asking us to do a vendor security questionnaire. And then you have the other end of the spectrum where you submit one and it's like, Hello, I'm a grad student doing some data analysis and I don't know what encryption is and I don't know what, like data, secure data at rest or secure data in transit or any of the terms are, I don't know what data segregation is. I don't know how to do login securely. And so you're like, okay, what, what is like our minimum bar and how do we ensure that there's some kind of validation we can do? It would be nice if we had, you know, through a whisper network, which does way may or may not exist of like, is this company likely to be half decent insecurity? Such that even if they don't have a SOC2, it's fine. But when you're dealing with like. Reading a lot of vendor security requests and trying to figure out what's gonna make sense and what's not gonna make sense. It's way easier to just be like, are they, So do they have a SOC2 Type And if the answer is yes, it's like, okay, they're probably somewhat got their shit together, like enough that they're not high up on our list of vendors to worry about in the event that there's like a data leak somewhere.

Thomas: 14:57

Well, there's like a, there's, there's like a lot of funny stuff in that, right? Like I, I get exactly where you're, I was, I'm not shooting anything you just said down. Right. But

Sarah: 15:04

I mean, you can, I would not care. That's why we're on this podcast.

Thomas: 15:08

Yeah, of course. So like you mentioned, like if somebody has a, SOC2 Type 2, they probably have their, you know, their shit at least somewhat in order, right? So like in SOC2 there is this notion of a Type 1 and a Type 2. Um, and it's not complicated. The difference between the two of them, a Type 1 is a point in time thing. You, you essentially can't flunk a Type because the Type 1 is just a document saying, Here's what you were doing at this point in time. Now, like there's a set of things you're supposed to. Up and running. When you get your Type 1, basic access control and a security policy in place, like I guess you, you wouldn't get a Type 1 if you for some reason refuse to have a security policy, then you'd have a problem, right? But for the most part, it's like they come in, they say you need these things. And I'm like, I only have three of these things. And they just say like, Here, use these templates and now boom, your Type 1 so you can't flunk it. The Type 2 is the same subject matter as that's Type 1. It's the same audit, but the Type 2 is backward looking. So what they do is they like, you know, if you say, you know, we have documented onboarding offboarding procedures for all of our employees, right? When they do the Type 2, they're gonna say, Okay, well give me, you know, three random people that you offboarded in the last year and then show me the documentation for how you off boarded them. And if you can't show that documentation, you get like a variance or whatever. You get some demerit in your SOC2 Type 2 report, which ostensibly people look for, right? So that's type 1 Type 2. Type 2 is the, obviously the harder one to do because they actually look to see if you were consistently doing stuff. But it's the same stuff, right? So the first thing I would say there is that my understanding, we talked to a lot of people about SOC2, both when we did the consulting article that I did a couple years ago and more recently when at Fly io we started a SOC2 process, got our Type 1 and all that stuff, right? And several very clueful, fairly large, pretty sophisticated security teams that I talked to had no problem taking Type 1 reports from people. One very sophisticated, very excellent, very strong team, has a vendor that gave them five consecutive Type 1s. Just they never, Which by the way, if you're listening to this podcast and you have to do something with SOC2, like the cheat code I just gave you is pretty powerful, right? Like anybody can pass the Type 1, there's no work for it, you don't have to prepare for it, right? And you can give, you know, other people's vendorsec programs, consecutive Type 1 reports it'll be fine. Right? So to all that, I would just say like if, if we're saying like, You know, SOC2 set some kind of bar. Does it set a bar? We were joking right before we, we kind of went on the air about how like one of the impetus for this is like everyone, everyone's on the security team, everyone's vendor sec dream is that they'll write a single master questionnaire and then when people come in asking like, you know, please fill up this stupid vendor questionnaire, right? Like you'll uh, you know, here's our master questionnaire that we all pre-prepared for. You're all set and no one takes that, right? That's the running joke in the industry is like you can spend the time doing that, but people won't accept it. Right? But like if it's between a SOC2 Type 1 report or a SOC2 Type 2 for that matter and a prewritten master questionnaire, both of those seem to convey the same value to me.

Sarah: 18:07

Assuming that the questionnaire is like filled out competently.

Thomas: 18:11

I might push back even on that, right? I think if you have enough clue to know that you should have a master questionnaire, it might not matter to me.

Sarah: 18:19

That's true. Like if, if you have enough foresight to be like, I should pre prepare some questions that people there are button the shade that people are gonna ask me and just straight up answer those questions, you're probably ahead of the game. More than most people who, when given a question, I was like, I've never had to do one of these before. I'm just kind of YOLOing these answers and hoping for the best.

Thomas: 18:44

And so you were in a vendorsec program and you were reviewing like this kind of documentation before. Okay. So I have a question for you.

Sarah: 18:51

All right.

Thomas: 18:52

When we were doing our Type 1 fairly recently, right? One thing that we ran into, we we're, um, we're a distributed team. Everyone's remote. We have people kind of all over the world. We have a bunch of developers in like Sub-Saharan Africa. Like we push people everywhere, right? And one of the things that SOC2 wants you to do is background check. Um, it's just like, one of the things auditors know to ask for is have all of your employees been criminally background checked? Right? And if you have employees in like France, you can't background check them. It's a, it's illegal in France to do that as an employer, right? So like,

David: 19:24

Good job, France

Thomas: 19:26

The auditors know that. They're like, Oh, there's a, there's a bunch of language we'll use for like, you know, your US employees are background checked, but like your, you know, European employees won't be, or whatever. And like, you read that and your natural response is like, Well, fuck that shit, right? Like, why am I letting the French people off the hook? Right? Like, if the French people don't have to be background checked, then none of our US employees are, too. Which was the, the, the tactic we took with it, right? And I asked a bunch of people if you got a SOC2, like the, the auditor's response was, it wasn't that they wouldn't give us a Type 1 without background checks. It was, well, people are gonna look at this Type 1 report or these, these Type 2s that you give them and see that you don't background check and they're gonna push back on it, right? And I asked a bunch of people about this, and no one really seemed to care.

Deirdre: 20:04

No one actually looked at your, your SOC

Thomas: 20:07

Yeah, so that's my broad question, right? Like, so Sarah, when you, when you're looking at these things, are you looking carefully at what they, I don't know, like because you're doing lots of them, like how careful are, do you think that the typical vendorsec program, you're doing everything, you're reading Absolutely everything. I believe that a hundred percent. Right. But if you had to guess the typical vendorsec program, are they even looking at the report?

Sarah: 20:26

I think it depends on what they focus on. So there's a lot to tease out of, out of this question of like, like in terms of, you know, you've got a company that's giving out, asking the, the third party providers that they, they wanna work with, like giving out these like questionnaires. I think a little bit of, of background and context setting is for this company that is setting out questionnaires, like what kind of company is it for it to really care? On the responses from the reports, because one of the things that I think is, is hard to tease out is a lot, I think a lot of people in the security industry understand, like background checks can be kind of nonsense, right? Like there's a lot of people from a variety of backgrounds that like, this is just simply not fair for them to be held back because something bad has come up as part of a background check. But separate to that, there are companies that are held to specific standards relating to like, especially if it's like a financial company. Like there are certain government regulations that are in place that simply say you need to have done due diligence and background checks on all of your employees, otherwise these employees cannot work at your company. Right? And those might be companies that also, like, either they have like a, a blanket exception for someplace like France, or maybe they don't, they just, they specifically don't hire in France. So like that kind of background check requirement may be useful. It happens to be useful in a soc. SOC2 report, but it might be that the person or the like security team that is like dealing with that, they don't really care to check about that too much and they care for the background check from the perspective of this is a financial company that is subject to other regulations. And from that perspective, like the legal team is gonna really care about it, but like from the security perspective, we don't care. We we're mostly caring about like, is the data gonna be secured? Does it seem like they have a good understanding of how their environment is set up? Do they have a good understanding of how security works in their environment? Do can we trust them that like data is gonna be actually secured properly and handled properly and not gonna lead that onto the wide open internet? Does that make sense?

Thomas: 22:25

Yeah, that makes, that makes perfect sense. Right. I guess like my, my mental model there is the SOC2 is a thing you get to get past, like, call it procurements or like, you know, kind of mechanized vendorsec, where it's like there's no conversation to really be had with some of these companies. You just have to have a SOC2, or they're, you're not gonna get through their vendorsec process. And then at the point where somebody's like, Well, we really care that your employees are background checked. That's a conversation you have with like your head of security on one end of the phone and their head of security on the other end of the phone. And you ha you hammer something out that goes in the PO where you make a specific commitment. Like the employees that touch my data will be background checked or something like that. Right.

Sarah: 23:02

Yeah.

David: 23:03

I don't know specifically about background checks, but there are, you know, having worked at a couple places where like the buyer was a security team, um, or might be a security team, there are situations where if the buyer is a large enough company that can kind of throw their weight around a little bit. It might even be that they will find things to complain about simply as a way to exert power over the other person because they get their willies from it and that this is like reasonably common and sometimes this will pop up, like reasonable concerns. It's like the classic, the, The Onion,"Heartbreaking, the worst person you know made a great point!" type situation. But other times they'll just be looking for things to complain about. And in that situation, I think, you know, complaining about a background check would might be just something they could complain about. And then you do exactly what you said and then the people that wanted to complain, get to complain and you get to continue not doing background checks. Even if you had done everything to the T, like some people will still find things to complain about. So you may as well take an opinionated stance. That's my opinion.

Thomas: 24:08

Yeah. So Sarah, you've done larger scale of vendorsec stuff than any of the rest of us. I've done like a very little bit of it, but not like a huge amount. Right. So I guess like the burning question for me is, so I get to choose kind of what goes in my sock. You report, like I get to, you know, kind of choose the controls, the, the rough like outline of a SOC2 is like they have high level kind of access control, right? And like, so they have like this kind of goal, like the security goal of like, you have some access control and you have controls that are mapped to that goal. And you could, you could roughly kind of pick what those controls are, right? So are there specific things, like, this is the big question I have about this whole thing for people that do vendor check is like, if I get to pick which things I'm gonna do in order to satisfy this, like which controls I'm gonna actually implement and claim in my SOC2. Do you care? Like, what are the specific things I, uh, what are the parts of the report that you're looking at carefully.

Sarah: 24:55

I meant you, you're gonna hate my answer because it's gonna really be like a million"it depends" kind of situation here. we're not gonna look at it like super closely if say there's no like, customer data going into it. Like I, I don't know, like a hypothetical example here is like say some manager is like, uh, you know, it's the pandemic. I want to do like some kind of offsite, I'm gonna engage this vendor called like, uh, Gather Town to host like some weird offsite. Like, are we really going to dig through into the SOC2 report and see how access control is done? If it's just gonna be like 10 employees playing a board game, virtually? No. Like, it does not make any sense.

Thomas: 25:33

Yeah, assume they're getting your Google auth tokens. Like serious, like, uh, like

Deirdre: 25:37

Oh.

Sarah: 25:37

Y Yeah, so like for, for more serious stuff, we would look really carefully and, and see, like, I think the thing that we mostly take a look for is, I, this is really weird to say, but it's like, does it sound like they know what they're talking about? Do the sentences they like, are the sentences they're putting in this report, do they make factual sense or are they completely nonsensical? Right. And we've had cases where people write stuff like write a bunch of of sentences hoping to kind of pull the wool over our eyes. But it turns out in, in the situation where I was reviewing vendor security questionnaires and reports and what have you, um, we had security engineers doing it so we could spot from like a mile away that this was clearly some people putting some buzzwords into a sentence and hop. That it would, it would pass through and then that would be where we would actually, if it wasn't clear from that report and we were dealing from with super, super sensitive information, we would actually go and try and hop on a call to be like, can we get someone who is like not a director and not a salesperson, but like an actual engineer to explain this to us, and then we would kind of do a check that way. That was really rare to do because it was extremely time consuming and it turns out they really don't like you trying to get hold of one of their engineers. But the companies that did allow, like us talking directly to their engineers, they were often the best to work with. It

Deirdre: 26:59

I don't know, man. Like I would take it gladly. If we have a customer that wants to talk to me and I'm a security engineer or even just an engineer, I'd be like,"Hey, I like you customer. You actually give a shit."

David: 27:13

is not the common reaction.

Deirdre: 27:17

I mean like finding time on my calendar.

David: 27:19

Sure you're not a product manager like

Deirdre: 27:22

Man. I like people who care about good security.

Thomas: 27:26

you're no longer allowed to talk to customers. I've demoted you from the talking to customers.

David: 27:31

I think it's great when people like wanna do that, but I, I don't think that's the common reaction.

Thomas: 27:38

It's, it's, it's not that, like I actually, I I kind of, I'm, I'm weird. I, I actually sort of enjoy filling out questionnaires cuz like I'm a preening asshole vulnerability researcher type. So it's like, I think I know, I think I know better than everybody that wrote those questionnaires to begin with. So I can just like,

Sarah: 27:51

We can see your kind of responses in the questionnaire like responses and we're like, This guy knows what the fuck he's talking about. For sure. Look at how, look at how proud he is of simply everything and like the amount of detail and how he's talking about certain standards are wrong and we're using this standard. And I was like, Yep. This guy certainly is on podcasts and he reads like the latest and greatest, like we can tell from a mile.

Thomas: 28:19

Yeah. This is the, that's the, the nice way of putting all of that stuff. And I like doing those calls for the same reason. It's just an opportunity to pre right? But like lots of people want that, that contact, right? Lots of people want to talk to a security engineer and get like, you know, warm fuzzies about what you're doing. But like where the rubber hits the road is what things people need in order to make a purchase. And the only people that need that contact to make a purchase have a crap load of money, right? Like, it's almost guaranteed if, like, if they need to take your SOC2 report and then get on the phone with an engineer to confirm it, then they are price insensitive, right? And so like, if you know that about a customer and you have a convenient way of actually applying a charge to a price insensitive customer, then you do it right? Like, so, so like at the point where somebody's reaching out and saying, you know, can I talk to a security engineer to, you know, get some details about the SOC2 report? I guess like you, my eyes are lighting up too. It's just that my eyes are lighting up with dollar signs of them.

Sarah: 29:15

I will note, I think that was a problem we did face later down the road where, I mean, we, we ran into some situations where like you would pass the security check and then we would go into the negotiation phase of like the, the money. And actually I think it, it depended, We, we changed the program like a couple of times. There were a few times where we locked in, like what the, the price contract was likely gonna be, and then it got hung in security, uh, or in, in vendor security, uh, assessment. Um, and then that's where it fell. And we've had the other way around where like people were like, Well, vendor security is taking way too long because they really wanna talk to every single person, every single security engineer at every single company apparently. And so then they would kick that off and then it would get later get hung over in, in the like negotiation contract phase where it's like, oh. this company has interpreted the intense zeal of this, the, the buyer, uh, from the security perspective as a, apparently a black check. And we have therefore screwed up in the process.

Thomas: 30:14

Like from what you said about, you know, for a sensitive, you know, vendor or whatever, you know, you look fairly carefully at the SOC2 report, and then if you have concerns, you set up a call. Right. That makes sense to me. And my takeaway from hearing that process is that I should never sweat. Whereas in my SOC2 report, I shouldn't worry about it ever. Right. Because the worst case is I'm just gonna wind up on a phone call that 80% of the time I'm gonna end up on the phone with anyways. Right. Like, so it's, it's like you're not gonna bomb out of, you know, a procurement or something like that over vendorsec just because of literally what's in your SOC2 report. You're just gonna have one extra call.

Sarah: 30:46

Yeah. You're gonna have one extra call. Let, let's put it this way, Like, like if you just fill out the questionnaire and there's no SOC2, we'll probably ask you for a call, but basically like if we're asking you for a call, there's, there's a hidden perverse incentive on our side, which. We're getting the big signal that someone really wants to purchase this vendor, right? Or someone really wants to, to move forward with this agreement because we have already tried pushing back, being like, This is really alarming. Or like, this is not like at the standard we want. Are you really sure we want to move forward? And they're like, Well, what's the concerns? And we're like, Well, these are the concerns. They don't have enough information on this. And then we'll be like, And then they'll be like, Why don't you get more information? In which case now we're doing the call. And then it turns out maybe like we've had situations where it's like the salesperson has filled out the vendor questionnaire. The salesperson has sent us a document that is really just a PDF with a picture of a SOC2 badge on it. And said that to us, which is absolutely not a great indicator of like what we actually, or not really good information what we need whatsoever. And so sometimes it's just us product to be like, can we get someone who has more engineering and security context to provide us the actual information we need in order to make about assessment? We are definitely treating a lot of potential vendors in extreme good faith, maybe more than necessary. Um, but that is because at least on the team that I worked with, um, we really wanted to give everyone the benefit of the doubt. And we knew that there were many companies that were like the traditional big companies that like maybe did have everything sorted out, but feature wise they were not. There was like some things missing that our teams, uh, our teams wanted and so they weren't exploring new vendors. We also. Fairly aware of the fact that like when you're a small, scrappy startup, the only way you get into the well known company phase is by getting more customers. And there's like def definitely different phases to that evolution, right? And so you may not have all of the things that the other companies have yet, and we're willing to give you that chance to kind of build up so that, you know, you, you're now actually competing with every, everyone else on, on a fair game. Like I don't think anyone actually like thought that way, but it certainly, like we engaged with a lot of much smaller vendors from the perspective of recently wanted to avoid some big ones because the smaller ones seemed kind of cool and they were not quite all the way into the level of security maturity that we would typically expect for a big vendor. But maybe we could like work with them a bit more to understand their security posture and we would have enough confidence from that that everything was gonna be fine.

Deirdre: 33:25

Yeah,

David: 33:26

I'm hearing is Thomas should replace a SOC2 report with a PDF of a Word document of Word Art that says he has SOC2

Sarah: 33:33

Yes, basically. And then he will immediately get a call because if anyone knows who thomas is, they'll be like, Thomas, why do you have a PDF with a SOC2 sticker on it?

Deirdre: 33:45

So that you'll just jump on a phone call with me and we'll just skip right to it.

Thomas: 33:50

I mean, I guess among like the big SOC2 opinions I now have, like there's a thing in the industry called the SSO Tax, which Rob Shaheen like maintains this website called SSO Tax and it's, what it does is it spells out the price difference between like the standard account and the account that you need to enable single sign out, like SAML or whatever, and all these different, different providers and some of them are crazy, like the markup is like 2000% to get into the account class where you can turn, you know, SAML on. And like the idea there is that this is a tax, right? It's a tax on people for security, right? And like you can go back and forth on, you know, the validity of the SSO tax, whether it's like literally people putting a tax on security or whether it's, you know, the people that actually need SSO are paying for, you know, the cheap accounts for everybody else, right? Like there's just different ways to look at it, right? But I, I can understand the argument where if you're charging people extra to have single sign on, then you're encouraging people not to do something that they need to do for security. You are kind of putting an extra cost on just security,

Deirdre: 34:54

And let's be explicit about why Single Sign On is better for security for your org than having a different login for every single vendor that you use,

Thomas: 35:05

I mean, I think all four of us are, will probably agree. You can't run for, for a serious company, you know, of any real size, um, without some kind of single source of truth for, you know, identity and authentication. You, you can't run that. It'll just be crazy. You like a single place to, you know, turn people's access off when they leave and, you know, set policy over who can access what. Um, the big one is like you, you want a single place where you can enforce, you know, phishing proof, you know, multifactor, you know, authentication, right? Like if everything is single sign on and a single sign on does, you know, WebAuthN you're kind of done, right? Like just mandate it and then, you know, move on. And if you're managing like 300 different vendors and you know, trying to make sure they all support web on just give up, it's not gonna work.

Sarah: 35:44

We, we definitely, for a lot of our vendors, we would go ahead and ask like, Do you support SSO? Do you support like SAML? Do you so that we can integrate with us? And if they didn't, we would say, Can you implement it so we can? And that was often actually one of the bigger like indicators of they were not ready to like work with us because it was just like, Do you know how big your company is? Do you know how many like vendors we have? It would be simply impossible to track and manually off board every single employee. At any time, like someone left the company from like all of these vendors at once, so that it was often a deal breaker for, for some vendors.

Thomas: 36:25

And if you're, if you're a product manager, like David is and I was, right. Like you look at that and you say, Well, what you just spelled out is the reason why it's associated costs more, right? Is because the people who need it, the people for whom it's make or break on the purchase are less price sensitive. They're larger companies, and so they don't care as much about the monthly cost of a single seat or whatever. So,

Sarah: 36:45

I mean, sometimes we did, but if the cost of your monthly seat, my favorite were like cost per employee per month, per day or per week. It was like all kinds of ridiculous like intervals sometimes. We would take a look at it and on the surface would be great, but then we'd realize like maybe you only 25 employees needed it. And so it's like, okay, do we really want to pay it for say, 10,000 people at the company? And it was like, well now that's an enormous price difference. And that's probably a whole other discussion of like, how would we handle that with like sso.

David: 37:16

I've seen Jira extensions that are like$11 per employee per month, and it's like, no, Like you're just not gonna pay that to get like LaTeX in Jira or something

Deirdre: 37:26

Yeah.

David: 37:27

when like eight people are gonna use it. And you have like a hundred or a thousand people at your company,

Thomas: 37:32

You'd have to pay me to let latex be in JIRA.

David: 37:35

Yeah. There's a lot to unpack from that whole statement, but,

Thomas: 37:38

but, but it's like, you know, it's like business class pays for the back of the plane for, for all of the rest of the people sitting in the plane. Right. And I mean, I, I think it's also, I think it's fine to say like, I, I'm, I'm kind of, I'm showing some of my cards here. Right. But I think it's kind of, it's, it's fine to have a set of product offerings. We've switched off of vendors here at Fly because of the price to enable SSO because we really need everything to be sso and like we use like, maybe like the, they have a suite of products and we only use one tiny little one that we like a lot, but like, you can only turn on us. So if you opt into all the rest of the products, well it's like, fuck it. Okay, we're just not gonna be able to use this anymore. Right. But I don't, I don't moralize it. Right. It's not gonna work for us. We're just, it's not a good package for us. We're gonna, you know, pick somebody else and we're happy with it. We're all adults in business and this is fine. Right. But like, there are people that very much moralize the, the SSO tax thing. Right. But I get it. I get why people moralize it. It's very important to have SSO to begin with. Right. But compare that to SOC2 where nobody needs soc. Like nobody, it doesn't make any difference whatsoever. It's not a very good standard as we've established, right? Like, and it's not gonna make or break security anywhere, right? So why would I give that away for free? If you're asking for SOC2, then you're big enough to have a vendor stack program. So like, it seems like a reasonable strategy to me is if you want to be generous about it, then you write up your own report that is basically isometric to, or isomorphic to what was in the SOC2 report, but it's, you know, not audited, you know, But you don't, Nobody cares. You're giving the same information away. Give that away for free. And then if people really need the SOC2 to file in their vendorsec program, it's like, well you need to be in the, you know, advanced super awesome, you know, account class to get that right. It's so, it's, I guess it's thing I like about SOC2, I guess I'm realizing it's a thing I like about SOC2 is that it's a security feature that you can gate on, you know, an account or something like that, that doesn't take away security if you don't have it enabled.

David: 39:28

The problem is as, as it goes down market for because of like Secureframe and Vanta and so on, it's no longer as good of an indicator of a like oracle of does this company have a lot of money or not, Because it might just be five people at a shed and they might only have like, uh, a little bit of money in seed funding.

Thomas: 39:47

this, yeah, this, this gets back to like, I think people don't push back hard enough on getting, So we, we waited a long time, um, to SOC2 here. I still think we probably did it too quickly. If I think about it, we weren't forced to do it when we did it. We could have, we could have waited. It was just like, we kept getting questions about it and we closed the business anyways. But like, you know, we're like any day and hour to get somebody, we really gonna want to close and we're gonna need SOC2. And then we pulled the trigger faster than we should have and we're reasonably like compared to like the seven people in a shed or whatever, we're reasonably large. Right. There's also like, I have lots of misgivings of, I like the, there. Particularly at Vanta. There are a lot of people at Vanta that I like a lot. I don't hate anything about Vanta or whatever, but I have like deep misgivings about Vanta as our product. Just in terms of like the approach that we took for SOC2 versus the approach that I've seen people take with things like Vanta, where if you're not familiar with Vanta, Vanta is a Y Combinator company. There's like the official way, the semi-official way that every YC company, SOC2s, right? Um, because they're all kind of in the same club together and it manages all the evidence collection for you. So like, you know, it gives you all the checklist things like when you're doing a SOC2 kind of on the fly, the basic process is that your auditors will give you a giant spreadsheet checklist called the DRL, the document request list. And it's just, it's like, it's basically the security questionnaire from hell and you fill it out and that's, and then they come back in and with your answers, they take screenshots to back it up. And that's the whole audit, right. It's kind of like a pre DRL where it's like a bunch of things you fill out before you get an auditor, and then when your auditor comes in, like they, you just plug them into Vanta and they don't talk to you. They just pull things out of your Vanta. And part of the way that works is they install agents on your systems to like automate the, the evidence collection from your servers. Which I mean, right off the bat there, that's, you shouldn't be doing anything resembling that just to get SOC2, There be other reasons to do agent based stuff, right?

Deirdre: 41:34

gives me the

Thomas: 41:34

two, that's for SOC2. It's like it's, it doesn't matter how good the agent is, it's just not enough of a value prop to add an agent to things. But that's actually not my Vanta problem. Right? My Vanta problem is that it pregenerates this, you know, big long list of, you know, good security things that you should be doing and filling out, but SOC2 doesn't require almost any of that stuff. Right? If you follow kind of the happy path with things like Vanta, I'm not just talking about Vanta. There's like, this is what these products look like right now, right? They're like security questionnaires where it's like you fill these things out and plug your evidence in and they're basically guiding you to the set of security things that you should get done, right? But. My take is, and I, I think Sarah's backing me up on this is that it does not matter what is in your SOC2 report, certainly not to the degree of granularity that Vanta is having you kind of

David: 42:16

I'm sure that's what Sarah wants the takeaway to be for.

Sarah: 42:20

Actually that that brings an interesting I, so I didn't know about this product for what it's worth, but like you describing, this product has very strong similarities to a lot of products I'm seeing coming out in like the privacy compliance space where for a while I was. What is going on with all of these companies that all seem to pop up and want to install agents on everything to suck in all your data, to send to the cloud and then from there generate like some kind of report. And I'm like, I understand. Yes. For privacy. I know, I know. I just wanna know that there was like one vendor I talked to who I asked is this on prem? And the answer was, Well, you connect us through like an AWS vpc. And I'm like, That did not answer my question at all.

Deirdre: 43:12

that's an orthogonal question.

Sarah: 43:13

that's, so it makes me wonder if that is the new trend for all the compliances that there are a lot of different companies that are kind of selling this kind of tool, which is install some agents on your thing because you, we know how much you hate doing things like basic inventorying and things like taking screenshots and things like capturing states and then piling it all into some like system that auto generates a report because also the people hate generating the reports because it turns out every like year or half year or whatever interval you're doing this audit, your engineers have changed all of the interfaces. Right? And can't find those systems anymore.

Deirdre: 43:57

What was the, like the inventory management tool that everyone had installed and then it got backdoored by a supply chain attack and it happened and it was very bad?

Thomas: 44:08

Solar Solarwinds

Deirdre: 44:09

that this is reminding me of that I don't like it.

Thomas: 44:13

I'm not saying that any of my concerns with Vanta are about solar and I

Deirdre: 44:17

No, no, no. But just like install this thing on all of your stuff just so it's easier for you to, to compile a report later. Like the irony of privacy stuff report aside. It's just sort of like, ha ha.

Sarah: 44:30

I mean, I probably would've had less problems if they actually offered an on-prem solution because it would be quite compelling. But like just the number of people were like, Well, but we need to utilize our magical machine learning in order to make things work. And we obviously can't give you the magical machine learning blog. We have to send it

Deirdre: 44:47

Trade secret or some crap.

Thomas: 44:51

Like, as I remember it, you don't have to use an agent with Vanta. Like you just fill out a bunch of stuff. Like a good motivating example of my concern is this, we went into our SOC2 expecting to have to do kind of end point fleet instrumentation. Like that's a thing we had not done before our SOC2 was, you know, have a kind of a rigorous standard for how we instrument people's dev machines and things like that. We're better at it now, but at the time it was a thing we expected, like when we were planning the work that we got to do to get our SOC2, like we, we figured, okay, this is a thing we're gonna burn a couple of weeks, getting all this like, you know, getting OS query ironed out and all that stuff. And then we went in and talked to our auditors. They're like, well you know what the trend now and, you know, 2022 or whatever. I guess yes, it's maybe 2021, I've lost track of time. Um, but the trend now is that people don't do endpoint in their SOC2s anymore. So it's like we have this stuff for like MDM and endpoint fleet monitoring in our plans. It's like well just strike those. It's not gonna go into your report. This is not what people do anymore. Right. But like if you, if you go through the Vanta checklist, there's definitely endpoint stuff in the checklist there. Right. And you don't have to do it like it's, no one's gonna bounce your SOC2 report for not having endpoint in it. Cuz like fairly expensive auditors are telling people not to do it anymore. Right. So like my concern is isn't, I mean the agents think bugs me, but you don't have to use the agents. But my real concern is just that people are spending a lot more time on their SOC2 than they need to because they're following kind of these, like these checklists that are not lined up with the reality of how SOC2 actually gets used. If that makes sense.

David: 46:17

suspect that there's still net saving time versus doing it themselves because they're being handheld through the process. But I think regardless of, like, I, I think you're, you're to some extent, like it's a race to the bottom in the market in the sense that like these things are here now and they're pushing the time from founding to SOC2. The time to SOC2 is getting shorter and shorter and shorter and like I'm with you that you should probably wait longer, but because SOC2 is infectious, because once you're SOC2, you have to ask for other people's SOC2, as long as a small percentage of people start doing it earlier and earlier and earlier, like it's just gonna make the problem get exponentially worse. Right. So I guess my advice to you is get your price discrimination while you can.

Sarah: 47:01

Do you know what? My favorite thing though about SOC2 and all the other compliances is just that a lot of them have huge overlaps in the kinds of questions. You cannot like easily copy, paste the answers from one to another. That bit is infuriating.

Thomas: 47:14

It's all infuriating. I agree. That bit is infuriating too. I, I think there are worse things than spending more time and money than you need to, to get sucked to. And one of those worse things is spending ongoing more time and money on security stuff you weren't gonna do otherwise or you would've done differently that you're locked into because you had to do it for SOC2 when we were first talking to people when we first, like when we were at the first blog post Right. And kind of circulated out there. Um, this is the SOC2 Starting Seven posts where we're just like giving you the seven basic things, like these are the only security things you should do to get SOC2, which I think we were indicated on, like, I think apart from the fact that you can strike MDM from that list, turns out Yep. Nope. You don't have to do a lot of stuff.

David: 47:58

I think it's interesting that if you like look at that blog post and you delete like your header and footer paragraphs that are about, SOC2, when you just look at the seven things, they're all like, for everything that we've just knocked on sOC2, those seven things are all like roughly the things that I would do if I was in charge of security or engineering or both at a startup, right? Like that's a good list of seven things to do.

Thomas: 48:18

Like on Hacker News for that blog post, I would get two kinds of orange feedback for it. The first orange feedback was, SOC2 doesn't require you to get single sign on. You don't have to do single sign on to get SOC2. Dramatically misses the point in one direction. And then there was the other people like, Well if these people don't take SOC2 seriously, then they're like, you know, they're not being serious about their security programs. And like a good security program is informed by SOC2 and it gets better because they're doing all this compliance stuff, which is wrong on the other direction. Right? So like I've talked to lots of people who are like, Well, one benefit of SOC2 is that it brings you up to speed on security stuff and it forces, you get serious about it. It's like, I just wanna slap, I'm like, Like false, like Batman, slap GIF, right? Like it's, it's not making you better at security. Like you get better at security by taking security seriously, by staffing and building a security practice. But like, if you're setting the template for your practice based on what your SOC2 auditors are telling you to do, they don't know what URL is,

Deirdre: 49:12

Yeah.

Thomas: 49:13

How are

David: 49:14

you're setting it based on Vanta or Secureframe, like you are gonna end up doing these seven things, though. So in that sense, like

Thomas: 49:21

Yeah. But you'll do a lot, you'll do a lot of other things like, you know, network security monitoring stuff that no one really does.

Sarah: 49:27

Yeah. The, the other thing too is that like, this is one of those cases of like intentions versus expectations where you go in with a particular parts of SOC you are, are designed with the intention of this is what the shape of a good security program of the company would look like. But now we've just translated it into this checklist and so someone coming in not having like maybe either with just like yellow intentions or just not treating security seriously are just gonna be like, this is a checklist. I am going to go and blindly pick each thing. Or do each thing in isolation without better understanding of the whole. And that is the bit that is quite unfortunate and and dangerous sometimes for these audits and, and for these like compliances is because if you had good, like all of us here are good intention security professionals, right? We go and take a look at this and we're like,

David: 50:21

outta the four.

Sarah: 50:23

we look at this and we're like, I see why this was designed this way. Right? And I see why like how this like shows all the different like major security pieces, like to the point that we have a blog post about like, these are the things you need in order to do And these are just generally good security practices. But not everyone thinks that way. Right? And the trickiest. Is when someone gives you a SOC2 report, you have to really figure out, is SOC2 really that indicator that they're taking security seriously or have they just blindly treated as like buzzword bingo or like a series of checklists to blindly follow. And sometimes for some folks, that is really hard to figure out.

David: 51:05

So how would we make suck to, if we were just redrawing the rules with no regard for anything? Like how would we make SOC2 more aligned with someone is actually taking security seriously. Whether that's by changing the audit process to changing technical controls, to changing something about it. Like how, what would be a, a approvals, uh, or certification program that is more of an indicator that you're taking security seriously than stock two? Or is this as good as you can?

Thomas: 51:38

We're, we're probably all pretty clear at this point that the only people that we would trust to evaluate are security practices and get a good bead on whether we're doing a good job are our peers. So what we want is some kind of, you know, peer based thing where like other people can sign off on, like, you know, and if you trust my peers that I get like, you know, you know, you want that kind of like peer-to-peer trust certification thing. So it seems kind of obvious to me that what, what, what you want here is the blockchain,

David: 52:09

Uh, you really had me on a hanger as if we were gonna go to GPG or to blockchain. Um,

Sarah: 52:15

Oh yeah, I was gonna say I knew exactly which direction it was going, but yes, I was trying to figure out whether it was GPG or blockchain.

Thomas: 52:22

it's you want as compliance NFTs, right? So, I don't know. I don't know that you can do a better job than what it's doing right now. Like I, to the extent that SOC2 forces people, it effectively forces people to turn on SSO and to stop letting people push directly to master, you know, and to have centralized logging. Which it doesn't really, Most people are gonna end up with centralized logging as a result of doing sot, right? Like to the extent that it nudges people towards doing a bunch of things that everyone should be doing, like it's doing its job and then really doing security certification for people. I don't think anybody's ever done that effectively. I have, I have no faith in it. There was an effort a while ago, and Sarah's probably more familiar with it than I am, but there was for a while, a consortium of people. Hammering out a standard security questionnaire. It had a name and a brand and everything. I forget what it was, right? But the idea was there's an industry standard questionnaire. You just fill this thing out and then you give that to everyone. That's like, that seems more credible to me than, than a certification thing. I kind like, I, I sound like I'm dunking on, or I'm trying to dunk on SOC2 auditors. I kind like the structure that SOC2 has, where like, there, there isn't a whole lot of technical subjectivity to it. It's not like a, a nerd pissing match. It's just people that know how to review and track documentation and have enough domain knowledge to, to roughly know what the documents are about and just, you know, getting people to write things down and then having those things be binding. Like I think it works well. And that sense, I just think, you know, we should be more realistic about what it means, right? Like that. It's not really setting a standard for what you should be doing for security. You still have to know what you're doing for security. And the only thing that really bothers me about SOC2 is the extent to which people that are just now trying to figure out what their security practice is gonna look like and who they should hire and what those people should be doing are taking their cues from something like SOC2 as opposed to really talking to people about what real security programs look.

Sarah: 54:16

I mean, I can add a little bit to that, which is that I had someone reach out to me recently. Uh, it wasn't SOC2 specific, it was actually PCI who started asking me a bunch of like specific security questions about how to handle data and in order to be like, compliant with pci. And I was like, Well, you are asking me enough questions. That hints to me that you need someone who's thinking seriously about security. To actually come on board to your startup. And so like from that perspective of like, if they're dealing with people who are wanting SOC2 or if they're starting to deal with like specific compliances and they're like, I think one indicator of like, actually we might need someone who is more full-time on security on board, one indicator is we're way in over our head, like trying to deal with this paperwork and like someone needs to come in and actually help explain to us what we need to do. Like I, I sent them like some blog posts and stuff of like, here's like some security people I know who've written like, how to security for your startup. And they're like, This is way too much. Can you just tell us the answer? And I'm like, I cannot tell you the answer in a Twitter DM in the middle of a work day right now. So you need to go and find some

Deirdre: 55:25

people Yeah, that's a big signal. It's like, no, you need to hire someone now.

Sarah: 55:31

Yeah. So from from that perspective, it's, it can be good to force people to start thinking about hiring good people for security. The, the key word there, the, the like word that is carrying a lot of weight is the word good? Because there are people who are in security who could be helpful questionably, so right and end up leading you totally down the wrong path. Uh, but you could still get SOC2 certified anyway. Right? So

David: 56:01

If you're not part of the solution, there's money to be made in prolonging the problem.

Deirdre: 56:08

Wow. So everyone's right, They're both useful, but also they could be bad

Sarah: 56:14

see, this is what I mean. It's the worst answer I can possibly give, which is,"It depends", right? So like, it's, it's depended on a bunch of different factors on like who, you know, what kind of company you are and like what size you are and like how many of the people that you're interacting with really take this seriously versus not.

Deirdre: 56:33

Yeah. Huh?

David: 56:34

I look forward to reading Fly dot io's Type 2, and marking it up with a pencil next to the pool, um, on my next vacation. So Thomas, please do send it to me when you do

Thomas: 56:44

Why would we, why would we ever do a Type 2? Everyone's just gonna accept our repeated Type 1s anyways. Like, I don't know, we're kind of ruining SOC2, whatever good it had. Cuz the other thing that SOC2 let people do, um, is if you already have a security team, right? And then you have a large rest of the company, the security team occasionally battles with then SOC2 is also a really good club for the security team, right? Like no one knows what, what's in SOC2, right? And if you look at like the standard DRL or a standard Vanta questionnaire or whatever, there's a lot of things in it, right? So like security can kind of pick and choose which of those, but it gives them a blank check to get a whole bunch of stuff done. Cuz the sales team really need SOC2 to happen. And so it's like, well you have to do all the things I wanna do cuz we need'em for SOC2. And I'm saying fortunately like, you know, the company managers that would benefit from this advice don't listen to our podcast. But if they were, I gave them a cheat code too, which is you could just tell the security people to go fuck off. Cuz SOC2 doesn't care about any of that stuff. Right. I also just wanna say before we wrap up that like, I feel like I spent. Maybe 10 minutes flagging Vanta. And I don't, I don't mean to do that. Right? Like I, like I said, I like Vanta people and I also think that like, I think there's, you probably would benefit from using Van to get SOC2 with the proviso that you go in knowing that you don't have to do. Everything that Vanta tells you by default to do. Like you should have a clear idea of which security things you want to do and can do sustainably, like when you do it. This is probably the most important bit of SOC2 advice I have for people doing SOC2, which is when you do your Type 1, minimize what you do on your Type 1. Absolutely restrict the scope of what you do in your Type 1. As much as possible, do not overachieve in 1 because whatever you do in your Type 1, if you don't do it in your Type 2, it's a variance or whatever they call it, right? Like there there's no, you don't lose any points for not doing extra stuff in your Type, in your Type 1 for your Type 2. But you do lose points if you do extra stuff in your Type 1 and don't keep doing it in your 2. So like, I'm sure Vanta is probably super effective for lots of companies. Like it'll, it'll probably get them through stock two a lot faster and also definitely cheaper. I've talked to lots of people about how the prices work out after the audit and everything and the whole Vanta package of doing Vanta and then doing the audit is cheaper than what we paid for our, you know, kind of bespoke, you know, SOC2. But whatever you do, make sure you're doing the minimum you possibly can in your first Type 1. You can add things in later. It's really easy to add new security controls. Very difficult to take them away.

Sarah: 59:06

Especially if you've made a mistake and you're like, That was an overly restrictive security control that has made our whole engineering team hate us. I wonder if we can remove it. Oh, wait.

Deirdre: 59:15

We can't, the auditors will yell at us

David: 59:21

My advice, never say that you're going to require two LGTMs that are non sticky on PRs. That's a way to get everybody to hate

Deirdre: 59:30

God. Oh God. Yeah. Yeah. Once upon a time I had to do an integration between two version control systems, and I had to sync up the, the collect, the auditing of the approvals on every patch and like all this sort of stuff. And yeah.

Sarah: 59:48

We should also add, be careful about having hyperactive security engineers who really want to be part of an audit because they've never seen one before, because they may also volunteer extra information that just causes the auditor to ask even more questions because they don't know. Now you've mentioned something like I, I, can't think of anything smarter than like a URL, but basically they've mentioned another key phrase that the auditor is like, Wait, what is that

Deirdre: 1:00:12

Tokens.

Sarah: 1:00:14

Tokens. Yes.

David: 1:00:16

Send those engineers to go talk to the customers.

Sarah: 1:00:19

basically

Thomas: 1:00:20

The audit itself is so boring. Any security engineer is only gonna do that once. Cuz once you through it, it's like, why would I ever sit through that again? I had another thing to say and it flew outta my head. So,

David: 1:00:30

So let's just end here.

Deirdre: 1:00:31

Yeah.

Thomas: 1:00:32

yes,

Deirdre: 1:00:32

SOC2. Don't do it until you absolutely have to. Sock one. Do the bare possible minimum.

Thomas: 1:00:43

we should have called the blog post Teenage SOC2: Don't Do It,

David: 1:00:49

All right. Well thank you to Sarah Harvey for coming on the podcast

Thomas: 1:00:53

Sarah. Thank you so much.

Sarah: 1:00:55

Thank you.

David: 1:01:02

Security, cryptography, whatever is a side project from Deirdre Connolly, Thomas Ptacek and David Adrian. Our editor is Netty Smith. You can find the podcast on Twitter@scwpod and the hosts on Twitter@durumcrustulum,@tqbf, and@davidcadrian. You can buy merchandise at merch.securitycryptographywhatever.com. Thank you for listening.