DevSec For Scale Podcast Ep 4: Shift-Left Product Security

Product security is a big subject. You have to consider aspects ranging from design to code reviews to static code analysis to CI/CD and through deployment and scaling. Not only that, the security focus changes as you grow into a larger organization and through IPO or M&A.

Neatsun Ziv, CEO of Ox Security, formerly VP Cybersecurity at Check Point, talked with me about all these ideas and how to make sure you can visualize and ensure security in a holistic manner, from the first line of code through production, in order to build a solid product.

In this interview, we talk about the SDLC, glaring security issues that come up over and over, and we get into Neatsun’s tips for early stage companies to improve developer security.

Watch the episode below

TRANSCRIPT

Jeremy Hess: Welcome to the DevSec for Scale Podcast, the show that makes security a first-class citizen for small businesses. My name is Jeremy Hess, head of developer relations at Akeyless, the Secrets Management SaaS platform. This interview podcast brings security experts and practitioners together to offer practical and actionable ways for startups to implement security best practices using shift-left principles, without interrupting developer lifecycles.

I’m very excited to welcome our latest guest on the podcast, Neatsun Ziv. He is the co-founder and CEO at Ox Security, a forward-thinking startup in the product security space.

Now, Neatsun, you worked at Checkpoint for many years and I want to get a little bit more into your background and what brought you to Ox Security. But before we do that, let’s get into a little bit more information, something that the audience could connect with. Just from your experience, can you explain a little bit about how product security has been done in startups in the past? From your experience, what you’ve seen.

Neatsun Ziv: Sure. So, over the past years we had the experience to work with various kinds of companies, starting with fortune companies down to startup that just started their journey right now. And we’ve seen for everybody that product security is a difficult task. It requires a lot of manual processes, just understanding the details of everything that is going on, and startups are usually not equipped to do this. Especially as they don’t have a dedicated security person at that stage. It usually has a person like this when they are, I would probably say, somewhere around 100 maybe 200 employees. Only over there you start getting one dedicated security person.

So, it’s part of the engineering’s work to actually do all of this. This leaves a lot of gaps when it is managed in an ad hoc way, and this is what you usually see in startups. By the way, it’s not that when you scale with the size of company it’s perfect. You start seeing other challenges and everybody on the scale has those kinds of challenges.

Jeremy Hess: Got it. Okay, so we’ll get more into that in just a little bit. But before we do Neatsun, I’d love it if you just told us a little bit about your journey. Of course, like I mentioned earlier checkpoint for ten plus years. So, you definitely have a lot experience with large enterprise security. So, tell us a little bit about that experience and then what brought you to found Ox Security.

Neatsun Ziv: Okay, so yeah for the past ten years, I was the vice president for cybersecurity in Checkpoint. It was a huge opportunity and a great honor to amend this position. We’ve done an amazing journey with so many great people. So, that was completely awesome. Especially seeing such a successful company as Checkpoint during this journey, doing the digital transformation and starting to evolve especially in today’s cloud and product security world. That was absolutely amazing.

What we’ve noticed is that as the development organization or the builder’s organization evolve, and difference between them is whether you’re talking about just developers also DevOps and additional functions such as machine learning experts and more people, that are working to build your product security or build your application. This is where it gets interesting.

So, we started seeing that there are more and more places that can be a pitfall. Especially with the recent security events such as it started with SolarWinds and then went to log4j and during the way we had CodeCov. And on a weekly basis you would find more and more of those issues. And just over the past weeks we’ve seen Nvidia, and just I think two days ago it was Ubisoft. So, we’re constantly seeing this more and more especially with companies that are more evolved and have a bigger stake.

So, the journey that we had is actually, we started seeing this by just doing initial design reviews and pen test and we’ve added tools and this is on a broad sense, not just in Checkpoint. You started adding more and more tools and it start the process breaks. So, somebody needs to pick up everything manually and try to get it to the right developer and try to chase the SLA. And it was very, very hard to actually get things done.

We started those risks to actually deliver into production. So, in many cases we see customers that they understand at any given moment in time, any number between 3% and up to 30% of their builds, either being released to the production with known security risks or that they need to delay the builds and actually deliver the value later on in time just to fix those security issues.

So, as we saw this going on and we started to see more and more gaps in actually covering the entire ecosystem, we started to understand it’s broken and there must be a better approach to address this problem.

Jeremy Hess: Yeah, there always seem to be gaps to fill. So, it’s very important that we always think forward and it seems like Ox Security is taking that challenge head on. So, good luck with that and let’s move forward here now.

Let’s talk a little bit more about the development lifecycle. So, a question that I have here is what do you think the typical startup SDLC looks like these days?

Neatsun Ziv: Okay. So, I think that right now every new startup that we have the opportunity to speak with are starting to build everything as a cloud native. So, you’d see the code repository as being either GitHub, GitLab, maybe some Bit Bucket. Recently, we started seeing some more Azure repos and everybody chooses those kinds of code repositories.

Now, as you start to think about automating your CI/CD, we’re starting to see more and more startups building everything as an automated path from day one. So, you have the entire build system automated. They will choose some kind of a CI/CD whether it’s an integrated component, like you’ve got in GitLab and GitHub, or using Jenkins, TeamCity and many more that we encounter.

Then you need to actually script everything or build those kinds of things and you would see probably the DevOps doing this, building the pipeline, making sure that everything is in track. And then what we notice is that you start having a few good tens, or maybe even hundreds, of repositories and it becomes extremely hard just to keep track of whether you’ve done the right things just to automate the build itself. At that stage when you’ve got something already up and running, maybe you’ve got the first tens of customers, you started thinking about what do you need to add to the security.

So, I think that right now everybody is kind of using some kind of static code analysis, or something very basic. It might be an open source, it might be something integrated to your environment, and then we started seeing software composition just to make sure that you’ve got the right open source. I think that most of the repositories that you’d see right now have these capabilities integrated. So, you kind of getting it out of the box, you just need to tick the right cases.

As the organizations mature, we started seeing more tools being added, started seeing more configurations being added, more scans being added, and then you started building the value. What we’ve seen with a lot of customers is that the point in time that they actually have a plan or program saying – this is how we’re doing going to do product security – is further along. Meaning year five of the company, year seven of the company. And until then everything is just built on things, ad hocs, that people connected. And hopefully the process kind of work, but it’s not very a tight program I would say.

Jeremy Hess: Right, so we definitely need to figure out ways to put security more in the hands of developers, where they’re able to, of course, without too much effort, integrate security into their lifecycles.

Neatsun Ziv: Exactly. So, we would actually distinguish between two different stages and the stage where the development organization holds the security. Usually, it’s being driven by the CTO, or somebody very technical, maybe the VP of engineering. And there is a transition where there’s a first dedicated person just doing this as a full-time job and then things get more organized as a program. Versus, oh we have an exposure here, I’ve heard a lot about this and this why don’t we integrate this product. Then it becomes something it’s not necessarily tight in the bid to actually control the entire ecosystem.

Jeremy Hess: Got it. So, there definitely needs to be a lot of communication from one stage to the next to make sure that things are flowing seamlessly. Otherwise, you just have a drop and then you never know what vulnerabilities you’re leaving open as you move from one stage to the next.

Neatsun Ziv: I would say it’s even more than that. You don’t even know what are the issues that you need to fix. Just knowing them and fixing them is a relatively good part to be. I mean you already know and then you can choose, if you got time, what are the things that make the maximum impact.

But I think that in the beginning stages, it’s really that you don’t have a good picture. Everybody is kind of working under the assumption that – hey, we’re still small. Everybody still owns the code. The first developers are still with us. So, they know what they’re doing. We brought in some awesome guys that definitely know. But then as you start maturing you start seeing actually that the pen test can find things.

I think that the first time that an organization is doing a pen test and somebody’s actually able to show you that what you thought is right, is actually completely wrong. This is the wake-up moment, for the first time.

Jeremy Hess: Wow, yeah, for sure. So, what do you think are some of those, and we’ve talked, I think about some of them. But what do you feel are some of the most basic product security issues that startups should be dealing with, from day one?

Neatsun Ziv: So, I would split it into different categories. I think that the first one, is having the right development process, so it means making sure that you have somebody reviewing your designs, so if, in day number one you’re doing a design without any consideration of security, then as you scale, you’re going to have a lot of different issues, just design issues. It might be authentication to the services. It might be places that your databases are exposed and there’s no segregation between functions and no right thinking about microservices and how they’re communicating with each other or whether you hold your keys.

So, there are a lot of different issues, and we’re currently helping a few organizations that started this way actually to correct themselves. But, then it’s a long investment because you need to take X amount of work and then just grab and do those changes, and it’s more painful to just do it after you bake the cake. So, it’s something that you need to get into the dough before you put it into the oven.

So, we started seeing manual processes about the design and then having the pen test at the end to make sure that you’ve got things kind of in place. Making sure that you are doing code reviews with somebody that actually has an experience doing code review and has a security consideration in place.

Threat modeling, so that would be the human part of it. Then you need to think about automation. Because security by itself is just reduction of risk making sure that you won’t get hit. And it’s a completely statistical point of view saying everybody will get hit at some point or another. So, you need to think about, okay, how do I reduce this exposure in the beginning. So, you have automated tools like software composition and static code analysis and dynamic assessment and cloud security posture and so you need to lot of things into consideration. They need to understand how you can actually do it with a startup budget.

So, startup can’t afford like $700,000 buying a huge branded platform to do cloud security or doing it for just code scanning because they need a lot of different components. So, they need to think of how do they get from point A to point B, from round A to B to C, in the most secure way, with the least amount of efforts in terms of headcount investments and hopefully also capital investment. So, people then tools and processes. Those are things that definitely we need to take into account.

Jeremy Hess: And of course, we all know that at the end of the day, you could have the tools and processes in place. But if the people aren’t following the guidelines that’s usually kind of like the lowest common denominator of issues when it comes to attacks, right? I mean you’re finding a way through some sort of manual process that was done, somebody put some code they didn’t realize somewhere, put it in production, whatever it was, right?

Neatsun Ziv: So, I’d probably say yes, people is the last component of the three. But I think that if we’re counting on people to actually do this, we didn’t build a process in the right way. The reason for that is, in most startups, there is a huge growth in headcount. How would you think about doing security training? You’d probably say yeah when somebody boards, I’ll do one day on security. Maybe have some kind of program. Maybe a learning system. But then as you get to work three months down the road from that point in time, you’re going to forget all of those things. You didn’t understood everything. You took some assumptions that somebody that did your code before you maybe already done and you’re not going to understand this and get into the details.

So, definitely counting on people to understand and follow up is extremely hard. I don’t know a lot of places that this model actually works. This is where a guided security is always the case. So, we see companies, more mature companies, having guilds of security champions and more security architects to help with this. But this stays usually in the theory level and the translation from theory to actual work is super complicated. I know just a few companies that are really hitting this on hand, making sure that they can actually influence this on a daily basis. Super hard, super high investment, I think this is a way for post IPO companies and software startups.

Jeremy Hess: Yeah, definitely. So, taking that same thread a little further, you know, talking about automating as much as possible and taking that stuff out of human hands. How does Ox Security, and you can give us a little bit of more detailed possibly if you want about what you do and how you do it, how do you make those processes more automated to ensure the security of a product?

Neatsun Ziv: Okay, so I’ll take you one step backwards. So, what we see is that there is a ratio between the amount of dedicated security personnel versus builders. So, the question is where are you in this ratio? Think that the more mature and security-oriented organizations would probably have a ratio of any number between 40 developers to one product security dedicated person. In most cases, you’ll probably see the ratio in the ranges of one to 100, maybe one to 200. Then you get to the place that you can’t do everything right. You don’t have the time to do this. You don’t have the time to track it. You don’t have the time to innovate and think about how do I actually automate.

So, as organizations scale, you start seeing more and more developers as part of the security organization to actually automate those processes. So, this is where the organizations are maturing, and when you look on the, I would say probably most innovative software security companies, or software companies with security orientation, you would see huge amount of development just to automate internal processes. What we’re trying to achieve is actually getting you a platform to do product security with this automation. So, you won’t need to invest so much headcount and time and thinking to get this right. I think it’s transformation that we’ve seen other, I would say places or markets, such as the XDR and incident response and SOAR, so, we’re started seeing this in other fields, but product security is kind of lagging right now.

And for us to do this, and from, I would say probably 200 customers, that we have been working with in some kind of capacity, they’re kind of repeating this. Only about 10% of them feel that they’ve got this right, but this is with huge investment. Meaning some of them might have 20, 25 dedicated developers, as part of their security team, just automating things.

We think that there’s a better way to do this from the beginning to the end. Taking into account what should be the framework. How do we get people to actually drive the KPI of security? How do we make sure that you don’t have any exposures throughout the entire process? What would be the meaning of security that you say I feel comfortable with this level versus the investment that I’m willing to do at this point in time? And then you can actually play the balancing, okay… I want to automate this.

These are the things that Ox can help you starting with integrating to your CI/CD, making sure that security is not something that you constantly need to ask, I would say, favors, organizational favors, from other entities such as the DevOps to integrate to your pipeline. Having a central place controlled by the security organization. Making sure that you’ve got this right, or a single person from engineering can actually see the picture. All the projects associated with a security architect, security champion. So, you can have everything covered. Instead of you inventing things, we can do this for you as a full journey.

Jeremy Hess: Great. Absolutely. Sounds good. So, one of the questions I always like to ask security startups is, do you eat your own dog food, right? Do you implement your own product in your own processes?

Neatsun Ziv: Oh, so definitely. So, we are in the stage that all of our developers are using our IDE plugins. So, as they’re developing, they can actually see the issues. We’re tracking the KPI. So, we’ve started just implementing the KPI, I’ll probably say a month ago. So, that is month number five for the company right now.

So, when we’ve done analysis, we started seeing graph going up for the first four months on security exposures. We deployed the IDE and then we started seeing that the line is not going up, and actually now we’re starting to see it going down.

So, every company that we installed with so far, you see a continuously accumulating graph of security issues. And what we are helping to do is actually stop this curve, making sure that there’s awareness and get this KPI down on the risk exposure. Actually, working with the builders, explaining to them within the IDE, how does this help, why is it important, what’s the easy fix that we can offer them. Making sure that we eliminate secrets, we eliminate code issues, software composition, deployment issues, cloud issues.

So, you can actually get a full picture per project or per application saying, this is how we ensure that there’s a good security. It helps with the reporting, it helps with dialogue, and most importantly it helps with the discussions that security needs to do with developers and the negotiation of saying, is it important, why is it important, what’s the context, it’s not connected to the internet.

We’ve been in hundreds of those discussions and it’s tedious, and we try to avoid this by actually showing the proof in the initial stage. Saying that, this is the exposure, this is why we think it’s important, this is the potential impact, and it’s really easy to fix it, let’s just do it together, we’ll show you how, and maybe even do it for you.

Jeremy Hess: Might even automate a solution for it or have something that’s already built in that would just automatically close some hole that was found pretty quickly.

Neatsun Ziv: Yeah exactly. Just making sure that the developers themselves can have an easy access to the solution that will synchronize the expectations of security and the actual developers. Any places that you don’t have the security dedicated. We’re actually providing those issues or policies out of the box.

This is what we recommend as a best practice and as you will scale. We want to make sure that we prepare you security-wise towards the IPO. So, you won’t get to the place that you’re going to have a lot of security exposures. We’ve seen even recently a few companies that were pre-IPO that had a security incident. It really hurt them in the preparation for the IPO.

Jeremy Hess: Sounds like it could definitely be a burden for sure. So, the last question I’ll ask is something that I have to ask all of our guests, which is, can you give us one or two tips for improving developer security specifically, that they will be able to implement relatively easily now at an early stage, that would, with minimum amount of disruption to developer work and developer lifecycles?

Neatsun Ziv: So, I would probably start with just thinking about security and what does it mean to you as a company. So, it’s not about doing this or a certain tool, it’s about thinking what is the framework that I’m adapting. It might be something extremely simplistic from the beginning. It might be just something like, all I want to do at day number one is make sure that they’ve got code review just as a beginning or maybe some kind of a design, a secure design. I’ll probably say start with the basic, just make sure they don’t have any architectural problems. Those are things that are extremely hard to fix later on, especially while you’re scaling.

Taking headcount from current investment in creating features and reallocating them to do refactoring, that is usually something that everybody hates. It’s better sometimes to invest a few more hours in the design. Maybe even hiring somebody to do this for you, just as a security design review. Just start with this. If you need one thing to begin with just start by understanding what you need to do. It’s way more painful.

Then I’ll probably say there are easy checks to do. You can do them with the open source if you don’t want to invest a lot of money, such as making sure dependencies, just basic vulnerabilities. Those are things that can be done without any investment or can be done with a very low investment. I think that there are great ways to do this. By the way, we are offering a free tier for startups that actually can get this from us without any payments especially as they are just starting their journey so they can have everything secured from day number one.

I think it’s super important especially in day number one, or in year number one, of the company, to build team front. That’s the first number tip just not tools, not processes, just making sure that you’ve got two things in place, that’s it.

Jeremy Hess: Got it, fantastic. There’re some very valid and solid tips that I think every startup should definitely consider looking at from day one. So, Neatsun it was really great speaking to you. Best of luck with Ox Security. Looking forward to seeing where you guys land in the next couple months and then in the years forward. So, good luck with it. And hopefully we’ll have you on potentially for another talk once you’re at the next stage.

Neatsun Ziv: Hopefully so. Jeremy, thank you very much. It was a pleasure.

Jeremy Hess: It was my pleasure and have a great day.

Neatsun Ziv: You too.