Thoughts on Starting an Application Security Program

In the course of our work, our CEO and founder, Christopher Emerson, is regularly asked to provide his opinion and insight on a variety of topics.  Clients and industry contacts are interested in getting feedback and direction from someone who has ‘done this before’ and who takes a very positive approach to building out security programs.

This happens often enough that we realized that it might be helpful to take some of the topics that are most often discussed and put them out to a broader audience.  So, we conducted a series of interviews…

That’s what follows - a portion of one of the interviews.  We took the questions that we’ve heard asked repeatedly to Christopher, organized them a bit into logical groupings, and then asked him again, but this time recorded his answers.  We hope that you find this series of posts useful and informative. [Please Note - we have edited things to get rid of side discussions (unless relevant), complaints about recording equipment, etc..  We’ve also generally just cleaned up.

What are the first steps that an organization should take in starting an application security program from scratch?

I think a lot of it starts with getting an inventory.  There’s a lot of great discussion these days about asset discovery, asset tracking, and maintaining a good CMDB, but in addition to that, when you are starting an application security program make sure that you’re including people, their skills, your existing processes, and policies.

The more information you have the more you can review and see what you have to work with - what kind of baselines do we have?  What is the foundation that is already here? Take that and trying to understand what is working and what isn’t working…. The best way to get that feedback is talking with the people who are doing the work, or are impacted by it, today.  

If you are fortunate enough to already have team members who have been doing some of this work - regardless of the type of security testing, or if you have the opportunity to talk with some of the developers and product team members, reach out to them and ask:

  • What’s working for you?

  • What’s not working for you?

  • What are your pain points?

  • How do you suggest changing things?

Are your results inconsistent?  Are your recommendations not applicable to the environment?  Are your findings not repeatable? DDo your customers understand what they are being told?n

Try to get your initial baseline on what you have and then poll your stakeholders to find out what works for them.  I believe that’s where you have to start.

Tools vs. Training - where to start?

I’m going to take the consultant way out and say ‘it depends’.  But it depends on who you are targeting with your application security program.  

Tools have more inherent scalability, so if you have a large group of people you are looking to enable, (i.e. multiple product teams), tools are a great way to start.  They have enough stuff on their plate with delivery, meeting deadlines, getting their stories completed, getting their QA testing done… But if you can help them by providing them easy tools that they can work into their current delivery process (waterfall, agile, etc.) easily your going to see return.  

If you’re targeting your security team,,, I would start asking them.  ‘What do they feel they need? Where do they think they need help? Do they not have tools to do their jobs effectively?’  That may be true regardless, but they may say ‘hey, this particular training is going to give me more job satisfaction then this tool that will increase my throughput… 10%.’   So getting that feedback from your team… I think that’s a big part of leadership - you have to delegate appropriately and then trust your team members to provide that feedback.  

These decisions will ultimately vary from org to org, but, it has to be part of a conversation and dialogue with those impacted stakeholders. Their input should help guide those decisions.

What are the first metrics that you would recommend tracking to show adoption and/or program progress?

When you start a program from scratch a lot of it depends on what you want to deliver.  What is tracked is what will be delivered.

There is a book called “Measure What Matters,” by John Doerr that discusses Objectives and Key Results (OKRs), and I feel that is a great way to look at your metrics.  They need to enable support what you are trying to achieve (Objectives) and be measurable (Key Results), and ultimately, they need to help you and your company make decisions.

In application security, some of the key metrics I prefer to start with are:

  • Number of Applications

  • Percentage of Applications undergoing security review (this can be broken down into types)

  • Number of security defects identified per application (broken down by severity)

  • Number of security defects identified categorized by OWASP Top 10 Category

  • Average time to remediate security defects per application (broken down by severity)

  • Percentage of developers & engineers who have undergone secure development training

Aggregating this data and watching it trend over time can help identify opportunities to make strategic decisions that will impact the overall security of an organization’s applications.  For example, if you notice that the largest quantity of security defects is in the the OWASP A1 - Injection category, you can start to identify if, as a company, you should provide the developers more training around preventing that issue in their code, or if there are technical controls you could enable to help remediate that for all development.  

The trending data will then let you identify if your changes have been successful, if you see less and less Injection defects being identified over time.

How can you successfully promote the application security program or getting others outside the program to see its value?

So that’s a tough one…   As much as those in security try not to increase the workload of other teams that’s basically what you are doing - you job is generating additional work and overhead for other teams.  That can sometimes be a tough pill to swallow… It’s up to you to try to find effective ways to communicate and build a culture where these teams feel empowered and able to fix these issues and have a desire to do so - that they are properly incentivized.  

Some of the things that I’ve seen work really, really well in that space are having security champions - essentially deputizing people who have interest in security from each of the product teams to get them to be that first line of security contact.  As developers are working through their stories and grooming the backlog that person can speak up (and feel confident enough to speak up) and say ‘That is something that has security implications - let’s make sure that we’re doing the appropriate security stories as part of that.’  They also need to feel confident and comfortable coming to the security team and saying ‘this has reached a point where I’ve reached my level of knowledge on this topic and I think the team should bring you in for additional feedback.’ So trying to empower people.

One of the ways that I’ve found to get people to want to participate as security champions would be something that we’ve called ‘ride-alongs.’  As we’re going through and doing a security test, whether it’s source code review or dynamic testing or a full-blown penetration test, setting aside some time on the schedule to have a developer who has interest to sit with you and understand your process - ‘here’s how we go through and find indicators of potential vulnerabilities, here’s how we start performing the research to identify whether these are true security defects or are they false positives, here’s how to create exploits for these types of vulnerabilities to show the true risk to the environment in question….’  

We’ve gotten really great feedback from the engineers and developers who sit with us - it helps to open their eyes - we hear things like ‘I never thought this could happen in my code’, ‘now that I’m seeing it I understand how you guys are doing it….’.  It kind of gets the security bug in them where they start wanting to do some of this on their own. Then it’s up to the security people to empower them - provide them with tools and guidance and education to allow them to do more on their own.