1、Application security metrics from the organization on down to the vulnerabilities,Chris Wysopal CTO Veracode ,November 13, 2009 11:30am-12:30pm,Agenda,Why use metrics? Challenges & Goals for Application Security Metrics Enumerations Organizational Metrics Testing Metrics Application Metrics WASC Web
2、 Application Security Statistics Project 2008 Future Plans,2,To measure is to know. James Clerk Maxwell, 1831-1879Measurement motivates. John Kenneth Galbraith. 1908-2006,3,Metrics do matter,Metrics quantify the otherwise unquantifiable Metrics can show trends and trends matter more than measurement
3、s do Metrics can show if we are doing a good or bad job Metrics can show if you have no idea where you are Metrics establish where “You are here” really is Metrics build bridges to managers Metrics allow cross sectional comparisons Metrics set targets Metrics benchmark yourself against the oppositio
4、n Metrics create curiosity,4,Source: Andy Jaquith, Yankee Group, Metricon 2.0,Metrics dont matter,It is too easy to count things for no purpose other than to count them You cannot measure security so stop This following is all that matters and you cant map security metrics to them: Maintenance of av
5、ailability Preservation of wealth Limitation on corporate liability Compliance Shepherding the corporate brandCost of measurement not worth the benefit,5,Source: Mike Rothman, Security Incite, Metricon 2.0,Bad metrics are worse than no metrics,6,Security metrics can drive executive decision making,H
6、ow secure am I? Am I better off than this time last year? Am I spending the right amount of $? How do I compare to my peers? What risk transfer options do I have?,7,Source: Measuring Security Tutorial, Dan Geer,Goals of Application Security Metrics,Provide quantifiable information to support enterpr
7、ise risk management and risk-based decision making Articulate progress towards goals and objectives Provide a repeatable, quantifiable way to assess, compare, and track improvements in assurance Focus activities on risk mitigation in order of priority and exploitability Facilitate adoption and impro
8、vement of secure software design and development processes Provide an objective means of comparing and benchmarking projects, divisions, organizations, and vendor products,8,Source: Practical Measurement Framework for Software Assurance and Information Security, DHS SwA Measurement Working Group,Use
9、 Enumerations,Common Vulnerabilities and Exposures Common Weakness EnumerationCommon Attack Pattern Enumeration and Classification,Enumerations help identify specific software-related items that can be counted, aggregated, evaluated over time,Organizational Metrics,Percentage of application inventor
10、y developed with SDLC (which version of SDLC?) Business criticality of each application in inventory Percentage of application inventory tested for security (what level of testing?) Percentage of application inventory remediated and meeting assurance requirements Roll up of testing results,10,Organi
11、zational Metrics,Cost to fix defects at different points in the software lifecycle Cost of data breaches related to software vulnerabilities,11,Testing Metrics,Number of threats identified in threat model Size of attack surface identified Percentage code coverage (static and dynamic) Coverage of def
12、ect categories (CWE) Coverage of attack pattern categories (CAPEC),12,SANS Top 25 Mapped to Application Security Methods,Source: 2009 Microsoft,Weakness Class Prevalence based on 2008 CVE data,4855 total flaws tracked by CVE in 2008,Basic Metrics: Defect counts,Design and implementation defectsCWE i
13、dentifier CVSS score Severity Likelihood of exploit,Automated Code Analysis Techniques,Static Analysis: (White Box Testing) Similar to a line by line code review. Benefit is there is complete coverage of the entire source or binary. Downside is it is computationally impossible to have a perfect anal
14、ysis. Static Source analyze the source code Static Binary analyze the binary executable Source vs. Binary You dont always have all the source code. You dont want to part with your source code to get a 3rd party analysis Dynamic Analysis: (Black Box Testing) Run time analysis more like traditional te
15、sting. Benefit is there is perfect modeling of a particular input so you can show exploitability. Downside is you cannot create all inputs in reasonable time. Automated dynamic testing (also known as penetration testing) using tools Manual Penetrating Testing (with or without use of tools) Create li
16、sts of defects that can be labeled with CWE, CVSS, Exploitability,Manual Analysis,Manual Penetration Testing can discover some issues that cannot be determined automatically because a human can understand issues related to business logic or design Manual Code Review typically focused only on specifi
17、c high risk areas of code Manual Design Review can determine some vulnerabilities early on in the design process before the program is even built. Threat Modeling,WASC Web Application Security Statistics Project 2008,Purpose Collaborative industry wide effort to pool together sanitized website vulne
18、rability data and to gain a better understanding about the web application vulnerability landscape. Ascertain which classes of attacks are the most prevalent regardless of the methodology used to identify them. MITRE CVE project for custom web applications. Goals Identify the prevalence and probabil
19、ity of different vulnerability classes. Compare testing methodologies against what types of vulnerabilities they are likely to identify.,18,Project Team,Project Leader Sergey Gordeychik Project Contributors Sergey Gordeychik, Dmitry Evteev (POSITIVE TECHNOLOGIES) Chris Wysopal, Chris Eng (VERACODE)
20、Jeremiah Grossman (WHITEHAT SECURITY) Mandeep Khera (CENZIC) Shreeraj Shah (BLUEINFY) Matt Lantinga (HP APPLICATION SECURITY CENTER) Lawson Lee (dns used WebInspect) Campbell Murray (ENCRIPTION LIMITED),19,Summary,12186 web applications with 97554 detected vulnerabilities more than 13%* of all revie
21、wed sites can be compromised completely automatically About 49% of web applications contain vulnerabilities of high risk level detected by scanning manual and automated assessment by white box method allows to detect these high risk level vulnerabilities with probability up to 80-96% 99% of web appl
22、ications are not compliant with PCI DSS standard * Web applications with Brute Force Attack, Buffer Overflow, OS Commanding, Path Traversal, Remote File Inclusion, SSI Injection, Session Fixation, SQL Injection, Insufficient Authentication, Insufficient Authorization vulnerabilities detected by auto
23、matic scanning.,20,Compared to 2007 WASS Project,Number of sites with SQL Injection fell by 13% Number of sites with Cross-site Scripting fell 20% Number of sites with different types of Information Leakage rose by 24% Probability to compromise a host automatically rose from 7 to 13 %.,21,Probabilit
24、y to detect a vulnerability,22,% of total vulnerabilities,23,White box vs. black box,24,Full Report,http:/projects.webappsec.org/Web-Application-Security-Statistics,25,Future Plans,Veracode processes over 100 applications and 500 Million lines of code per month Collecting data:vulnerabilities found/
25、fixed Application metadata: industry, time in dev cycle, application type Vulnerability trends Industry/Platform/Language differences,26,Further reading on software security metrics & testing,NIST, Performance Measurement Guide for Information Security http:/csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf Security Metrics: Replacing Fear, Uncertainty, and Doubt by Andrew Jaquith, The Art of Software Security Testing by Chris Wysopal, Lucas Nelson, Dino Dai Zovi, Elfriede Dustin,27,Q&A ,
