The Dumbing Down of Product Security Certifications.

In the past any industry desiring to provide technology to government agencies was compelled to submit their products to be certified. For each country and government market the company would have to go through a certification process. The certification was required to be considered as a qualified vendor in the acquisition process. Given the numerous countries and agencies, with multiple security certifications, this was very expensive. That is each certification was very expensive –in terms of money, labor, and time. It was not uncommon to hear companies lament ‘the certification would complete around the time the product was obsolete’. In the end, fewer and fewer companies were undergoing certifications.

To address the cost / time issues, a number of countries agreed on a Common Criteria (CC) for security evaluations. This was later adopted as ISO/IEC 15408.  The CC had a number of unique elements. The first was that evaluation assurance levels (EALs) 1 through 4 would be mutually recognized by all countries participating in the CC recognition agreement, while higher levels (5-7) were not. The higher levels were destined for items that required high assurance levels – i.e. military or other sensitive areas. The Common Criteria also introduced a process called Assurance Continuity.  Assurance Continuity enabled a product to be updated and then re-certified in a fraction of the time for a fraction of the original cost, with certain restrictions. Finally the CC changed the security certification philosophy from one of ‘Did you implement a security feature with these explicit functions in this exact manner?’ to one of ‘What security feature did you implement and how did you implement it’ or ‘‘Did you implement a security feature with these explicit parameters’. At the end of the certification process the certificate, as well as the supporting information (Security Target (ST) or Protection Profile (PP), were publically available. Having the ST or PP and the results of testing published was found to be highly valuable to consumers in understanding what was certified and if it was applicable to their environment. It also helped in understanding which product was potentially more secure. For example if two products were certified, one at EAL2 and the other at an EAL4, a consumer could review the ST or PP documents and see what was done. This is important as each vendor may have certified different security elements in their products. Hence the EAL4 might have only certified one element or feature while the EAL2 might have certified 24 elements or features. Thus depending on your requirements you might choose the product that was certified at the lower EAL2 as opposed to the product that was certified at the higher EAL4. If the feature sets were the same, then you might choose the EAL4 certified product as demonstrating greater confidence in its correct implementation of the features.  In general the CC Certification process provided sufficient information, for any entity (government and non-government), to determine if their security requirements were being met by a certified product.

So what’s the problem? A number of years ago one certifying agency decided to only focus on certifying products with STs at or above EAL4, or for a product being sold directly to a government agency at a lower level ONLY if there was a PP. Then a few years ago they decided to only certify products at or below EAL2. Now, they are pushing to validate products without a specified EAL (which turns out to be EAL1). Why validate and not identify an EAL instead of assigning the applicable an EAL1 – EAL7? It would seem that they believe the consumers are being misled by the numbers. The scenario they point to is one in which the acquiring party is not reading the PP or ST, but is just looking at the EAL number associated with the certification. This agency is concerned that a consumer might buy a product with an EAL4 which had fewer security features implemented or claimed than a competing product that received an EAL2. This sounds bad, however if one is performing an assessment of product capabilities (i.e. reading the ST or PP the product was evaluated against) this would not be a problem.

Why is the use of no EAL extremely bad? According to the mutual recognition agreement, each certifying agency is allowed to decide what they will certify and what levels they will certify up to. However the organization (that brought us policies of EAL4 minimum, followed by EAL2 maximum, followed by no EAL) has decided to try and persuade other organizations of the importance and value in not having an EAL. So while we in the security arena may be laughing at this proposal, it is making headway and, much to our chagrin, other certifying agencies are in fact adopting it.

This proposal may have numerous issues. It may break a number of valuable features such as Assurance Continuity and Mutual Recognition, but we don’t know all of the details yet. We can only assume (hope or pray) that this new certification approach has no real standing, or definition, at this time in the CC (by virtue of the time it takes for a change to the ISO process). We are also not sure of which agencies will be following the lead of this organization (four of fifteen certifying agencies have "drunk the cool-aid" at this point), nor what the result will be. If past experience is an indicator, most companies will go to agencies that will continue to certify under the current CC mechanism because their customers are asking for the higher EALs.

Our greatest concern is that the new schema will introduce changes that resulting in a less useful CC; all because of an assumption is that consumers are ignorant or incapable of reading the ST’s or PPs and determining if their needs are being met.

As a final thought, it is interesting that the certifying agency, pushing this change indicated in the past that they were finding inconsistencies in the results (and methods) across the test labs for which they were responsible.  It even went as far as to indicate that, in some cases, the results of testing were not repeatable.  If so, the CCRA is clear that it is the responsibility of the certifying agency to monitor the performance of the labs it oversees and to ensure the consistency of evaluations in their scheme. 

They now state on the certification scheme's home page that the overall goal of their change is "Achievable, Repeatable, and Testable evaluation results".  Maybe this is their way of dealing with their issues and the real reason they want to reduce assurance levels to those for which specific and objective evaluation and testing methodology can be defined for a product type identified in the PP.

Perhaps rather than dumbing down the entire program (including the other schemes and labs) they should address their inconsistency issues.

| Follow us on Twitter | Follow us on Facebook