Verification management
Design Verification is estimated to consume about 70% of resources for chip development. A clear understanding of how so much resource is being spent often eludes Verification management and engineering teams alike. With rising complexity of designs, verification managers are on the lookout for effective methodologies that can reduce the risk of failure, streamline verification activity and provide management the required systematic information about progress and quality.
Verification plan development is often a laborious ritual held at the start of a large project only to be cast aside as the project unfolds. Developing a verification plan is one thing and keeping it in sync with the requirements as they evolve is quite another. Attempts to manually keep the requirements, the plan, and the execution environment in-sync is time consuming and unreliable to say the least. Analysis of execution results is a daunting task due to the sheer volume to the data generated in the process. Globally distributed teams executing the plan add another dimension of the complexity.
Agile methods for software development have proven themselves over the years since their first introduction in 2001. Such projects are characterized by higher success rates as evidenced by higher productivity, better quality, lower costs, and overall higher satisfaction.
In order to achieve such success rates in design verification projects, what is needed is a tool and methodology that uses agile principles to keep the verification plan, requirements, and all pertinent verification data in a central, dynamic, accesses controlled system from which a project completion strategy can be derived.
The Verification Management Problem
Let’s look at the problems that we are trying to address in some detail. These problems plague many a modern verification teams.
Verification planning
Detailed planning for Verification is often not attempted. Often it boils down to an activity that the team engages in at the start of the project.
There are very good reasons for not attempting to develop detailed verification plans. Why would verification teams spend time on something that is a resource sink rather than working on something that reduces their work load? As it is, verification teams are typically hard pressed for resources. Who has the time to write a document or a spreadsheet which will never be looked at once the project is underway?
Using a document editor or spreadsheet to create a static plan is a dud as the document becomes obsolete soon after it is created. During the lifetime of the project change comes from a variety to sources, such as:
· Requirement changes
· Design changes
· Changes to the verification strategy
· Adding more details in the verification plan as the project matures
· Resource changes which has an impact on verification priority and what gets verified.
· Schedule and Milestone changes
These changes affect the verification plan one way or the other. A static document is unable to capture these dependencies and it is impractical to change it every time some change happens. The result is an unchanging document that does not reflect the reality and is of little or no value to the engineering team and management.
Constantly evolving and highly dynamic verification
The exercise of Verification is, by its very nature highly dependent on the design changes. For example, as features are introduced appropriate testbenches and tests need to be setup to ensure that it is working as expected. As issues are discovered, new strategies are formed to address them.
Too much data, everywhere
Verification generates an excessive amount of data. The various versions of design, the testbenches, the regression runs, the random tests and the plethora of control variables, all contribute to the increase of data. Even with detailed coverage reports, it’s easy to lose track and be caught up in a numbers game.
Versions of design files, software revisions, verification goals, bugs …
It is highly typical of design teams to use some form of version control system to keep track of changes and manage design configurations. Verification teams often use a similar system to make sure they are aware of what it is they are verifying.
Dispersed data sources
Industry leading companies have begun to go beyond functional simulation and embrace the use of formal verification, emulation and acceleration to drive greater verification success in the drive for greater quality and schedule predictability. In addition, multi-vendor and multiple methodologies are now a norm rather than an exception. As a result, important information pertaining to the status and quality of the project is spread out.
Verification using various methodologies
Functional, Formal and Emulation based methodologies have different formats and schemes to report information about project progress.
Verification methodologies constantly evolve to keep pace with design complexity. New techniques and languages are constantly being evolved. Leading companies often have various teams using different methodologies with different degree of maturity.
Variety of Vendors with their specific formats
The trend to use multi vendor solutions is growing in the industry due to two reasons: a. customers want to maintain leverage with their EDA tool supplier and b. As the technology is maturing the tool performance and functionality are comparable from various vendors. Although UCIS(Universal Coverage Interoperability Standard) is working to ameliorate the problem of different vendors using their own database and reporting formats, however, as of this writing it’s still not a released work.
Other sources of information
The other sources of information include such things as:
· Bug tracking systems
· Project management system
· Requirement capture system
Distributed design and verification teams
With the growth of outsourcing and cross cultural design and verification teams a common platform for sharing data is absolutely essential for communication.
Possible solutions
A Verification management team can deploy a whole bunch of solutions to manage verification planning, execution and monitoring.
Static Documents and Spreadsheets
Historically, teams have used standard Office tools for managing verification projects. However, this is useful for small projects with a single, centralized team with less than 5 members.
It’s laborious and time consuming to keep such a spreadsheet up-to-date with changing requirements, plan and execution. Without any relational integrity it becomes difficult to maintain and share.
Using Wiki
Using a web based wiki is a good option for small teams. A wiki can be highly effective for collaboration. However, the unstructured nature of the wiki and lack of real time data prevents it from being used as a comprehensive platform for managing verification.
Home grown custom application
Given enough time and resources, a customized application can be constructed. It’s a classic buy vs. build problem, and companies can decide based on how much free time they have to devote to non-core activities. One thing to keep in mind is that creating an application is one thing, but scaling and evolving it with changing requirements often requires a dedicated team.
Using IVerifySpec (IVS)
IVS solves the problems mentioned in this paper. IVS is a Web2.0 application with a relational database backend. It gives verification team a central portal that can be accessed in a distributed environment.
IVS is designed around the theme of “separation of concerns”. It enables everyone on the project team to work in their specific niche area without the “concern” for how work is carried out in the other area. This loose coupling is advantageous because it avoids needless cross connections between the teams. The verification team-lead or the verification manager can create a verification plan while the verification team executes the plan, as changes are made to the verification plan, execution team automatically sees the updates. When the latest execution results are available, they are automatically entered into the system for everyone to see. Analysis charts are also created so project management can automatically keep abreast of the status of the project.
Developing Agile Verification Plans
It is rare that a verification plan will be created by a single individual, and the plan will be executed without any modifications. It is more typical that the plan would need constant adjustment to accommodate the verification dynamics, be it changing requirements, changing resources or changing priorities. What IVS brings is an ability for a group of individuals to collaborate and jointly develop and modify the plan.
Capturing Requirements
Capturing design requirements is often neglected by teams busy to implementing design and verifying functionality. However, maintaining a requirement set has the benefit of keeping everyone on the same page, which is an effective safeguard against miss-features and oversight. IVS provides agile, collaborative techniques to capture design requirements and with close proximity to the plan for verifying them.
Closing the loop on a plan
IVS provides links to a variety of data sources that provide information back to the system about the status of the verification project. For example, test pass/fail status, code coverage and functional coverage from the simulation environment, status of bugs from a bug tracking system, and any user define data that gives useful information for the project is available for the user to track in real time.
Such feedback information seen alongside the initial plan provides speedy closure to the project. IVS provides a Dashboard and many analytic charts to see current and historic trends in the project.
Success indicators
IVS includes analysis tools that provide deep insight into the verification team status. Past runs, code coverage, functional coverage, rate of increase of code coverage, rate of decrease of bugs, rate of increase of lines of verification code, rate or increase of design code, all such indicators are captured and presented in a meaningful way so that conclusions about the status of the project, its rate of progress and its estimated completion time can be predicted.
Ultimately what methodology is chosen to make the team agile depends on the seriousness of the problem of managing change.
Processing your request, Please wait....
