|
Building
the VoiceXML Forum Certification Program
Introduction
The
VoiceXML Forum is developing a certification
program for VoiceXML products for the benefit of its
members, vendors, and the buying public. This
program would allow vendors to submit their products
(or services) for evaluation against a suite of tests
that would demonstrate conformance to recommended
specifications or standards needed
to produce a conforming VoiceXML product. Initially,
the Certification Program will concentrate on VoiceXML
platforms; testing programs for tools and applications
may follow at a future time. A product passing the certification
program would then be permitted to be labeled as conformant.
The test suite and certification process would be managed
and developed by the Forum members and an independent,
arm’s length certification authority.
The
certification program operates to give vendors independent
evaluation and protection of their intellectual property,
and when desired, the opportunity to publicly or privately
communicate their certification to customers and partners.
The VoiceXML Forum Certification Program
will be open to all vendors who wish to have their products
conformance tested. VoiceXML Forum Promoter
and Sponsor members can participate in
the test suite development and maintenance process,
and the management of the certification process, as
appropriate.
As
of late October 2001, a request for proposals (RFP)
has been provided to organizations that expressed interest
in providing the services of the certification authority.
The VoiceXML Forum expects to be able to complete the
RFP process and make a selection of acceptable candidates
before the end of the December 2001.
What’s
Included in the Certification Program?
The
Certification Program includes conformance testing of
all recommended specifications and standards that would
be necessary to build a conforming VoiceXML language
processor, under the testing architecture shown in Figure
1. This diagram shows the product to be tested, the
system under test or SUT,
and the testing system that is comprised
of the test driver (which may be just a telephone handset
or some other automated test driver) and the reference
web server that contains the suite of tests and records
the test outcomes. The test cases are built around
the features in a specification that must be built in
order to satisfy define a conforming implementation;
furthermore, the specifications include the definitions
of conformance. As such, conformance process begins
with the specifications themselves.
Figure1:
The Conformance Testing Architecture
A
recommended specification is a document produced by
an organization that, through some development process,
creates recommended practice or behavior for some technology.
The World Wide Web Consortium (W3C http://www.w3.org)
and the Internet Engineering Task Force (IETF http://www.ietf.org)
are two such widely recognized bodies; the W3C typically
produces recommended specifications in the areas of
Internet content (such as HTML) and the IETF typically
produces recommended specifications for Internet protocols
(such as HTTP).
A
standard is a specification produced through the internationally
recognized standards bodies that typically organize
around country representation rather than company representation;
examples of these include the International Organization
for Standardization (ISO http://www.iso.org) and in the United States, the
American National Standards Institute (ANSI http://www.ansi.org).
For
VoiceXML the primary body producing specifications is
now the W3C. On October 23, 2001 the VoiceXML Forum
and the W3C announced a Memorandum of Understanding
concerning VoiceXML activities, in which the VoiceXML
Forum recognized the W3C as the sole body responsible
for the development of the VoiceXML Language specification,
and a coordination of the conformance efforts between
the VoiceXML Forum and the W3C. We’ll return to how
this coordination will work a little later.
Building
a VoiceXML Processor requires the implementation of
features from a variety of specifications; however,
the conformance testing of these specifications would
be confined to their execution as part of a VoiceXML
processor as there may be features that are outside
of the VoiceXML context. In other words, some specifications
will be subject to full coverage testing,
that is, conformance testing of all mandatory features;
those where a VoiceXML processor needs only a few features
will be subject to sufficient coverage
testing. Sufficient coverage testing does not do conformance
testing of mandatory features, but tests to show a VoiceXML
processor's adequate use of these specifications, not
its complete use of them.
For
full conformance coverage testing the following specifications
are expected to be included:
- W3C
VoiceXML 2.0: Currently this is in the Working Draft
(WD) step in the W3C recommendation process; see http://www.w3.org/TR/2001/WD-voicexml20-20011023/
- W3C
Speech Recognition Grammar Format (SRGF) 1.0: Currently
this is in the Working Draft (WD) step; see http://www.w3.org/TR/speech-grammar/
- W3C
Speech Synthesis Markup Language (SSML) 1.0: Currently
this is in the Working Draft (WD) step; see http://www.w3.org/TR/speech-synthesis
- W3C
Semantic Interpretation for Speech Recognition 1.0:
Currently this is still under development for its
first Working Draft (WD) thus it is not publicly disclosed
as yet. It will be used to provide semantic interpretation
features to support the speech recognition grammar.
Other
specifications such as the W3C Natural Language Semantics
Markup Language (NLSML) (http://www.w3.org/TR/nl-spec/)
may be included in the requirements and testing in the
future.
For
sufficient coverage testing the following specifications
are expected to be included:
For
the full coverage tests, the test requirements fall
into three categories: required, optional,
and platform-optional. Required features
are denoted by "must" or “shall” in the specification,
or otherwise are indicated as a mandatory feature of
a conforming implementation. For a required feature,
an implementation is required to support the syntax
and semantics as described in the specification. Optional
features are denoted by "should" in the specification.
If an implementation wishes to be certified that it
supports the optional feature, the conformance test
would show that the feature is implemented according
to the syntax and semantics as described in the
specification. If an implementation does not support
the optional feature, it must throw the appropriate
"error.unsupported"
code as described in the specification. Platform-Optional
features are denoted by "may" in the specification.
An implementation may or may not support a platform-optional
feature. Examples include various optional or suggested
audio formats, ASR formats, etc. These are not tested
under conformance testing since their behavior may be
platform specific.
Who’s
who in the Certification Program.
The
Certification Program involves several groups: the
VoiceXML Forum itself, an Independent Certification
Authority (CA), Vendors, and possibly,
Independent Testing Labs (TL). The Independent
Certification Authority is an organization that, under
contract to the VoiceXML Forum, will operate the certification
program at arms-length, to ensure confidentiality of
certification tests. Vendors will have products that
are candidates for certification. Vendors may include
but are not limited to VoiceXML Forum members. Independent
Testing Labs are companies that, perhaps recommended
by the VoiceXML Forum, or perhaps on their own, would
offer to provide independent testing of VoiceXML products
against the conformance test suite. The key activities
of these parties in the certification program are summarized
below.
VoiceXML
Forum:
- Define
the policies and procedures of the certification program;
the key task will be to create a process that is open,
fair and gives confidence to both vendors and the
buying public.
- Manages
the contract with the CA; this will require ongoing
work both in monitoring the technical work of the
CA, as well as the contractual obligations and any
financial maters.
- Co-develops
and co-maintains the test suite, with the CA; this
will be a task largely borne by the VoiceXML members
who participate in the conformance activities.
- Controls
the problem reporting process, and provides resolutions
of test suite deficiencies (with CA); this will be
in part a function of administration, but again will
involve VoiceXML members who participate in the conformance
activities.
- Work
with the W3C to resolve ambiguities in applicable
W3C specifications; this will be done through our
W3C Liaison channel, as well as VoiceXML members who
participate on the W3C and additional relations that
may be established between the VoiceXML Forum and
the W3C for this purpose.
- If
desired, recommends TL; the VoiceXML Forum may in
the future choose to qualify, and then recommend companies
that wish to provide VoiceXML testing services. However,
at this time, we are focused only on selecting the
certification authority.
- Controls
and manages the certification brand. The VoiceXML
Forum and its administrative and legal staff will
control the use of the VoiceXML Forum certification
marks, to ensure that only those companies who have
successfully met the obligations of the certification
program can advertise their product using our designated
marks.
CA:
- Co-develops
and co-maintains the test suite, with the VoiceXML
Forum; the CA would bring its technical expertise
to the table to create the main elements of the test
suite, and all the application programs needed for
the reference web server.
- Distributes
the Test Suite to Vendors and other authorized entities;
the CA would act as the channel and collect all appropriate
revenues.
- Manages
the certification application, testing, and notification
process; as described in the next section, the Vendor’s
initial relationship during certification is with
the CA, not with the VoiceXML Forum, so that the Vendor
can maintain its confidentiality. Consequently, the
CA manages much of the execution of the certification
process on a day-to-day basis without the involvement
of the VoiceXML Forum.
- Operates
the reference web server; this web server would be
used for all conformance testing when a product is
going through formal certification; thus all vendors
would be subject to the same reference test suite.
Since the CA is the only body allowed to collect the
conformance test results, it must have responsibility
for this critical component of the testing activity.
- Operates
the problem reporting process, and provides resolutions
of test suite deficiencies (with VoiceXML Forum);
again as the main interface for vendors, problem reporting
must come through the CA, and the identities of Vendors
protected. However, part of this process also belongs
to the VoiceXML Forum as the technical authority for
the test suite. Part of the problem reporting process
will also be public so that vendors can understand
the status of unresolved issues.
Vendors:
- Test
their platforms against the test suite managed by
the CA, or have their platforms tested by the TL against
the CA test suite.
TL:
- Provide
independent testing services for Vendors; a vendor
may wish to have the added marketing value of having
their formal conformance testing done by an independent
third party, and therefore could make a marketing
claim about how their testing was done, as separate
from the certification activity itself.
- Provide
other services outside of the certification program,
such as performance testing.
Next
we’ll see how this all fits together in the daily operation
of the certification program.
Continued...
back
to the top
Copyright
© 2001 VoiceXML Forum. All rights reserved.
The VoiceXML Forum is a program of the
IEEE
Industry Standards and Technology Organization
(IEEE-ISTO).
|