|
Answers
to Your Questions About VoiceXML
In
this monthly column, an industry expert will answer
common questions about VoiceXML and related technologies.
Readers are encouraged to submit questions about VoiceXML,
including development, voice-user interface design,
and speech technology in general, or how VoiceXML is
being used commercially in the marketplace. If you have
a question about VoiceXML, e-mail it to speak.and.listen@voicexmlreview.org
and be sure to read future issues of VoiceXML Review
for the answer.
This
month, we're going to tackle the first few reader-submitted
questions. These are a great start, and we look forward
to hearing from the readership about what questions
most need answers in this forum..
Q: To evaluate the performance of an VoiceXML interpreter,
is there any benchmark specified by the W3C or the VoiceXML
Forum?
A: Currently, there are no officially sanctioned
metrics or benchmarks for evaluating VoiceXML interpreter
performance. However, it is definitely the case that
large companies who are actively building VoiceXML-based
solutions are considering performance when evaluating
potential vendors.
The distributed nature of VoiceXML, which allows application
logic to be cleanly separated from the underlying speech
and telephony infrastructure, is one of its greatest
strengths. It revolutionizes the IVR market by making
outsourced IVR practical and cost-effective. However,
this same characteristic presents significant technology
hurdles that VoiceXML platform vendors must solve in
order to guarantee reliable performance and scalability.
.
As food for thought, I will present a few key metrics
that I believe make sense to consider when evaluating
VoiceXML platforms--either for on-premises or outsourced
deployments. These are not in any way meant to be exhaustive;
rather they are indicative of the kinds of tests customers
should be thinking about.
It's important to note that many of these critical performance
metrics are not entirely specific to the VoiceXML interpreter,
but also involve the relevant subsystems required for
actually "rendering" a VoiceXML application,
such as speech recognition, audio processing, etc.
Performance
Metrics for all VoiceXML Platforms (a subset)
All metrics should be considered under normal (~50%)
and stressed (~80+%) capacity levels.
- The
number of simultaneous inbound/outbound ports per
unit of rack space (port density).
- The
time to begin playing a prompt of several standardized
sizes, both first time and subsequent times in a session
(ms).
- Recognition
time for standardized static grammars of various sizes/complexities
(ms).
- Recognition
time for standardized dynamic grammars of various
sizes/complexities (ms).
- Recognition
time for utterances matching application-scope grammars
of standardized size (ms).
- Inter-state
execution time for forms/event handlers within the
same document (ms).
- Inter-state
execution time for forms/event handlers in new documents,
discounting Web server response time (ms).
- Execution
time for standardized blocks of JavaScript (ms).
- Processing
time for standardized blocks of <record>ed audio
(ms).
- Performance
variations for calls in progress when a single given
call enters a JavaScript loop.
Additional
Performance Metrics for Outsourced VoiceXML Platforms
(subset)
- Percentage
packet loss metrics from customer Web servers to vendor
network.
- Latency
from customer Web servers to vendor network (ms).
- Consistently
available IP bandwidth from customer Web servers to
vendor network (Mbit/sec).
- Number
of simultaneous audio streams without audio breakup.
- Bandwidth
consumed for standardized use of standardized applications
of various complexities (measuring cache performance).
- Latency
in reflecting changes to application content (measuring
cache performance).
- Number
of simultaneous <record> posts without performance
degradation of other calls.
Continued...
back
to the top
Copyright
© 2001 VoiceXML Forum. All rights reserved.
The VoiceXML Forum is a program of the
IEEE
Industry Standards and Technology Organization (IEEE-ISTO).
|