Spokestack's Blog RSS

Learn to Use Custom Wake Word and Text-to-Speech on a Raspberry Pi

Wed, 23 Jun 2021 00:00:00 GMT

One of the primary motivations for working on spokestack-python was to allow our models to run on embedded devices like Raspberry Pi. We are excited to show you how easy it is to use Wake Word and TTS models on these devices.

Spokestack Account

You will want to login and get your API keys for this tutorial. If you do not already have a Spokestack account, please create one.

Hardware

This tutorial is geared toward the Raspberry Pi 4B and Zero W. Technically, the minimum hardware requirements for this tutorial are a device that runs Python, a microphone, and at least one speaker. The recommended hardware is listed below. In addition, we have created a wishlist on Adafruit that is exactly what we used for this tutorial. If you have issues using other hardware or want to show us what you made, feel free to contact us. We are working on more hardware guides, so stay tuned!

Parts List

Raspberry Pi 4 Model B or Zero W
Adafruit Voice Bonnet
Mono Enclosed Speaker
Official Raspberry Pi Power Supply
SD/MicroSD Memory Card
Note: The Zero W runs a little slower than we would like

Raspberry Pi Setup

For the initial setup of the Raspberry Pi we recommend following the Adafruit Voice Bonnet tutorial. This guide walks you through everything from OS installation to sound configuration. In addition to the Adafruit instructions, there are a few Spokestack-specific instructions/tips in the following. These instructions should be followed while connected to your Raspberry Pi via SSH.

Audio

PulseAudio and the Adafruit Voice Bonnet do not interact well so you will want to disable PulseAudio with the following:

systemctl --user stop pulseaudio.socket pulseaudio.service

If you would like to enable PulseAudio afterward you can restart the service with:

systemctl --user start pulseaudio.socket pulseaudio.service

System Dependencies

The following are some system dependencies that need to be installed before installing spokestack on the Raspberry Pi.

sudo apt-get -y install portaudio19-dev libblas-dev libmp3lame-dev

Install Rust for Tokenizers

The command to install Rust is taken directly from the instructions. On a Raspberry Pi 4, I didn’t have any issues compiling Rust, but if you are using the Zero, you may need to cross-compile. We are currently working on an easy solution for the smaller embedded devices.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

TFlite Interpreter

For this, we can go with TensorFlow’s recommended apt package. We’ve used the pip versions in the past, but this one is easier to install. These commands are directly from the original instructions.

echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install python3-tflite-runtime

Installing Spokestack

Spokestack should be installed through pip. We are currently using v0.0.20 for this tutorial.

pip install spokestack==0.0.20

Testing with a Project

We will test with our “Hello, World!” project. Keep in mind we installed the dependencies in the previous sections, so you will not need to follow that project’s README.

git clone https://github.com/spokestack/python-hello-world.git
cd python-hello-world

You will want to add your API keys to the const.py in KEY_ID and KEY_SECRET. Now we should be able to run the app. The project will automatically download the default wake word models. Once running, the default text-to-speech voice will respond when you say “Hey, Spokestack!”

python app.py

Wrapping Up

In this tutorial, we covered how to set up Spokestack on an embedded device. This should get you started with using Spokestack with your projects. If you run into any trouble be sure to reach out through our support channels.

What's a Keyword Model, and Why Would I Use One?

Mon, 14 Jun 2021 00:00:00 GMT

So you’ve read the description of a keyword recognition model, but you still have questions. Not a problem! Keyword recognition (sometimes called “keyword spotting”) isn’t quite the household term that “ASR”, “NLU”, and “TTS” are. In this post, we’ll lay out what this type of model does, where it fits into the Spokestack speech processing model, and why you might (or might not) want to use one.

How Keyword Recognition Works

The Spokestack libraries support keyword recognition via three separate TensorFlow Lite models, much like our wake word models. They’re very lightweight and run entirely on-device. The first two models preprocess and encode audio for the third, which is trained to detect a number of specific words or phrases chosen ahead of time. In the research literature, each word or phrase would be called a “class”, and the task of choosing one of these from a collection of many is known as multiclass classification. In our web interface, we call the classes “keywords” to avoid overusing technical jargon. The notion of a “class” is important, though, as we’ll see later on.

All this processing happens in near-real time, with keyword recognition running every time Spokestack’s speech pipeline deactivates, and we’ll talk about what that means in the next section.

So Is It ASR, NLU, … A Unicorn?

First, the short and sweet answer: in Spokestack’s libraries, keyword recognition is a type of automatic speech recognition (ASR). It can be used with or without a wake word, but using it as your app’s ASR means that your app shouldn’t need NLU at all. Let’s unpack that a bit.

Spokestack transforms speech audio into text via a series of processing steps collectively called the speech pipeline. The speech pipeline includes voice activity detection (VAD), wake word detection, and ASR. These stages are mix-and-match: you can use any or all of them (though it only makes sense to run them in the order I’ve listed them in here).

When ASR is actively processing audio to transcribe it into text, we say the pipeline is “active”. You get to choose whether that activation happens as a result of:

the VAD detecting speech (all speech audio will be run through ASR)
a wake word being detected (in which case you should be running VAD to send only speech audio through the wake word detector)
a button press in the app’s UI

All the most common configurations are made available via pipeline profiles.

Using a keyword recognition model as ASR means that it will process audio received while the speech pipeline is active, making its decision when the pipeline deactivates. Deactivation happens when speech stops for a preset amount of time (usually a few hundred milliseconds)or when the pipeline has been active for a particularly long time. Both of these durations are configurable, but the profiles include sensible defaults that should work in most situations.

The speech pipeline informs your app about activation, deactivation, and speech recognition events asynchronously through an event listener, or observer. See the documentation for your chosen platform on how to establish a listener.

Did You Say I Don’t Need NLU?

OK, so it’s ASR, but how does a keyword model make NLU unnecessary? Recall our discussion about keyword classes above. A single class — or keyword, in the web interface — can have multiple members, just like a class in a schoolroom has multiple students. We call these class members “utterances”, and combining utterances with keywords gives you super powers. OK, not quite, but it’s close.

When you included multiple utterances in a keyword, the model will detect any utterance in the user’s speech, but it will transcribe that utterance using the name of the class. This means that, for example, you can name a keyword “volume_up” and have utterances that detect “volume up”, “turn it up”, “louder”, and so on. If a user says any of those utterances, your app will receive a transcript of volume_up.

This collapsing of a collection of utterances into a single command is a stripped-down version of what NLU does. You’ve configured the keyword model by hand, so you have a complete list of all the keywords you could possibly see in transcripts. If the model isn’t confident enough that it heard one of the utterances it was trained with, it will fire a timeout event instead of a recognition event, which is the equivalent of a fallback intent in NLU.

Great! I’m Using This for All the Things!

Not so fast. Keyword models are a powerful tool, but other forms of ASR and NLU do exist for a reason. Let’s talk about some of the reasons for and against using keyword recognition ASR.

Pro ✅: Network Usage

Keyword recognition runs entirely on the device running your app. No network connection is necessary, saving both time and data, and making the primary functionality of your app’s voice interface available anywhere your user happens to be. This is great for, say, a music app used by trail runners out in the woods.

Pro ✅: Power Consumption

No network requests means less battery drain, and keyword models are more lightweight than on-device ASR models, requiring fewer cycles to do their work.

Pro ✅: Privacy

On-device ASR means your users’ speech never travels to the cloud.

Pro ✅: Simplicity

If you have an app that is naturally controllable via a limited set of commands, keyword models make your life easier. You don’t need to use Spokestack’s NLU module at all, which saves you writing training data for said model, downloading its files, and accounting for it in your app model.

Con ❌: Limited Vocabulary

The last point has a flipside. The fact that a keyword model’s vocabulary is limited means that your app won’t be good at handling unexpected requests — in fact, it’s likely to give you a timeout event if the user says something you didn’t anticipate, leaving you with no idea what the user said. This makes it more difficult to log user requests and use that information to improve your model.

In other words, if you want to support requests made in full sentences or long phrases and allow your users to come up with novel ways to word their requests, a keyword model isn’t for you.

Con ❌: No NLU

Again with the flipsides. NLU isn’t just for collapsing similarly worded requests into the same command to make them easy to process. The command part is the intent in an NLU result, but there are also slots. You need slots if your users’ request can be parameterized — if you think about a spoken command like it’s a method/function call in your app, the intent is the function’s name, and the slots are its arguments.

You can’t reasonably support a command like “Order me a large pizza with pepperoni and sausage” with a keyword model, because you’d need a different keyword for each size and topping combination possible. Talk about a combinatorial explosion. That sort of command is exactly where NLU models excel.

Con ❌: Data-Hungry

Keyword recognition models have to be trained. For a small project with just a few users, a personal keyword model should be fine; you can just ask your users to record data samples and use that as your training data.

For an app you want to distribute widely, you’ll want to collect data for each of your utterances from a variety of voices to ensure it works well for as many of your users as possible. We can help with data collection, but there is a cost involved, and it gets larger the more samples you need to collect.

Summary

Here’s what we’ve covered:

Keyword recognition is a type of ASR, taking the place of platform-provided or cloud ASR in the speech pipeline.
You decide the vocabulary for a keyword model ahead of time.
When you use keyword recognition-based ASR, you don’t need to train an NLU model.
They’re a great tool, but whether they’re the right tool is specific to your use case.

If you still have questions, let us know in the forum or via any other link in the Community section below!

A Swear Jar in 100 Lines of Python

Thu, 10 Jun 2021 00:00:00 GMT

In this tutorial, we’re going to use Spokestack’s AutoML training tool to create a keyword recognizer model and use it, along with the spokestack-python library, to make a digital swear jar. The full code is on GitHub if you’d like to run it for yourself.

The concept of a swear jar is simple: you have a list of words or short phrases that you’d like to stop saying, or stop someone else from saying. Every time one of those words does slip out, you drop a coin in a jar. When the jar’s full, you take it to the bank and open a savings account to spend when you’re better behaved.

We should note that if swearing’s not your thing, this concept also lets you reward yourself for saying nice things—call it a “compliment jar”. We wouldn’t want to limit this tutorial to us reprobates.

Anyway, the first thing that might come to mind for this use case is a wake word detector: you don’t want your app running ASR all day, continuously streaming a pipe of data to wherever; you just want it to notice when you’ve said certain things. But there’s a wrinkle: if movie ratings have taught us anything, it’s that certain words are just … ickier than others. We’d like to make it more expensive to say \<insert epithet here> than, say, \<insert milder epithet here>. (You didn’t think we were actually going to suggest profanity for your list, did you? This is a family-friendly tech site we’re running here.)

A keyword recognizer is perfect for this. Trained on a small set of utterances, it acts as a wake word recognizer and ASR in one, letting you know which utterance it heard without activating a cloud service in the process—your bad habits stay on your device forever.

Now that we know what we’re doing and why, let’s get to doing it.

Creating a Model

Note: You’ll need a Spokestack Maker or higher account to follow along with model creation. Running the sample app is totally free, so go ahead and take it for a spin first. Keep in mind that the models provided with the sample code are personal models, so performance with your voice may vary. They were trained with a relatively deep voice.

If this is your first time using Spokestack, you’ll need an account. If you have a free account, you’ll need to upgrade to the Maker tier. Once that’s taken care of, head over to the keyword tool. Click the “New Model” button and name the model whatever you like. We’re going with “swear jar” here, saving our creativity for the actual list of words.

Speaking of, now’s the moment you’ve been waiting for: channel your inner George Carlin and click that “add keyword” button until you’ve accumulated enough to break (or reinforce) your habit.

While adding keywords, you’ll notice that you also have the option to add multiple utterances to each keyword. Keywords and utterances interact like this: the keyword is the text that will be returned to your app when the model recognizes any utterance listed under it. So if one of your swears is, say, “beef”, but you also want to stop saying “beefsteak”, you might group those together by making them separate utterances under the keyword “beef”.

Once everything is typed in, find a nice secluded space where no one will overhear enough to be concerned about your mental state, and prepare to swear at your computer. Or, as we call it, “Tuesday”.

Click the “record” button next to an utterance to start recording samples for it. You’ll need at least three samples of each utterance to train a model, but there’s no upper limit. More samples only help performance, so really get in touch with your inner thespian. Don’t go too far, though; keep in mind that you want your training samples to still sound like you, so use whatever emotional affect you want your app to recognize as your voice.

When you’ve recorded all your samples, click “train”. Model training takes a few minutes, so take this opportunity to do some deep breathing to wrap up your day’s therapy session.

Don’t forget to download your model once it’s done training; we’re about to need it.

Writing the App

Our final product will be a simple Python command-line app that you can keep running in the background on any computer with a microphone attached. We’ll be using spokestack-python to do all the signal processing. You can make a full-featured voice app with the library, including NLU and TTS for responses, but we only need the speech pipeline to demonstrate the usefulness of a keyword model.

from spokestack.agc.webrtc import AutomaticGainControl
from spokestack.asr.keyword.tflite import KeywordRecognizer
from spokestack.io.pyaudio import PyAudioInput
from spokestack.nsx.webrtc import AutomaticNoiseSuppression
from spokestack.pipeline import SpeechPipeline
from spokestack.vad.webrtc import VoiceActivityDetector, VoiceActivityTrigger

FRAME_WIDTH = 10
SAMPLE_RATE = 16000
MODEL_DIR = "models"

pipeline = SpeechPipeline(
    input_source=PyAudioInput(
      frame_width=FRAME_WIDTH, sample_rate=SAMPLE_RATE
    ),
    stages=[
      AutomaticGainControl(sample_rate=SAMPLE_RATE,
                  frame_width=FRAME_WIDTH),
      AutomaticNoiseSuppression(sample_rate=SAMPLE_RATE),
      VoiceActivityDetector(
        frame_width=FRAME_WIDTH,
          sample_rate=SAMPLE_RATE,
          vad_fall_delay=500
      ),
      VoiceActivityTrigger(),
      KeywordRecognizer(list(WORDS.keys()), model_dir=MODEL_DIR)
    ]
  )

# we'll talk about this function in just a bit
pipeline.context.add_handler("recognize", swear_heard)
print("OK, I'll be listening. No funny business.")
print("Press Ctrl-c to exit.\n")
pipeline.run()

Believe it or not, this is the bulk of the code you’ll need for this app. We’re setting up the speech pipeline with the help of some hard-coded default values. WORDS isn’t shown here, but it maps the keyword classes we just created to their values in cents so we can report how much we owe later on. See the GitHub repository for more details.

In most cases, you’ll want to read keyword classes from the metdata.json file distributed alongside the model, but for sake of demonstration, it’s easier to put them in a static dictionary. The key gotcha is that if you hardcode, you still have to list the keywords in the same order as they appear in metadata.json because the model will reference them by index.

If you do want to load from the metadata file, this is how you’d do it in Python:

import os
import json

def load_keyword_classes(model_dir):
    metadata = os.path.join(model_dir, "metadata.json")
    config = json.load(metadata)
    return [clazz["name"] for clazz in config["classes"]]

The other piece of our puzzle is the app’s reaction when it hears a word on our list. For that we’ll use the pipeline event we registered above:

from spokestack.io.pyaudio import PyAudioOutput

AUDIO_DIR = "audio"
OUTPUT = PyAudioOutput(num_channels=1, sample_rate=44100)

def swear_heard(context):
  value = WORDS.get(context.transcript)
  if value:
    sound_path = os.path.join(AUDIO_DIR, f"{value}.wav")
    with wave.open(sound_path, 'rb') as coin:
      frame = coin.readframes(1024)
      while frame != '':
        OUTPUT.write(frame)
    print(f"Uh-oh! I heard {context.transcript}. That'll be {value} cents.")

Notice that this function was registered to listen to the recognize event, hinting at the fact that the keyword recognizer is acting as our ASR here. If we were using a wake word detector instead, it would fire the activate event, and we’d have to wait for recognize from a separate ASR to get a meaningful value in context.transcipt.

Thanks to some pre-recorded and cleverly named audio files, we’re able to not only print a scolding message to the terminal, but play the sound of a hard-earned coin dropping into the jar.

Go Deeper

Keep a running total of the amount deposited throughout the day
Grab a housemate and train using both your voices
???

Conclusion

In order to run our application from the command line, you’ll need to wrap the functions above in standard if __name__ == "main" Python boilerplate (again, see the sample code for a working setup).

Other than that, though, the keyword model and the code above are almost all you need to make a working swear jar using Spokestack.

Have fun, and happy (?) swearing! Get in touch (contact info of all types is listed below) to let us know if you enjoyed this tutorial or if there’s anything else you’d like to see us talk about.

What Are Personal AI Models?

Thu, 10 Jun 2021 00:00:00 GMT

The Spokestack Maker service is designed for two audiences: (1) hobbyists who want to personalize their projects, and (2) developers who want to prototype a project as realistically as possible before committing to training a universal wake word model or studio-quality text-to-speech (TTS) voice. This is a great way to test drive the technology without breaking the bank.

What are the Differences Between “Personal” and “Universal” Voice AI Models?

Like any neural models, the personal models created with these tools are only as good as the data they’re trained on. In other words, a wake word created in Maker will not respond to everyone else in the world as well as it does to you if it’s shipped in a mobile app — it will only work reliably for voices that resemble the one(s) it was trained with.

For a home automation project, though, this specificity might be a feature — have you ever wished that your smart speaker didn’t listen to the kids quite so well? (Of course, if you do want your DIY smart speaker to listen to everyone, you can call the kids over to the computer and have them help you train it!)

Similarly, with Maker personal TTS you can train a clone of your voice in just 75 sentences. However, its pronunciation and similarity to the human it’s mimicking isn’t going to be quite as good as a voice trained on dozens of hours of audio recorded in a studio. It’ll still be plenty recognizable — check out the voices on our showcase along with their training criteria — but high-quality samples, and lots of them, always produces better models than low-resource data. If you happen to have good recording equipment and a quiet environment, we do have thousands of scripts queued up in the training tool, so go ahead and record as much as you have patience for. Our automatic model trainer will adjust to make the best use of the data you provide it.

With Spokestack Maker, you’ll still be able to use the NLU model trainer and TTS showcase, but you’ll also get access to three new tools for training your own wake word, keyword, and TTS models using data you record yourself.

You’ll be able to download your custom models and make unlimited requests to your custom TTS voice as long as your Maker subscription is active. If you decide you’d like to take it to the next level and make a universal wake word/keyword model or a studio-quality TTS voice, let us know!

How to Change Alexa's Voice

Mon, 07 Jun 2021 00:00:00 GMT

At Spokestack, one of our core goals is to make it possible for every mobile app or voice skill to have its own unique audible voice. Amazon and Google make it easy to use their text-to-speech services, but then your skill (/action) ends up sounding like all the rest. If you’ve ever wondered how to change Alexa’s voice, we have an article on Voice Tech Podcast just for you.

It’ll give you some background on what’s required to actually replace the system voice and walk you step by step through the use of a low-code example in Python to set up a skill of your own. You can use a free Spokestack account to try everything out, and if you agree that your skill should stand out from the crowd, you can upgrade to the Maker tier and create a completely new voice that only you can use.

The whole process is a very low-effort way to see the benefits of a Spokestack account in action, so head over to Voice Tech Podcast for the whole article!

Introducing Spokestack Maker - AutoML Personal Voice for Creators

Wed, 26 May 2021 00:00:00 GMT

Spokestack exists to help put voice into software because voice is a humane way to interface with software and because software needs better, more accessible tools to utilize voice.

Spokestack Maker introduces sophisticated machine learning tools for voice to a new audience of prototypers, enthusiasts, and makers. Create, train, distribute, and use state-of-the-art personal wake word, keyword, and text-to-speech (TTS) models — with your own voice.

Now all developers, not just voice assistant experts or machine learning specialists, can create sofware featuring voice. Spokestack Maker turns voice technology into just another interface developers can utilize—like a mouse & keyboard or a touchscreen—to interact with users.

Lowering the barriers to voice technology means reducing technical cost in addition to the sticker shock that custom machine learning models often carry. Voice AI is a difficult field, full of papers with irreproducible results, hidden pitfalls, and undocumented code. You can end up spending all day in Jupyter notebooks babysitting training jobs instead of building your killer app. We know, we’ve done it before! Spokestack Maker turns voice into just another interface developers utilize—like a mouse & keyboard or a touchscreen—to interact with users.

Spokestack is open-source

Spokestack already offers powerful, industry-leading libraries for utilizing voice in Python, React Native, Android, Node, and iOS. With Spokestack Maker, that voice becomes personal and customizable, utilizing state of the art self-service AutoML technology.

With Spokestack Maker, you have the power of three new tools for training your own wake word, keyword, and TTS models using data you record yourself. You’ll still have access to the same NLU model trainer and TTS showcase available to all existing free accounts.

Live Maker Q & A with the Spokestack team

Have more questions, or want a live demo of Maker? We’re handing out swag and answering questions Wednesday June 2 at 1pm EDT. The whole Spokestack team will be on to talk about why we created Maker, some ideas of how you can use the tools in your software, and answer any questions from the chat! Can’t make it then? The live event will be archived on our Youtube for on-demand viewing.

Features and Pricing

Want to test drive cutting-edge machine learning features like keyword recognition, wake word activation, and custom AI voices without breaking the bank? Spokestack Maker brings this enterprise-level technology to the creator market for the first time.

Spokestack’s Maker subscription pricing is tailored for two audiences: hobbyists who want to personalize their projects and developers who want to prototype a project as realistically as possible before committing to training a universal wake word model or studio-quality TTS voice.

You’ll be able to download your personal models, train and retrain your personal models and enjoy API access your personal TTS voice as long as your Maker subscription is active. If you decide you’d like to take it to the next level and make a universal wake word/keyword model or a studio-quality TTS voice, join the Spokestack Pro waitlist!

Why Haven't We Seen a Killer Voice App?

Mon, 10 May 2021 00:00:00 GMT

In the interest of fostering discussion about the state of voice tech, we’re thinking of starting a new regular feature. It’s tentatively titled “Ask the Cranky Computational Linguist” — which, of course, would be me. The title’s not necessarily official, but I do get cranky from time to time, so it fits.

Our first topic is the question you clicked on to get here, but the answer is part of our Substack publication. Posting there lets us send new issues directly to your inbox and interact in the comments section, but we don’t want to leave our web site readers behind, so we’ll add new issues as summary blog posts just like this one.

If you’re interested in discussions like this, let us know by liking/subscribing on Substack, and feel free to submit new topics there or via email. Thanks for reading!

Converting a TensorFlow Model to TensorFlow.js in Python

Mon, 12 Apr 2021 00:00:00 GMT

At Spokestack, we use several different types of TensorFlow model, and we have various deployment targets for them: big models for the cloud, small TensorFlow Lite models for mobile devices, and even Tensorflow.js models for browsers.

In cases where a single model architecture is trained and deployed to more than one of these targets, it’s nice to be able to do the training, any necessary conversion, and deployment all from one place. It’s even better if this can be managed from a single Python module.

This is easy enough for formats like SavedModel and TensorFlow Lite, but the JavaScript target is a little trickier. Google provides instructions for converting a SavedModel, but the only documented path for doing such a conversion is by using a command line utility. That’s a bit of a bummer for our Python module, since we’d really rather not shell out to a library that’s written in Python itself. Surely there’s a better way.

To GitHub!

Thankfully, the code for tfjs-converter is on GitHub for us all to see, so let’s dive in and see what we can find.

Note: Since we’re talking about an undocumented workflow, and part of it uses protected functions, it almost goes without saying that this is all subject to change. At the time of writing, it worked using version 2.4.1 of tensorflow and 3.3.0 of tensorflowjs. Both are available via pip install, though installing tensorflowjs will get you a compatible version of tensorflow for free.

First things first: if you have a SavedModel on disk somewhere that you’re looking to convert, you can skip the slight messiness of the rest of this post and just do the following (source code):

from tensorflowjs.converters.converter import tf_saved_model_conversion_v2 as convert

# ...

# see the source code for other valid kwargs
convert.convert_tf_saved_model(saved_model_dir, output_dir)

See? Easy. But it’s not quite what we wanted, as it requires us to export our model to the SavedModel format first, and of course that exported version has to be re-loaded to do the conversion. So let’s break it down a bit.

The body of convert_tf_saved_model loads the model, freezes the weights, builds a protocol buffer version of the signature function we want to convert, and exports the frozen graph to JavaScript format. All those subtasks exist as protected functions, but since this is Python, no one’s going to ask any questions about us just calling those — no one except Pylint, that is, and it can be bribed with a comment line if necessary.

Here’s the replacement I came up with for the public function above. It’s essentially a reproduction that doesn’t require a directory as input. You of course don’t need the Pylint comments if you don’t use Pylint.

from tensorflowjs.converters.converter import tf_saved_model_conversion_v2 as convert

# ...

def convert_func(concrete_func: tf.Graph, output_dir: str) -> None:
  if not os.path.exists(output_dir):
    os.makedirs(output_dir, exist_ok=True)

  # pylint: disable=protected-access
  frozen_graph = convert._freeze_saved_model_v2(
    concrete_func, control_flow_v2=True
  )

  inputs = [x for x in concrete_func.inputs if not x.dtype == "resource"]

  # pylint: disable=protected-access
  signature = convert._build_signature_def(
    frozen_graph, inputs, concrete_func.outputs
  )

  output_graph = os.path.join(output_dir, "model.json")
  convert.optimize_graph(frozen_graph, signature, output_graph, TF_VERSION)

I’ve retained the mypy type hints because I think they help the explanation. The concrete_func argument here is a full TensorFlow graph, but it has that name because the model in question uses tf.function to make retrieving its signatures easier. If you have access to these functions, you can call get_concrete_func to get an input for our convert_func function.

Another way to get a valid concrete_func is to use a SignatureDef. Let’s say you have a SavedModel from which you want to export several (but not all) signatures:

import tensorflow as tf

model = tf.saved_model.load("path/to/saved_model")
for signature in ["sig1", "sig2"]:
  output_dir = f"path/to/output/{signature}"

  # conversion will fail if the parent directory doesn't exist
  if not os.path.exists(output_dir):
    os.makedirs(output_dir, exist_ok=True)

  concrete_func = model.signatures[signature]
  convert_func(concrete_func, output_dir)

Notice the creation of a new output directory for every signature. This is important, as internally TensorFlow.js will create a model.json and binary weights file(s) for each signature, and if you have a single output directory for all the signatures, it will overwrite each one in turn, leaving you with a single set of converted files at the end.

Conclusion

And just like that, we’ve come out the other side of our spelunking trip into the tensorflowjs source code, only a little worse for wear. Models converted using the above method can be loaded in TensorFlow.js via tf.loadGraphModel.

A well-designed repository can now host code for both model training and TensorFlow.js deployment, all in the same language, without maintaining separate shell scripts for model conversion, and all it took was reading a little source code. Who knows, maybe Google will see the value in this approach and choose to officially support this workflow with more public functions and some user-facing documentation in the future.

I hope this speeds our collective journey to infer all the things in the browser, or at least makes it a bit more comfortable.

Question Answering With Spokestack and Transformers

Wed, 13 Jan 2021 00:00:00 GMT

The ability to find information is a fundamental feature of the internet. Often, the information sought is the answer to a question. When it comes to answering a question about a specific entity, Wikipedia is a useful, accessible, resource. This tutorial will teach you how to use Spokestack and Huggingface’s Transformers library to build a voice interface for a question answering service using data from Wikipedia.

Learning Objectives

By the end of this tutorial, you receive the following:

The ability to give your app a voice with Spokestack.
A basic understanding of how to incorporate Huggingface’s Transformers with Spokestack.
An interactive voice interface that allows you to answer questions from Wikipedia.

Setting up the Project

First, let’s make a directory to hold the project.

git clone https://github.com/spokestack/wikiqa-python
cd wikiqa-python

Now let’s set up the python virtual environment. We use pyenv to manage virtual environments, but any virtual environment will work.

pyenv install 3.8.6
pyenv virtualenv 3.8.6 wikiqa
pyenv local wikiqa

Then the dependencies.

pip install -r requirements.txt

TensorFlow

For this tutorial, we are using the full tensorflow package for a little more functionality than is included with the TFLite Interpreter. If this is the first time you are installing tensorflow you should follow the system-specific installation. For those who already have the ability to install via pypi the following will install the library to your environment.

pip install tensorflow

Transformers

In addition to TensorFlow, you will also need to install HuggingFace’s transformers library for the question answering model.

pip install transformers

Speech Pipeline with Profiles

Profiles are preset configurations for our Speech Pipeline. For this tutorial, we will use the Spokestack wake word and ASR profile. The wake word model runs on device, and can be activated by saying “Hey, Spokestack”. ASR is in the cloud though, so you will need to get your API credentials to use it. If you already have a free account, log in. If you do not, you will need to create one. The credentials can be found in your account settings. This is everything you need to speak to your app.

from spokestack.profile.wakeword_asr import WakewordSpokestackASR

pipeline = WakewordSpokestackASR.create(
    KEY_ID, KEY_SECRET, model_dir="tflite"
)
pipeline.start()

Natural Language Understanding (NLU)

Natural Language Understanding (NLU) is how we transform what the user says into action. For more explanation on the NLU see our docs. The NLU model for this project is already included in the GitHub repository. However, we will briefly discuss the model configuration in the next section.

from spokestack.nlu.tflite import TFLiteNLU

nlu = TFLiteNLU("tflite")

Model Configuration

We’ve included a pre-trained model, so you can follow along with this tutorial, but if you want to create your own, here’s a quick introduction to writing NLU training data. See our documentation for more information on our data format and how to train your own model. We are using a basic NLU template which includes intents like greet, accept, and help. In addition, we need a way to identify an entity in the user utterance to perform a Wikipedia search (more on this later). The name of this intent is ask.question, and the utterance templates are simple ways to ask a question. Naturally, these templates could be more complex and cover a wider variety of utterances, but for the purpose of this tutorial these will be enough.

[slots.subject]
type = "entity"

[generators.adjective]
type = "list"
values = ["long", "tall", "wide", "far"]

[utterances]
values = [
    "who is {subject}?",
    "what is {subject}?",
    "what is a {subject}?",
    "what is an {subject}?",
    "{subject}",
    "how {adjective} is the {subject}?,
    "how {adjective} is {subject}?
]

Question Answering (QA)

For the QA model, we are using a SQUAD-style span detection method. The problem is framed as, “given this question and this context that contains the possible answer, identify the text span that contains the answer”. At this point, you most likely are thinking, “where do we get the context?“. We will answer that in the next section.

Context Retrieval

Earlier, we mentioned that we wanted to be able to identify entities in the user’s utterance for a Wikipedia search. Context retrieval is the reason for this. Our QA model needs a context from which to draw the answer. Therefore, we will follow the assumption that the question is about a specific entity. For example, “What year was Ada Lovelace born?“. From this utterance, we pull out the entity “Ada Lovelace” and do a search to retrieve the Wikipedia page. Then, we feed the entire Wikipedia page into the model as context for our question.

Setting Up The Model

We are using a pre-trained DistilBERT model, which is fine-tuned on the SQUAD dataset. The choice of model is based on the size, similarity of the fine-tuning task to our objective, and availability at the time of writing. The collection of pre-trained models is always growing, so feel free to try any of the other models that were fine-tuned on a similar task. You will most likely find a model that works better than this basic one. We would love to hear about the one you discovered, or even better, trained!

from transformers import AutoTokenizer, TFAutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")
model = TFAutoModelForQuestionAnswering.from_pretrained(
    "distilbert-base-cased-distilled-squad"
)

Dialogue Manager

The dialogue manager for this project is a smart speaker-style interaction. A user asks a question, and the bot speaks the answer. The full example can be seen below. The process starts with retrieving the entity (person/place/thing) found in what the user said. Then, we take that entity and look up its page on Wikipedia. Next, we take the full text of the Wikipedia page and feed it, along with the question, to the QA model. The model gives us the location of the answer, which we use to grab the answer from the full text. Then, we return the answer in the form of a response. This setup is pretty simple and just meant to get you started. Definitely expand on this dialogue manager and conform it to your needs.

"""
Simple QA dialogue manager
"""

import tensorflow as tf
from mediawiki import MediaWiki
from spokestack.nlu.result import Result
from spokestack.nlu.tflite import TFLiteNLU
from transformers import AutoTokenizer, TFAutoModelForQuestionAnswering


class DialogueManager:
    """ Simple Question Answering Dialogue Manager """

    def __init__(self, log_path: str, base_model: str) -> None:
        self._wiki = MediaWiki()
        self._nlu = TFLiteNLU(log_path)
        self._tokenizer = AutoTokenizer.from_pretrained(base_model)
        self._answerer = TFAutoModelForQuestionAnswering.from_pretrained(base_model)

    def __call__(self, utterance: str) -> str:
        result = self._nlu(utterance)
        if result.intent == "ask.question":
            return self._answer(result)
        elif result.intent == "greet":
            return self.greet()
        elif result.intent == "command.exit":
            return self.exit()
        elif result.intent == "request.help":
            return self.help()
        else:
            return self.fallback()

    def _answer(self, result: Result) -> str:
        if result.slots:
            # get the tagged entity for page search
            entity = result.slots.get("entity").get("raw_value")
            # perform the search to find the wikipedia page
            entity = self._wiki.search(entity)[0]
            # get the page content to feed as context to the qa model
            passage = self._wiki.page(entity, auto_suggest=False).content
            # prepare qa model inputs
            inputs = self._tokenizer(
                result.utterance,
                passage,
                return_tensors="tf",
                padding=True,
                truncation=True,
            )
            # compute answer span
            start_scores, end_scores = self._answerer(inputs)
            start, end = tf.argmax(start_scores, -1)[0], tf.argmax(end_scores, -1)[0]
            # prepare the passage ids for slicing
            tokens = self._tokenizer.convert_ids_to_tokens(
                (inputs["input_ids"].numpy()[0])
            )
            # retrieve only the answer from the passage
            answer = self._tokenizer.convert_tokens_to_string(tokens[start : end + 1])
            return answer
        return "I don't have an answer for that"

    @staticmethod
    def greet() -> str:
        return "Hello, Ask me anything"

    @staticmethod
    def exit() -> str:
        return "Goodbye"

    @staticmethod
    def fallback() -> str:
        return (
            "I'm having trouble understanding your request, could you please "
            "repeat it"
        )

    @staticmethod
    def help() -> str:
        return "Ask a question like, how long is the amazon river?"

OK, now we have our response from the dialogue manager. The next question on your mind is probably “how do we give it a voice?“. Check out the following section for how to set up Spokestack’s text to speech service.

Text to Speech (TTS)

Now, let’s give the app a voice. Similar to the profile section, you will need your Spokestack API keys. We offer a TextToSpeechManager class which requires a TTS client and an output source. In most cases, the PyAudioOuput class should work. It uses the default system speaker.

manager = TextToSpeechManager(TextToSpeechClient(KEY_ID, KEY_SECRET), PyAudioOutput())
manager.synthesize("hello, world", "text", "demo-male")

Now that we have given our bot a voice, we can put everything together. At this point, you are just one section away from a voice interface that allows you to get answers to simple questions. Let’s move on to the complete working example.

Putting it All Together

We have all the modules set up. Now we need to add the logic that will respond to events in the conversation. For this, we use our Pipeline Events. Pipeline Events are simply events that occur while the pipeline is running. To use them, you decorate functions with an event decorator. Most applications will want an event handler that does something when speech is recognized. For ours, we want to process the question and play the response. This is defined in the on_recognize handler. The function we are using for this example can be seen below. For more information on the included events take a look here.

@pipeline.event
def on_recognize(context):
    pipeline.pause()
    answer = dialogue_manager(context.transcript)
    manager.synthesize(answer, "text", "demo-male")
    pipeline.resume()


manager.synthesize(dialogue_manager.greet(), "text", "demo-male")
pipeline.start()
pipeline.run()

That pretty much wraps it up! You should now be able to ask some questions, and it will (hopefully) be able to find the right answer. I hope you found this tutorial useful, and thanks for taking the time to read it!

Contact Us

If you have any questions while getting this set up we have a forum, or you can open an issue on GitHub. In addition, I am more than happy to help if you want to reach out to me personally via email or Twitter.

Integrating Spokestack in Android

Thu, 17 Dec 2020 00:00:00 GMT

Using voice as an interface can be a daunting proposition. First, you have to be able to actually process voice as input. Then, you have to decide what to do with that input. You probably want to deliver responses via voice too. After all that, you still have to decide how to integrate the voice interface with your touch interface.

If you’re here, you probably know that the Spokestack library helps you with the first three of those challenges. With Spokestack Tray, you’ll also have an answer for the last one.

What is it?

We created Spokestack Tray to give developers a full-featured visual UI to reflect an app’s voice interactions with the user. At first glance, it’s just a microphone button that sits on the side of the screen. When the user taps the button (or says the wake word), it opens up into a full tray that displays the user’s speech as well as system responses, reading those responses aloud as it displays them.

How do I use it?

First, add the dependency to your app’s build.gradle:

dependencies {
  // ...
  implementation 'io.spokestack:tray:0.4.4'
}

The tray is implemented as a Fragment, so to include it, add this to your activity’s layout.xml:

  <!-- nested in the main layout, after other views/sublayouts -->

  <include
    android:id="@+id/tray_fragment"
    layout="@layout/spokestack_tray_fragment"
  />

Note: Depending on your app layout, you may also have to add android:clipChildren="false" to the fragment’s parent layout(s) to avoid the microphone tab disappearing as the tray opens.

Then make your activity itself extend TrayActivity (a subclass of AppCompatActivity), implement the methods it requires, and the library will take care of the rest. Voice interaction will be handled by the Tray, which you’ll have access to via an instance variable named tray that’s initialized during onStart.

You’ll need an ID and secret key from your Spokestack account in order to set up the TrayConfig that TrayActivity requires.

If you’d prefer to do the setup yourself, here’s a sample that doesn’t use TrayActivity, and demonstrates a minimal complete TrayConfig as well:

import io.spokestack.tray.*

class MyActivity : AppCompatActivity(), SpokestackTrayListener {

  lateinit var tray: SpokestackTray

  // ...

  override fun onCreate(savedInstanceState: Bundle?) {
    val config = getConfig()
    supportFragmentManager.fragmentFactory = SpokestackTrayFactory(config)

    // note that the factory is instantiated and set on the manager BEFORE calling
    // `super.onCreate()`
    super.onCreate(savedInstanceState)
  }

  private fun getConfig() {
    return TrayConfig.Builder()
      // credentials from your Spokestack account
      .credentials("spokestack-client-id", "spokestack-secret-key")
      .wakewordModelURL("https://path-to-wakeword-models")
      .nluURL("https://path-to-nlu-files")
      // note the implementation of `SpokestackTrayListener` in the class declaration
      .withListener(this)
      // optional builder customization; see the documentation for more details...
      .build()
  }

  override fun onStart() {
    // set the value of the lateinit `tray` var
    tray = SpokestackTray.getInstance(config)
    super.onStart()
  }

When you download Spokestack wake word or NLU models, you’ll have several URLs to different files. wakewordModelURL and nluURL above only require the path to the relevant directory, not full file URLs. So for the demo “Spokestack” wake word, set wakewordModelURL to ”https://d3dmqd7cy685il.cloudfront.net/model/wake/spokestack/”.

The Tray is designed for seamless use across activities — for example, to allow a user to continue giving a voice command while the app switches activities — so its state is stored outside the fragment itself and survives fragment destruction. If your app needs to release resources held by the Tray and its underlying Spokestack instance, call the tray’s stop() method. If you then need to re-enable voice control before the current Tray fragment instance is destroyed, you must call start().

If you want to keep tray state intact after process death, you can store it in its parent activity’s onSaveInstanceState and onRestoreInstanceState methods using the Tray’s getState() and loadState() methods; see their documentation for more details.

Responses

Chances are that if you’re allowing the user to talk to your app, you want the app to talk back. Tray is integrated with Spokestack’s TTS service, so synthesizing audio is just as easy as transcribing it.

When you extend TrayActivity, one of the methods you’ll have to implement is getTrayListener(), which creates and returns a SpokestackTrayListener. This interface assists your app in reacting to events received and produced by the Tray. Because each use case is unique, all its methods are optional; the one we’re interested in here is onClassification. This method is called after a user’s speech has been transcribed by ASR and classified by NLU. It supplies your app with the NLU result and asks you to return a response:

override fun onClassification(result: NLUResult): VoicePrompt {
  return if (result.intent == "your-special-intent") {
    VoicePrompt("I hear you loud and clear")
  } else {
  VoicePrompt(
    "Sorry; I didn't catch that",
    expectFollowup = true)
  }
}

The optional second parameter in the VoicePrompt constructor lets the Tray know if you’re expecting a response — if you are, it will resume active listening after your prompt is played so the user doesn’t have to use the wake word or a button for each interaction.

Configuration

The above sample will get you up and running with minimal fuss, but it’s far from all that Spokestack Tray offers. When you’re building a TrayConfig instance, you can choose to configure and provide the underlying Spokestack builder itself. This will let you do things like change ASR providers, set up custom listeners for events from individual systems, and add custom speech processing components if you need to. You can read about the Spokestack builder here.

There are also a range of options that are applicable to the Tray itself, accessible via helper methods on the TrayConfig.Builder instance. See the documentation for more details, specifically the documentation on TrayConfig.Builder.

Conclusion

We hope this brief introduction to Spokestack Tray has inspired thought about what your app could do with voice control. With Tray, experimenting with this exciting interface takes just a few lines of code. See the README for more details about the Tray and the customization options available. If you need help, please open a GitHub issue or check out one of our multiple support channels.

Integrating Spokestack with Google App Actions, Part 3

Tue, 15 Dec 2020 00:00:00 GMT

This tutorial is part of a series:

Part 1: Working with Google App Actions
Part 2: Adding your own voice experience with Spokestack
Part 3 (You are here!): Using Spokestack Tray to add a voice UI

Now that we know what we need to get started with voice control, both on Google’s end and in our app, we’ll polish our user experience a bit by adding visual elements to give the user more feedback about what the app is hearing and saying. With the changes we’ll make in this tutorial, an interaction with Google Assistant will look like this:

To follow along, run git checkout spokestack-tray in your copy of the sample repository. Since this tutorial involves changes to the activities we’ve already introduced, we’ve made them on a separate branch for easy comparison to the “headless” version of Spokestack.

The why and how of Tray

As we’ve seen in other tutorials, you can add voice control to your app using just the Spokestack library. Spokestack Tray exists to help the user visualize their voice interactions without you having to build an entire UI for them from scratch. It includes a microphone button used to activate ASR without the use of a wake word, visual feedback while ASR is active, and a chat-like interface that displays the conversation between user and app.

Let’s walk through the process of dropping it into our project, starting with the changes to build.gradle. Since the Tray exists to mediate your users’ interaction with Spokestack, it bundles the dependencies necessary for that. That means we can replace all our Spokestack-related dependencies with the following:

    implementation 'io.spokestack:tray:0.4.4'

Continuing with the declarative parts of the project, Tray is implemented as a Fragemnt, so each Activity that wants to include it will want to add a few lines to its layout XML:

    <!-- nested in the main layout, after other views/sublayouts -->

    <include
        android:id="@+id/tray_fragment"
        layout="@layout/spokestack_tray_fragment"
        />

This will add a nested ConstraintLayout containing the SpokestackTray fragment into your layout. If you’d rather manage the layout XML yourself, take a look at the layout file we’re including. You’ll want to use the same tag for the fragment as it does, as it’s important elsewhere.

Note also the clipChildren attribute on the parent layout. Depending on how your app is set up, you might also need to add this to any layouts serving as parents to the tray fragment. Our sample app does this in all its activities.

On to the fun stuff!

That takes care of the setup, so let’s move on to the code from part 2 that we’ll need to change. Spoiler alert: Thanks to delegating management duties to Spokestack Tray, we’ll be deleting more than we add, resulting in a much simpler developer experience!

First, the Voice class goes away entirely — all Spokestack interaction goes through Spokestack Tray.

VoiceActivity gets a new parent class, the TrayActivity convenience class, which inherits from AppCompatActivity and deals with some Android platform unpleasantness related to Fragment construction. We’ve moved our main Spokestack configuration (now configuration for the Tray, which is similar) from Voice to VoiceActivity so we can provide it to TrayActivity. This configuration includes URLs instead of paths for wake word and NLU files since the Tray downloads them automatically; this means that our app bundle can be smaller, and we don’t have to write the download code.

We’ve also taken the opportunity to provide a fallback voice response that’ll be available to all subclasses:

fun fallbackPrompt(): VoicePrompt {
  return VoicePrompt(
    "Sorry, I didn't understand. Please try again.",
    expectFollowup = true
  )
}

We’ll be using this fallback a lot, since we haven’t built out a full voice experience for this sample. Everything else that was in that class dealt with managing Spokestack, so it just goes away. Feels good, doesn’t it?

Next, let’s look at SearchActivity to see how we deal with a voice command that can come from either Google or Spokestack:

private fun setUiFromIntent(data: Uri?, fromTray: Boolean = false) {
  data?.getQueryParameter("item")?.let {
    runOnUiThread {
      binding.searchContent.text = it
    }

    // `fromTray` is our cue for whether the command originally came from
    // Spokestack Tray or Google Assistant. If it's the former, the Tray
    // will automatically display and play the response; if the latter,
    // we'll need to explicitly respond.
    if (!fromTray) {
      tray.say(response(it))
    }
  }
}

The comment there reveals our secret. If the user opens this activity with a voice command from somewhere else in the app or from Google Assistant, we generate a response and pass it through Tray using the say method. If SearchActivity is active when the voice command is received, it can set fromTray to true and generate the response elsewhere.

That’s as good a segue as any to talk about the other notable change in our activities — the listener. We’re converting all our nested SpokestackAdapters to SpokestackTrayListeners, which doesn’t change much in our case because our sample app was only interested in NLU events to begin with. Our nluResult handlers become onClassification handlers, and now they’re expected to return a response for the Tray to display and read.

We won’t add any new logic to smooth the voice experience here other than supplying an error message. The message itself comes from VoiceActivity and plays any time we can’t recognize a search request for a specific item. The only real change to the listener method, then, is the return:

return if (item != null) {
  response(item)
} else {
  fallbackPrompt()
}

Conclusion

Just like that, we’ve gone from a cool-but-confusing “headless” voice experience to a UI-supported voice interface that puts the user in control and removes code from our codebase. It’s a win-win. A quick checklist for integrating Spokestack Tray in your app:

✅ add/update dependency
✅ include fragment layout in activity XML
✅ subclass TrayActivity in activities that include the fragment
✅ supply a TrayConfig suited to your app
✅ implement SpokestackTrayListener and supply appropriate responses in onClassification

Integrating Spokestack with Google App Actions, Part 2

Mon, 30 Nov 2020 00:00:00 GMT

This tutorial is part of a series:

Part 1: Working with Google App Actions
Part 2 (You are here!): Adding your own voice experience with Spokestack
Part 3: Using Spokestack Tray to add a voice UI

In the first part of our tutorial, we talked about how to make an Android app’s features available via Google App Actions. In this part, we’ll take it to the next level and show how to continue the user interaction via voice once Google Assistant has dropped the user off inside the app.

From actions to intents

First, we’ll need to recreate the ability to act on natural-language user requests, which is a job for an NLU. Luckily, we’re already halfway to configuring one, thanks to the actions.xml file required by Google. As a quick reminder, it’s in the res/values/xml folder in the sample app.

If you’ve worked with voice platforms before (or read the page linked in the last paragraph), you might notice some familiar concepts in that file, especially if you’ve used any custom intents in your app. intentName is … well, the name of the intent, parameters are slots, and queryPatterns are utterances. If you’re using a built-in intent, like GET_THING in the sample app, queryPatterns is hidden from you, handled entirely by Google, but the other things are still there.

We’re going to exploit that similarity and convert our XML directly into Spokestack’s NLU format, using it to create a custom NLU model that will replicate the features we’ve just defined for Google Assistant in our app itself. This is a great opportunity to add new intents to your in-app NLU that would be too tricky or infeasible to expose via Google Assistant.

We won’t go over the converted versions of all the intents here, but here’s the Spokestack version of the navigate.settings intent from actions.xml:

description = "the user wishes to view the settings screen"

[generators.verb]
type = "list"
values = [
  "see",
  "show",
  "give me",
  "open",
  "change",
  "update",
  "go to",
  "take me to"
]

[utterances]
values = [
  "settings",
  "settings screen",
  "settings screen please",
  "{verb} settings",
  "{verb} my settings",
  "{verb} the settings",
  "{verb} the settings screen",
  "i want to see my settings",
  "i want to change my settings"
]

In Spokestack’s format, you can achieve the same effect as Google’s conditionals (the parenthetical words and question marks in the XML) using generators. We’ve added a few more utterances to the Spokestack config than were present in actions.xml because they’re a little easier to express here.

The description field is optional but can help you think about the interplay among your various intents.

For sake of demonstration, we’ve included all the Spokestack NLU files in src/main/assets/spokestack-nlu. We’ve renamed the GET_THING buit-in intent to command.search for in-app usage and supplied our own utterances for it. You won’t need to actually create a model with these to run the tutorial code, though, because the trained model is in the assets folder as well. We’ll talk about how to use it … right now.

App logic

Integrating Spokestack

Now that we have both App Action and Spokestack NLU configuration in place, let’s look at how to handle voice input with Spokestack once the user’s already in your app.

You probably noticed in the first part of the tutorial that our sample app is a bit of a hodgepodge with no particular purpose other than to cover a few common use cases for voice control. Humor us here. The app has 4 total layouts, one that serves as a main menu and three others that demonstrate different potential features. Each scene has its own activity.

To process voice input, all you need is an instance of Spokestack and a subclass of SpokestackAdapter to receive events. Since we have multiple activities and want them all to be voice-enabled, we’ve made a Spokestack instance available via a singleton (see the Voice object) and created an abstract VoiceActivity to be extended by all activities that need access to it. Each VoiceActivity is responsible for creating its own SpokestackAdapter because each has its own UI that needs to be updated by voice commands. This could likely be DRYed up too, but this is a demo, after all, not production code.

Notice that this design means there’s very little business logic to add to any given activity to add voice interactions. You don’t have to fiddle with the microphone, explicitly start speech recognition, etc. — that’s all handled by Spokestack, which is managed by the parent VoiceActivity.

There are, however, two details that are particularly important:

The Spokestack setup in Voice, specifically how it deals with wake word and NLU data files

For this tutorial, we’re taking the simple approach of including all our data files in the assets folder (thus distributing them with the app itself) and simply decompressing them to the cache folder on startup. To decrease app size, you can choose to omit these files from the distribution and download them if absent, possibly forcing the app to redownload them based on version changes. It’s important to think up front about how you’ll distribute updates to your NLU model, since it essentially determines which features are available via voice.

The permission request in VoiceActivity

Spokestack needs the RECORD_AUDIO permission to use the device’s microphone. It’s automatically included in your manifest when you declare the Spokestack dependency, but starting with Android API level 23, you’ll also need to request it at runtime. Since the user can revoke the permission at any time in their settings, we check for it and re-request every time a VoiceActivity is created.

The way things are set up here, the microphone permission will also be requested on app startup. In a real app, you’ll want to reorganize this to explain the permission before requesting it in order to provide a better user experience.

Recreating the Google Assistant

With our voice integration set up, let’s take a look at how it’s used. Open up DeviceControlActivity and scroll to the bottom.

inner class Listener : SpokestackAdapter() {
  override fun nluResult(result: NLUResult) {
    if (result.intent == "command.control_device") {
      val dataUri = Uri.Builder()
        .appendQueryParameter("device", result.slots["device"]?.rawValue)
        .appendQueryParameter("command", result.slots["command"]?.rawValue)
        .build()
      setUiFromIntent(dataUri)
    }
}

There are several other methods in SpokestackAdapter that we could take advantage of to make our UI more responsive and log errors, but all we’re interested in for the purpose of this demonstration is receiving results from the app’s NLU. What we’re doing here is reusing our deep link processing that’s in place for Google Assistant: when Spokestack’s NLU gives the app an intent that matches the current scene, we use that intent’s slots to construct a URI containing the query parameters we’ve set up for our App Action. The schema and host don’t matter because we’re already in the activity we want to be in. If we wanted to transition somewhere else, we’d have to make a full URI and use the startActivity method; this is what MainActivity does to route button presses to different activities.

At the risk of sounding like a broken record, a proper voice experience does require a bit more code than this. You’ll want to handle intents that aren’t meant for the current activity, respond intelligently, and so on. To do all this in a maintainable, understandable way, you want what’s called a dialogue manager component. Watch this blog for more on that in the future!

A couple more notes about DeviceControlActivity:

Take a look at onResume. It routes the URI in intent?.data to setUiFromIntent just like our NLU event handler above. The presence of a URI in data is how you’ll know if you were reached via voice command, unless you’ve explicitly deep-linked to this activity somewhere else. If that’s the case, you’ll want to include an extra query parameter somewhere to help the app tell the links apart.

We’ve overridden onResume from VoiceActivity here to avoid an awkward scenario where the TTS response starts playing before the system has completed the transition to the new activity, which causes playback to pause when the transition does finish.

populateVoiceMaps does a simple synonym mapping of potential slot values (what the user might actually say) to canonical device names — in this case, we’re actually mapping straight to the UI components that represent those devices. This is because Google only lets us specify parameters for custom intents as plain text, rather than allowing the full expressive power of entities that’s available to built-in intents. Hence, we can’t do that normalization in actions.xml. In Spokestack’s format, we can fix this using a selset slot, but since user queries could come from either Google or Spokestack, we’ve left the parsing logic in the app so both query types can be handled the same way.

Time to talk back

Once you’ve mastered the basics of voice navigation we’ve talked about here, you’ll naturally want to start thinking about how your app should respond to users. We’ve given quick examples of this in both DeviceControlActivity and SearchActivity, but let’s talk briefly about the latter.

In SearchActivity’s setUiFromIntent, we extract the “item” slot (the presumed search term) from the data URI and use it in a TTS response to the user. We end the response by asking the user if they want to search again. To make this a seamless experience for the user, we’ve added the following event handler to our Listener inner class:

override fun ttsEvent(event: TTSEvent) {
  when (event.type) {
    TTSEvent.Type.PLAYBACK_COMPLETE -> spokestack.activate()
    else -> {}
  }
}

This snippet automatically reactivates ASR when Spokestack finishes playing the audio for a TTS prompt, so the user can give another search if they want. If they say nothing, the ASR will deactivate after a timeout.

That brings us to one last point: in its current state, voice integration in the sample app can only be accessed via wake word (“Spokestack”, in this case). It’s easy enough to add a microphone button of your choosing, and it should call spokestack.activate() just as our TTS listener above. If you want to make your button work like a walkie-talkie, you can call spokestack.deactivate() when the user releases it; otherwise, calling deactivate is unnecessary.

Conclusion…or is it?

Congratulations; you have an app that not only makes its features accessible via Google Assistant, but continues that voice interaction via its very own voice layer! We’ve only scratched the surface of making a fully immersive voice experience here, but check out our other tutorials and documentation to learn more.

On that note, if you were frustrated by our final caveat—that we don’t have any UI feedback for our voice interactions—then part 3 of the series is for you. We’ll take what we’ve developed here and drop in Spokestack Tray so that our users can see what they’re saying and interact with Spokestack more naturally.

Integrating Spokestack with Google App Actions, Part 1

Mon, 23 Nov 2020 00:00:00 GMT

This tutorial is part of a series:

Part 1 (You are here!): Working with Google App Actions
Part 2: Adding your own voice experience with Spokestack
Part 3: Using Spokestack Tray to add a voice UI

The terms “smart speaker” and “voice assistant” are now full members of our cultural vocabulary. Talking to a voice assistant has become so common that voice interaction has grown from an exciting new feature into an expectation. Consumers are habituated to the tech and would like to use it in mobile apps — if only the apps supported it.

Making that last part a reality is what Spokestack’s all about. In this tutorial, we’re going to solve one of the problems mentioned in the article linked above:

Those approaches enable a user to employ Siri or Google Assistant to deep link into an app to a specific screen or to execute an action. However, once the user gets to that point, the Siri or Google Assistant session has ended.

The “approach” the author’s talking about, in Android’s case, is the use of an App Action to open an app or a specific screen/Activity via Google Assistant. This is a great feature that Google’s provided, and having the interaction between the user and Google Assistant end after your app has been opened is exactly what you want! That’s your user experience to manage from there on out, and any data provided by the user (your customer) should stay between the two of you.

It does, however, leave you with the burden of maintaining the UX expectations set up by that initial interaction, which in this case means continuing the conversation via voice if the user wants to. Let’s talk about how to do that.

Show me the code!

In this tutorial, we’ll be talking through a sample app we’ve set up to demonstrate an integration with App Actions. It doesn’t have any groundbreaking features or much of a UI to speak of, but it should give you a basic idea of how to handle multiple Intents/Activitys via voice, using Google Assistant for the initial interaction and following up with your own voice responses from then on.

Setup

Google has some prerequisites for App Actions development. You’ll need to meet those in order to try out the Google Assistant interactions. Spokestack’s features are available either way; the only prerequisite for them is a physical device (not the emulator) to test voice input.

Dependencies

OK, with that out of the way, let’s get our dependencies sorted. Check the app/build.gradle file in the sample project — these are the important bits:

implementation 'io.spokestack:spokestack-android:11.5.2'
implementation 'org.tensorflow:tensorflow-lite:2.6.0'
implementation 'androidx.media:media:1.3.1'
implementation 'com.google.android.exoplayer:exoplayer-core:2.14.0'

There are several non-Spokestack dependencies here because Spokestack takes a “use only what you need” approach and doesn’t include many transitive dependencies by default. The sample app will use everything, though, so we’ll need to pull in some libraries to support wake word, NLU, and TTS that we wouldn’t need if we didn’t want that functionality.

Declaring App Actions

The main addition to your app when integrating App Actions is adding a res/values/actions.xml file that describes the pieces of your app you want to make available to Google Assistant. Google’s documentation for the format is here, but in summary:

actions is the top-level element, and it contains one or more nested action elements.
Each action describes a single intent (or command) you want to expose to Google Assistant.
You choose between a built-in and custom intent via the intentName attribute:
- A name that matches on of Google’s built-ins will automatically inherit the parameters described by that built-in.
- A name that does not match a built-in means that you’ll need to specify both a queryPatterns attribute that points to natural-language examples of how a user would invoke this intent and nested parameter elements that describe any parameters that you want to capture in those utterances.
Each action must have at least one nested fulfillment element that describes what part of the app should handle the intent. There are several ways to handle this. Our sample app uses custom schemas (example://feature?{param1,param2}), but an app tied to an established web presence should probably use http/https deep links instead.

Here’s an example of one of our actions, this one using a custom intent:

<action
  intentName="navigate.settings"
  queryPatterns="@array/NavigateSettingsQueries">
  <fulfillment urlTemplate="example://settings" />
</action>

This is the simplest action we have, because it doesn’t include any extra parameters. We’re only using this one to take the user to a certain screen in the app. The queryPatterns attribute points to an item in res/values/arrays.xml:

<string-array name="NavigateSettingsQueries">
  <item>settings</item>
  <item>settings (screen)? (please)?</item>
  <item>go to settings</item>
  <item>go to my settings</item>
  <item>go to the settings</item>
  <!-- ... -->
</string-array>

Note the use of conditionals, a feature from another part of the Google Assistant ecosystem brought over to make these query configurations a little more concise.

Fulfilling the App Actions request

As mentioned above, the fulfillment element specifies what sort of URI a given intent should produce, but the routing for those URIs happens in your manifest (AndroidManifest.xml). An activity that should handle a deep link must declare an intent-filter element that matches said link. Here’s the intent filter for the URI above, which leads to the settings activity:

<intent-filter>
  <action android:name="android.intent.action.VIEW" />
  <category android:name="android.intent.category.DEFAULT" />
  <data
    android:host="settings"
    android:scheme="example" />
</intent-filter>

The key here is the data element, where we break down the custom URI we used in actions.xml. This data element matches a URI of “example://settings”.

Once properly declared in the manifest, links handled by an activity will show up as the data field of the activity’s Intent. See any of the sample app’s activities for an example.

This is, of course, only one way of setting up deep links. For a large app (or maybe just for sake of convenience), you might prefer an alternate solution like airbnb’s DeepLinkDispatch.

Try it out!

This is a good time to take a step back and run the demo to see what we’ve been talking about. You can run the app itself on either the simulator or a device, but you’ll need a physical device with Google Play services and a Google developer account in order to test the actual App Actions. If you have those, head over to this Codelab for instructions on how to modify the app’s ID, upload a test build to the Play Console, and get the App Actions Test Tool.

Once you’ve uploaded the build, make sure you’re signed into Android Studio with the same developer account you used for the upload (click the avatar icon at the top right of the IDE window to sign in). Then, follow the instructions in the “Test App Actions” section later in the Codelab to create a preview build.

You can send test actions either directly from the test tool or by invoking Google Assistant on the device like you normally would. Try saying things like “OK Google, turn the office light off” or “OK Google, find me some shoes”. We’ve worked out the kinks for this demo, so everything should go smoothly; if it doesn’t, please open an issue in the sample app repository! If you run into any errors as you start to make changes to the sample app, check Google’s troubleshooting tips; the error messages themselves aren’t always intuitive.

But wait…there’s more

That does it for the first part of the interaction — getting from Google Assistant into a specific part of your app. That’s a step toward a better user experience already, but check out the next part of our tutorial to learn how to continue the voice interface the user just tried to open. See you then!

Spokestack Python Library

Tue, 06 Oct 2020 00:00:00 GMT

Spokestack’s tagline, “Give your app a voice interface,” has mainly focused on mobile apps since we launched in January. Today we’re expanding the type of apps we can voice-enable to almost any project with the launch of Spokestack Python. Want to try it out now? Will Rice, who built the library, has written a tutorial for the “Minecraft Helper” example we developed for our mobile platforms.

Just as they do with our iOS, Android, and React Native voice libraries, developers can use Spokestack as a complete speech pipeline solution to talk to their customers. All you need to do to get started is create a Spokestack account and get your API credentials. The same API that powers our mobile libraries also powers Spokestack Python. We want developers to be able to write once, talk everywhere using Spokestack.

You have everything you need to add a voice interface to any project.

As with all of our libraries, developers can create a custom wake word and a text-to-speech voice that is unique to their brand and application. With the Spokestack Python library, the voice interface is less tied to a platform, opening up a number of application opportunities.

Got an Alexa or Google Assistant app you want to set up as a kiosk? Now you can build one on a low-energy, small-form device such as a Raspberry Pi or Tinker Board. Maybe you want to set up something on an old PC with a mic and a speaker. Simply export your NLU model, and you can do it!

Or maybe you want to build a voice bot companion for a game or social platform. Whatever your project, there are a ton of ways to add an independent voice interface, and you have complete creative freedom using Spokestack Python.

Over the next few weeks, Will is going to create video tutorials and demonstrations to show what you can do with Spokestack. We hope this will spark everyone’s imagination and stir your thoughts about what is possible with a flexible speech pipeline that’s customizable for any use case you can imagine. Be sure to follow us on Twitter or sign up for our newsletter for stream announcements.

If you have any questions or feedback, please send us an email. We look forward to seeing what you build!

Porting the Alexa Minecraft Skill to Python Using Spokestack

Mon, 05 Oct 2020 00:00:00 GMT

This is a tutorial on how to port a simple Minecraft recipe skill to Spokestack using the spokestack-python library. It is similar to our mobile tutorial series, but the Python version not have any GUI components. This makes the experience closer to that of a smart speaker. We will discuss the concepts for each part of the user interaction briefly, but for a full description check out our documentation. Before we get into the programming, we will need to get API keys from our Spokestack account.

Signing Up for a Spokestack Account

Create a Spokestack account.
Click “Add token” in the API Credentials dashboard.
Copy the secret key when it is displayed; you’ll need it later.

Setting up the Project

First let’s make a directory to hold the project.

git clone https://github.com/spokestack/minecraft-skill-python
cd minecraft-skill-python

Then there are some system dependencies.

macOS

brew install lame portaudio

Debian/Ubuntu

sudo apt-get install portaudio19-dev libmp3lame-dev

Now let’s set up the python virtual environment. We use pyenv and pyenv-virtualenv to manage virtual environments, but any virtual environment will work.

pyenv install 3.7.6
pyenv virtualenv 3.7.6 minecraft
pyenv local minecraft

Then the python dependencies.

pip install -r requirements.txt

TFLite Runtime

In addition to the Python dependencies, you will need to install the TFLite Interpreter. You can install it for your platform by following the instructions at TFLite Interpreter. Note: this is not the full Tensorflow package.

If you would like to try out the final version of the Minecraft app, you can now run it with:

python app.py

To follow along with the tutorial, we recommend making a new python file titled myapp.py or similar so you can compare it to the original app.py.

Using the Speech Pipeline

An essential piece to any voice interface is the ability to detect when the user is speaking, then convert the spoken phrase into a text transcript. Spokestack has an easy-to-use speech pipeline that will handle this for us. The speech pipeline consists of three major components: a voice detection module, a wake word trigger, and a speech recognizer.

Microphone Input

Accepting audio input is always the first step in the pipeline. For this demo, we will use the included input class that leverages PyAudio to stream microphone input to the pipeline. The class is initialized like this:

from spokestack.io.pyaudio import PyaudioMicrophoneInput

mic = PyaudioMicrophoneInput()

Voice Activity Detection

The second component we are adding to the pipeline is the VoiceActivityDetector. This module analyzes a single frame of audio to determine if speech is present. This will be the component that allows audio to flow through the rest of the pipeline. For simplicity, we will use the the default voice activity detection settings. The voice activity component can be initialized with the following:

from spokestack.vad.webrtc import VoiceActivityDetector

vad = VoiceActivityDetector()

Now that we have a way to determine if the audio contains speech, let’s move on to the component that activates the pipeline when it hears a specific phrase.

Wake Word Activation

The wake word component of the pipeline looks for a specific phrase in the audio input and signals the pipeline to activate ASR when it is recognized. For our purposes, we will be using “Spokestack” as the wake word. As with most voice assistants, “Hey Spokestack” will work as well. The process to initialize this component mirrors the way we set up voice activity detection. The directory passed to model_dir should contain three .tflite files: encode.tflite, detect.tflite, and filter.tflite. These can be found inside the tflite directory of the project GitHub repository.

from spokestack.wakeword.tflite import WakewordTrigger

wakeword = WakewordTrigger(model_dir="tflite")

Once the skill is actively listening for user speech, all we have to do is transcribe what the user says.

Automatic Speech Recognition (ASR)

ASR is the most critical piece of the speech pipeline, because it produces the transcript that is used to turn speech into actions. However, critical components do not have to be difficult to add. The following initializes the ASR component.

Note: This is where you will need your API keys from the account console.

from spokestack.asr.speech_recognizer import CloudSpeechRecognizer

recognizer = CloudSpeechRecognizer(
    spokestack_id="your_spokestack_key",
    spokestack_secret="your_secret_key",
)

Activation Timeout (Optional)

An issue you may run into is the ASR not being activated long enough or being active for too long. To configure this for your use case, you can add the ActivationTimeout component to the pipeline with a minimum and maximum value in milliseconds. This component can be initialized with the following:

from spokestack.activation_timeout import ActivationTimeout

timeout = ActivationTimeout(min_active=100, max_active=5000)

Speech Pipeline

Now, we can put it all together in the pipeline. After this step, you will be able to wake the assistant by saying “Spokestack” and produce a text transcript of what is said next. For the Minecraft skill you would say something like, “Spokestack, what is the recipe for a snow golem?“.

from spokestack.pipeline import SpeechPipeline

# without timeout
pipeline = SpeechPipeline(input_source=mic, stages=[vad, wakeword, recognizer])

# with timeout
pipeline = SpeechPipeline(
    input_source=mic, stages=[vad, wakeword, recognizer, timeout]
)

Events

We know that the goal of the pipeline is to produce a transcript of the user’s speech. However, we haven’t discussed how to access that transcript. The pipeline is designed to run continuously, but we can use event handlers to access the transcript without stopping the pipeline. For this tutorial, we want to pass the completed transcript to a module that helps us understand what the user has said. To accomplish this, we register an event handler with the pipeline:

@pipeline.event
def on_speech(context):
    transcript = context.transcript
    print(transcript)

In the application, we don’t want to print the transcript, but we’ve added that so you can see the results if you’ve been running the code as you follow along. In the subsequent sections, we will discuss a couple new components and also flesh out this event handler to allow the Minecraft skill to understand the user’s request and select an appropriate response.

Natural Language Understanding (NLU)

The Natural Language Understanding, or NLU, component takes a transcript of user speech and distills it into unambiguous instructions for an app. The paradigm used in most systems is the intent and slot model. Essentially, an intent is the function the user intends to invoke, and the slots are the arguments the intent needs to accomplish its action. For example, a user may say What is the recipe for a dark prismarine?. In this case, the intent is RecipeSearch, and the slot is dark prismarine . The initialization of the TFLiteNLU should look familiar at this point. The directory passed to model_dir contains three files: vocab.txt, metadata.json, nlu.tflite. These files are necessary to run our on-device NLU model and are in the tflite directory of the GitHub repository.

from spokestack.nlu.tflite import TFLiteNLU

nlu = TFLiteNLUModel(model_dir="tflite")

Now is a good time to add the NLU to our on_speech event handler:

@pipeline.event
def on_speech(context):
    transcript = context.transcript
    results = nlu(transcript)

Now that we know what recipe the user is looking for, you may be wondering how we turn this into a response. The following section will explain just that.

Dialogue Management

The Minecraft dialogue manager is fairly simple. The basic component necessary is a way to look up the recipes, and the rest is just string interpolation. Since we have a relatively limited number of recipes, we can implement the lookup as a simple dictionary in Python. Below is a snippet of the recipe “database”.

DB: Dict[str, str] = {
    "snow golem": "A snow golem can be created by placing a pumpkin on top of  two "
    "snow blocks on the ground.",
    "pillar quartz block": "A pillar of quartz can be obtained by placing a block of "
    "quartz on top of a block of quartz in mine craft.",
}

This makes looking up a recipe very concise: DB.get("snow golem"). There can be an issue with using the dictionary lookup alone though. Let’s say that due to an ASR error the parsed slot isn’t a full match for snow golem, but it is something like sow golem. A simple dictionary lookup will not be able to resolve those slots. However, there is a method that we can add to deal with small errors like that. This method is called fuzzy matching, and based on the similarity between snow golem and sow golem we can make sure that the latter resolves to the actual entity. In this tutorial, we will use the python library fuzzywuzzy to make these matches. Below is the way it is used in the tutorial repository.

from fuzzywuzzy import process

matched, score = process.extractOne(slot["raw_value"], self._names)

if score > self._threshold:
    recipe = self._recipes.get(matched)

We are simply overwriting the parsed entity with the one that is the closest match from the set of possible entities. The full dialogue manager can be seen below:

from fuzzywuzzy import process  # type: ignore

from minecraft import recipes
from minecraft.responses import Response

class DialogueManager:
    """Simple dialogue manager

    Args:
        threshold (float): fuzzy match threshold
    """

    def __init__(self, threshold=0.5):
        self._recipes = recipes.DB
        self._names = list(self._recipes.keys())
        self._threshold = threshold
        self._response = Response

    def __call__(self, results):
        """ Maps nlu result to a dialogue response.

        Args:
            results (Result): classification results from nlu

        Returns: a string response to be synthesized by tts

        """

        intent = results.intent
        if intent == "RecipeIntent":
            return self._recipe(results)
        elif intent == "AMAZON.HelpIntent":
            return self._help()
        elif intent == "AMAZON.StopIntent":
            return self._stop()
        else:
            return self._error()

    def _recipe(self, results):
        slots = results.slots
        if slots:
            for key in slots:
                slot = slots[key]
                if slot["name"] == "Item":
                    return self._fuzzy_lookup(slot["raw_value"])
                return self._not_found(slot["raw_value"])
        else:
            return self._response.RECIPE_NOT_FOUND_WITHOUT_ITEM_NAME.value

    def _help(self):
        return self._response.HELP_MESSAGE.value

    def _stop(self):
        return self._response.STOP.value

    def _error(self):
        return self._response.ERROR.value

    def _fuzzy_lookup(self, raw_value):
        matched, score = process.extractOne(raw_value, self._names)

        if score > self._threshold:
            recipe = self._recipes.get(matched)
            return recipe
        return raw_value

    def _not_found(self, raw_value):
        return self._response.RECIPE_NOT_FOUND_WITH_ITEM_NAME.format(raw_value)

Now we can add the dialogue manager to the event handler.

@pipeline.event
def on_speech(context):
    transcript = context.transcript
    results = nlu(transcript)
    response = dialogue_manager(results)

OK, that was a lot to cover, but we are almost to the finish line. In the next section, we will learn how to convert the app’s text responses into speech.

Text to Speech (TTS)

Much like the name suggests, TTS translates written text into its spoken form with a synthetic voice. This tutorial assumes you are using our default voice, but if you have a paid plan you can replace demo-male with the name of a custom voice. To initialize the TTSClient, you simply do the following:

Note: This is another part where you will need your Spokestack API keys. However, notice that the URL for TTS is slightly different than for ASR.

from spokestack.tts.clients.spokestack import TextToSpeechClient

client = TextToSpeechClient("your_key", "your_secret_key")

Another important aspect of this section is playback. We have a PyAudio-based output class that will play through your system’s default playback device. As a convenient way to manage speech synthesis and playback, we have the TTSManager. Look below to see how to initialize that with an output source.

from spokestack.io.pyaudio import PyAudioOutput
from spokestack.tts.manager import TextToSpeechManager

output = PyAudioOutput()

manager = TextToSpeechManager(client, output)

Now we can update our event handler to read the response.

@pipeline.event
def on_speech(context):
    transcript = context.transcript
    results = nlu(transcript)
    response = dialogue_manager(results)
    manager.synthesize(response, "text", "demo-male")

Let’s Run it!

We now have a fully working example except for two final commands. We have to start and run the pipeline!

pipeline.start()
pipeline.run()

Then you can start the skill by running this command in the terminal. The skill will remain running until you say “stop”.

python myapp.py

Now you should be able ask your skill about how to craft things in Minecraft. If you get a chance, I recommend trying it out in-game; despite its simplicity, it is actually very useful.

At this point you may have already cloned the repository, but if you have not, check out the full example on GitHub.

Contact Us

Why We Built Spokestack Tray

Thu, 24 Sep 2020 00:00:00 GMT

We started Spokestack to give developers the freedom to build the conversational experiences they wanted to build without limitations. So we created iOS, Android, and React Native libraries that give mobile developers complete flexibility for adding voice interfaces to their mobile apps. We made it possible for a mobile developer to design and deploy a voice interface any way they felt like it.

But sometimes a blank slate is daunting. Asking someone with no background in linguistics to build a conversational experience from scratch was like asking someone who hasn’t skied before to pull off a ski jump.

Not a real developer. Don't try this at home.

As it turns out, few mobile developers understand how to design conversational experiences. And few voice developers know how to build mobile apps. At Spokestack, we’re hoping to nudge both sides closer to each other.

With Spokestack Tray, we’ve created a ready-made “voice kit” that still allows for customization without having to design a voice interface from scratch. Will you still need to build a conversation? Yes, but you don’t have to think about how the user will interact with the conversation. Spokestack Tray makes it much easier to begin experimenting with adding a voice interface to your app and conversing with your customers.

Please check out Spokestack Tray and tell us what you think. If you have any questions or feedback, email us at hello@spokestack.io, and let’s talk conversational interfaces!

Introducing Spokestack Tray – a Turnkey Voice Interface for Mobile Apps

Mon, 21 Sep 2020 00:00:00 GMT

Today we are excited to announce our latest developer feature: Spokestack Tray, a mobile library that lets developers easily add a voice interface to their existing app through a UI component that opens inside the application. Spokestack Tray incorporates our voice services, such as a wake word, speech recognition, natural language processing, and custom text-to-speech voices, into one easy-to-add package along with any existing mobile application. We’ve built a demo, the Bartender mobile app, that uses Spokestack Tray.

Here’s a video of Spokestack Tray working in our demo Bartender app:

Download the Bartender app and try Spokestack Tray on your own phone:

iOS: https://apps.apple.com/us/app/get-the-bartender/id1530425843

Android: https://play.google.com/store/apps/details?id=com.spokestack.bartender

Why is this a big deal?

We’ve made adding a voice interface to a mobile app easier. We’ve simplified the process of building a voice interface for mobile apps and reduced it to just dropping a new UI element into your app! Even better, the UI element can be customized to fit into your app’s scenes and branding.
Using Spokestack Tray for search and navigation is fast. Tray manages the entire conversation on the user’s phone. We believe speed will drive adoption by both developers and users who want an easier way to find information and services when accessing mobile applications.
User conversations are private. As described above, our on-device voice interface libraries keep user conversations private. Because our natural language models operate on the device, developers and consumers do not need to worry about their conversations being listened to by third parties.
It works well with Apple Siri shortcuts and Google Assistant shortcuts. Speaking of third parties, protecting conversations between companies and their customers is a big selling point in our pitch. We provide hooks to integrate Siri shortcuts and Google Assistant shortcuts so your users can initiate a voice interaction from anywhere while keeping you in control of your and your users’ data.

Getting started with Spokestack Tray

We want to make it as easy as possible for you to get started with Spokestack Tray. Here are the steps to get a prototype up quick:

Sign up for a Spokestack account
Check out the documentation for React Native Tray or iOS Tray
Follow a tutorial: Spokestack Tray iOS Tutorial or Spokestack Tray React Native Tutorial
Get help in our Spokestack Tray Help forum

If you’re new to voice, we’ve put together an introduction to how Spokestack thinks about voice. For more on how to design the voice experience, walk through the Design section of our docs.

Customizing Spokestack Tray

At Spokestack, we believe flexibility and customization are keys to building successful conversational interfaces. We will be adding more design resources and tools over the next few weeks. Additionally, we help you further customize your app with custom wake words, TTS, and your own cloud or on-device NLU model.

Want a free wake word, NLU training, and custom TTS?

After you get a demo up and running, let us know if we can build a custom wake word or TTS for your mobile app. We’ll hook up the first three developers who show us a working version of Spokestack Tray in their mobile app with a free wake word, NLU training, and custom TTS. Email us your working demo and we’ll help you add these features to your app so you can publish it live.

What’s next?

Expect an update on design tools, the Android Tray library, and additional tutorials soon. Follow us on Twitter for all updates. Can’t wait to see how people use Tray with their mobile apps.

Have fun talking to your customers!

Integrating Spokestack in iOS

Fri, 18 Sep 2020 00:00:00 GMT

As much time, talent and treasure that Apple has put into Siri, ASR, and NLU, integrating a custom voice-enabled app experience is still challenging! Spokestack changes all of that. Our mission at Spokestack is to make it as easy as possible to make your apps fully voice-enabled.

After building the services needed to make voice interaction work, including Wake Word, Speech Recognition, Natural Language Understanding, and Text-to-speech, we started working on ways users could integrate these services without having to completely rewrite their applications.

Introducing spokestack-tray-ios

spokestack-tray-ios is an iOS framework that is designed to work in any application, regardless of its layout or navigation. It utilizes spokestack-ios, to add voice experiences. With on-device wake word, ASR, and NLU, the tray’s silent mode works completely offline–TTS is the only service that requires a network.

With a few required props (and lots of optional ones), you can start building a customizable voice experience without the hassle that usually comes with listening for a wake word, working with a microphone, or playing audio in iOS.

This tutorial will guide you through the process of installing spokestack-tray-ios as well as using the SpokestackTray framework to respond to your users.

Installation

CocoaPods is a dependency manager for Cocoa projects. For usage and installation instructions, visit their website. To integrate Spokestack into your Xcode project using CocoaPods, specify it in your Podfile:

pod 'SpokestackTray-iOS'

From your terminal in the project directory run

pod install

Either in your Podfile or your Project / Target settings set the minimum iOS target to iOS 13.

Edit Info.plist

Spokestack requires access to your device’s microphone and Apple’s speech recognition service.

without these your app will crash

<key>NSMicrophoneUsageDescription</key>
<string>For making voice requests</string>
<key>NSSpeechRecognitionUsageDescription</key>
<string>For understanding your voice requests</string>

Edit the AppDelegate.(swift|m)

Spokestack is dependent on the appropriate AVAudioSession to be configured. For most apps this can be done in the AppDelegate.(swift|m)

do {
    let session = AVAudioSession.sharedInstance()
    try? session.setCategory(.playAndRecord, options: [.defaultToSpeaker, .allowAirPlay, .allowBluetoothA2DP, .allowBluetooth])
    try? session.setActive(true, options: [])
}

Usage

The spokestack-tray-ios example app uses the “Spokestack” wake word and sample Minecraft NLU models.

To create your own copy of the Spokestack/Minecraft Helper, add the SpokestackTray framework:

import SpokestackTray
import Spokestack

override func viewDidLoad() {

    super.viewDidLoad()

    let configuration: TrayConfiguration = TrayConfiguration()

    /// When the tray is opened for the first time this is the synthesized
    /// greeting that will be "said" to the user

    configuration.greeting = """
    Welcome! This example finds tutorials for Minecraft crafting. Try saying, \"How do I make a castle?\"
    """

    /// When the tray is listening or processing speech there is a animated gradient that
    /// sits on top of the tray. The default values are red, white and blue

    configuration.gradientColors = [
        "#61fae9".spstk_color,
        "#2F5BEA".spstk_color,
        UIColor.systemRed
    ]

    /// Apart of the initialization of the tray is to download the nlu and wake word models.
    /// These are the default Spokestack models, but you can replace with your own

    configuration.nluModelURLs = [
        NLUModelURLMetaDataKey: "https://d3dmqd7cy685il.cloudfront.net/nlu/production/shared/XtASJqxkO6UwefOzia-he2gnIMcBnR2UCF-VyaIy-OI/nlu.tflite",
        NLUModelURLNLUKey: "https://d3dmqd7cy685il.cloudfront.net/nlu/production/shared/XtASJqxkO6UwefOzia-he2gnIMcBnR2UCF-VyaIy-OI/vocab.txt",
        NLUModelURLVocabKey: "https://d3dmqd7cy685il.cloudfront.net/nlu/production/shared/XtASJqxkO6UwefOzia-he2gnIMcBnR2UCF-VyaIy-OI/metadata.json"
    ]
    configuration.wakewordModelURLs = [
        WakeWordModelDetectKey: "https://d3dmqd7cy685il.cloudfront.net/model/wake/spokestack/detect.tflite",
        WakeWordModelEncodeKey: "https://d3dmqd7cy685il.cloudfront.net/model/wake/spokestack/encode.tflite",
        WakeWordModelFilterKey: "https://d3dmqd7cy685il.cloudfront.net/model/wake/spokestack/filter.tflite"
    ]

    configuration.cliendId = "YOUR_CLIENT_ID"
    configuration.clientSecret = "YOUR_CLIENT_SECRET"

    /// The handleIntent callback is how the SpeechController and the TrayViewModel know if
    /// NLUResult should be processed and what text should be added to the tableView.

    let greeting: IntentResult = IntentResult(node: InterntResultNode.greeting.rawValue, prompt: configuration.greeting)
    var lastNode: IntentResult = greeting

    configuration.handleIntent = {intent, slots, utterance in
        switch intent {
            case IntentResultAmazonType.repeat.rawValue:
                return lastNode
            case IntentResultAmazonType.yes.rawValue:
                lastNode = IntentResult(node: InterntResultNode.search.rawValue, prompt: "I heard you say yes! What would you like to make?")
            case IntentResultAmazonType.no.rawValue:
                lastNode = IntentResult(node: InterntResultNode.exit.rawValue, prompt: "I heard you say no. Goodbye")
            case IntentResultAmazonType.stop.rawValue,
                    IntentResultAmazonType.cancel.rawValue,
                    IntentResultAmazonType.fallback.rawValue:
                lastNode = IntentResult(node: InterntResultNode.exit.rawValue, prompt: "Goodbye!")
            case IntentResultAmazonType.recipe.rawValue:

                if let whatToMakeSlot: Dictionary<String, Slot> = slots,
                    let slot: Slot = whatToMakeSlot["Item"],
                    let item: String = slot.value as? String {

                    lastNode = IntentResult(node: InterntResultNode.recipe.rawValue,
                                            prompt: """
                                            If I were a real app, I would show a screen now on how to make a \(item). Want to continue?
                                            """
                                )
                }

            case IntentResultAmazonType.help.rawValue:
                lastNode = greeting
            default:
                lastNode = greeting
        }

        return lastNode
    }

    /// Which NLUNodes should trigger the tray to close automatically

    configuration.exitNodes = [
        InterntResultNode.exit.rawValue
    ]

    /// Callback when the tray is opened. The call back is called _after_ the animation has finished

    configuration.onOpen = {
        LogController.shared.log("isOpen")
    }

    /// Callback when the tray is closed. The call back is called _after_ the animation has finished

    configuration.onClose = {
        LogController.shared.log("onClose")
    }

    /// Callback when a `TrayListenerType` has occured

    configuration.onEvent = {event in
        LogController.shared.log("onEvent \(event)")
    }

    let tray: SpokestackTrayViewController = SpokestackTrayViewController(self, configuration: configuration)
    tray.addToParentView()
    tray.listen()
}

clientId and clientSecret

The clientId and clientSecret props are where you pass your API tokens generated in your Spokestack account. First, create a free account. Then, generate a token on the account settings page. Don’t worry, there’s no hidden subscription.

nluModelUrls

These URLs point to the sample Minecraft NLU model files we have hosted on a CDN, which are downloaded automatically by SpokestackTray the first time the app launches (and only the first time). The files are then saved to the app’s cache for future use so they only need to be downloaded once. NLU models can vary drastically in size, and it’s a good idea to include them in the app bundles and instead load them lazily.

At this point, you may be wondering what an “NLU” does. While Automatic Speech Recognition provides us with a way to process speech into text, that text is rarely enough to figure out what the user wants the app to do. Natural Language Understanding (NLU) is the next step to process the text into what voice platforms call “intents”.

A good example is searching with voice. If your app has just said, “What would you like to search for?” and the user says, “Bananas,” you can be reasonably sure that the user means for the app to search for bananas without the help of an NLU.

But if the user initiated the interaction and said, “Search for bananas”, the NLU can parse that statement into an intent (e.g. “search”) with variables (e.g. “bananas”). If you were only using ASR, you’d probably end up searching for the whole sentence, “search for bananas”, rather than just “bananas”, which may yield different results. For more information on NLU in Spokestack, please check out our guide. If you’ve already built an NLU model in Alexa, Dialogflow, or Jovo, check out our guide on exporting existing NLU models from other platforms.

handleIntent

So, intents are commands for the app based on what the user said. Now that you know how you get intents, it’s your responsibility to respond to those intents. There are two questions to answer for any given intent:

What should the app say in response to the user? This is the return value of handleIntent and is always required.
Should the app update the UI? Note that not all intents will need to make UI changes. In the voice search example, if the user has just searched for “bananas”, the answer to question #1 might be to say, “Here are your search results.” The answer to question #2 would probably be to show the search results. Remember that the NLU doesn’t do the search for you; it just tells you the proper search terms.

These answers could be written like this…

configuration.handleIntent = {intent, slots, utterance in

    if intent == "search" {
        let results = searchService.search(slots.ingrident)
        viewModel.addSearchResults(results)

        // Return a response
        return {
            prompt: "Here are your search results.",
            node: "search_results"
        }
    }
}

The prompt will be synthesized using Spokestack’s TTS service, and then played using the device’s native audio player (unless the tray is in silent mode).

The node property is metadata to help you track conversation state, and the value is completely up to you. The only reason SpokestackTray needs it is to determine whether to listen again after the prompt has been said.

If the node is specified in the exitNodes prop, the conversation will stop and SpokestackTray will close.

If the node is not an exit node, SpokestackTray will stay open and listen to the user again, and the process will repeat.

Conclusion

Hopefully, we’ve given you a glimpse into just how powerful spokestack-tray-ios can be. Add the SpokestackTray framework to your iOS app to start building elegant conversational user experiences.

For complete documentation, check out spokestack-tray-ios on GitHub.

For support, we offer multiple support channels to help you get started.

Integrating Spokestack in React Native

Tue, 01 Sep 2020 00:00:00 GMT

Not long ago, adding voice to any mobile app was not only time-consuming, but difficult and convoluted. This was especially true when adding voice experiences on multiple platforms, as might be the case when using React Native. Our mission at Spokestack is to make it as easy as possible to make your apps fully voice-enabled.

Introducing react-native-spokestack-tray!

react-native-spokestack-tray is a React Native component that is designed to work in any application, regardless of its layout or navigation. It uses multiple existing React Native plugins, including react-native-spokestack, to add voice experiences. With on-device wake word, ASR, and NLU, the tray’s silent mode works completely offline–TTS is the only service that requires a network.

This tutorial will guide you through the process of installing react-native-spokestack-tray as well as using the <SpokestackTray /> component to respond to user intents. We won’t go through the process of setting up a new React Native project, but be sure to use the React Native CLI path and not expo.

Installation

First, install the dependencies. Here’s a one-liner to install react-native-spokestack-tray and its peer dependencies.

npm install react-native-spokestack-tray react-native-spokestack @react-native-community/async-storage @react-native-community/netinfo react-native-video rn-fetch-blob react-native-haptic-feedback react-native-linear-gradient react-native-permissions

If you’re wondering why these dependencies are needed, we’ve listed all of their purposes in the tray’s README.

Now that you have all of the node modules, we need to update some native files. We’ll go through each of them by platform.

iOS installation

Edit Podfile

Our main dependency (react-native-spokestack) makes use of relatively new APIs only available in iOS 13+. Make sure to set your deployment target to iOS 13 at the top of your Podfile:

platform :ios, '13.0'

We use react-native-permissions to check and request the Microphone permission (iOS and Android) and the Speech Recognition permission (iOS only). This library separates each permission into its own pod to avoid inflating your app with code you don’t use. Add the following pods to your Podfile:

target 'SpokestackTrayExample' do
  # ...
  permissions_path = '../node_modules/react-native-permissions/ios'
  pod 'Permission-Microphone', :path => "#{permissions_path}/Microphone.podspec"
  pod 'Permission-SpeechRecognition', :path => "#{permissions_path}/SpeechRecognition.podspec"

We need to use use_frameworks! in our Podfile because a couple of our dependencies are written using Swift.

target 'SpokestackTrayExample' do
  use_frameworks!
  #...

For the time being, use_frameworks! does not work with Flipper, so we also need to disable Flipper. Remove any Flipper-related lines in your Podfile. In React Native 0.63.2, they look like this:

  # X Remove or comment out these lines X
  use_flipper!
  post_install do |installer|
    flipper_post_install(installer)
  end
  # XX

Remove your existing Podfile.lock and Pods folder to ensure no conflicts, then install the pods:

$ npx pod-install

Refer to the Podfile in our example for a working Podfile.

Edit Info.plist

Add the following to your Info.plist to enable permissions. In XCode, also ensure your iOS deployment target is set to 13.0.

<key>NSMicrophoneUsageDescription</key>
<string>This app uses the microphone to hear voice commands</string>
<key>NSSpeechRecognitionUsageDescription</key>
<string>This app uses speech recognition to process voice commands</string>

Edit AppDelegate.m

Add AVFoundation to imports

#import <AVFoundation/AVFoundation.h>

AudioSession category

Set the AudioSession category to enable microphone input and play from the speaker by default. This also enables input and playback over bluetooth.

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
  AVAudioSession *session = [AVAudioSession sharedInstance];
  [session setCategory:AVAudioSessionCategoryPlayAndRecord
     mode:AVAudioSessionModeDefault
  options:AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionAllowAirPlay | AVAudioSessionCategoryOptionAllowBluetoothA2DP | AVAudioSessionCategoryOptionAllowBluetooth
    error:nil];
  [session setActive:YES error:nil];

  // ...

Remove Flipper

While Flipper works on fixing their pod for use_frameworks!, we must disable Flipper. We already removed the Flipper dependencies from Pods above, but there remains some code in the AppDelegate.m that imports Flipper. There are two ways to fix this.

You can disable Flipper imports without removing any code from the AppDelegate. To do this, open your xcworkspace file in XCode. Go to your target, then Build Settings, search for “C Flags”, remove -DFB_SONARKIT_ENABLED=1 from flags.
Remove all Flipper-related code from your AppDelegate.m.

In our example app, we’ve done option 1 and left in the Flipper code in case they get it working in the future and we can add it back.

Android installation

Edit AndroidManifest.xml

    <!-- For TTS -->
    <uses-permission android:name="android.permission.INTERNET" />
    <!-- For wake word and ASR -->
    <uses-permission android:name="android.permission.RECORD_AUDIO" />

Edit app/build.gradle

Add the following lines:

android {
    // ...
    defaultConfig {
        // ...
        multiDexEnabled true
    }
    // ...
    packagingOptions {
        exclude 'META-INF/INDEX.LIST'
        exclude 'META-INF/DEPENDENCIES'
    }
}

Usage

The react-native-spokestack-tray example app uses the “Spokestack” wake word and sample Minecraft NLU models.

In this example, the following code is used to add the <SpokestackTray /> component:

<SpokestackTray
  clientId={process.env.SPOKESTACK_CLIENT_ID}
  clientSecret={process.env.SPOKESTACK_CLIENT_SECRET}
  handleIntent={handleIntent}
  nluModelUrls={{
    nlu: 'https://d3dmqd7cy685il.cloudfront.net/nlu/production/shared/XtASJqxkO6UwefOzia-he2gnIMcBnR2UCF-VyaIy-OI/nlu.tflite',
    vocab:
      'https://d3dmqd7cy685il.cloudfront.net/nlu/production/shared/XtASJqxkO6UwefOzia-he2gnIMcBnR2UCF-VyaIy-OI/vocab.txt',
    metadata:
      'https://d3dmqd7cy685il.cloudfront.net/nlu/production/shared/XtASJqxkO6UwefOzia-he2gnIMcBnR2UCF-VyaIy-OI/metadata.json'
  }}
/>

`clientId` and `clientSecret`

Rather than hardcoding these values, we recommend saving them in your environment on your local machine and in CI for deployments. Once they’re saved, run the React Native packager in a new terminal and start the app using npm run ios or npm run android -- --device (note that Android requires a real device for ASR to work).

`nluModelUrls`

These URLs point to the sample Minecraft NLU models files we have hosted on a CDN, which are downloaded automatically by SpokestackTray the first time the app launches (and only the first time). The files are then saved to the app’s cache for future use so they only need to be downloaded once. NLU models can vary drastically in size, and we thought it better not to include them in the app bundles, but instead load them lazily.

At this point, you may be wondering what an NLU does. While Automatic Speech Recognition provides us with a way to process speech into text, that text is rarely enough to figure out what the user wants the app to do. Natural Language Understanding (NLU) is the next step to process the text into what voice platforms call “intents”.

A good example is searching with voice. If your app has just said, “What would you like to search for?” and the user says, “Bananas”, you can be reasonably sure that the user means for the app to search for bananas without the help of an NLU.

`handleIntent`

What should the app say in response to the user? This is the return value of handleIntent and is always required.
Should the app update the UI? Note that not all intents will need to make UI changes.

In the voice search example, if the user has just searched for “bananas”, the answer to question #1 might be to say, “Here are your search results.” The answer to question #2 would probably be to show the search results. Remember that the NLU doesn’t do the search for you; it just tells you the proper search terms.

These answers could be written like this…

function handleIntent(
  intent: string,
  slots?: any,
  utterance?: string
) {
  if (intent === 'search') {

    // Search for recipes with "bananas"
    const results = searchService.search(
      slots.ingredient
    )

    // Show the results
    navigation.navigate('SearchResults', {
      results
    })

    // Return a response
    return {
      prompt: 'Here are your search results.',
      node: 'search_results'
    }
  }
}

The prompt will then be synthesized using Spokestack’s TTS service. It then gets played using the device’s native audio player (unless the tray is in silent mode).

If the node is specified in the exitNodes prop, the conversation will stop and SpokestackTray will close.

If the node is not an exit node, SpokestackTray will stay open and listen to the user again, and the process will repeat.

Conclusion

Hopefully, we’ve given you a glimpse into just how powerful react-native-spokestack-tray can be. Add the SpokestackTray component to your React Native app to start building elegant conversational user experiences.

For complete documentation, check out react-native-spokestack-tray on GitHub.

For support, we offer multiple support channels to help you get started.

What If You're the Product?

Wed, 29 Jul 2020 00:00:00 GMT

“If you’re not the customer, you’re the product.” By now, tech-savvy consumers are familiar with this warning. It’s almost a trope. Usually it’s a caveat about Google’s collection of personal data or Facebook’s stewardship of their own data trove, but there are plenty of other applications as well. And it’s true: Personal data can be dangerous in the wrong hands.

It’s not true only for consumers, though; it’s also true for your business. For real-world examples, see Allbirds. Or these Alexa Fund participants. Or the creator of an eerily Echo-like device. There’s a reason trade secrets are called “secrets”.

An uncomfortable truth

If you’re browsing this site, it’s probably safe to say you’re familiar with voice technology and its meteoric rise over the past several years. Smart speakers are in people’s homes, voice assistants are on consumers’ phones, and voice is being touted as the next great way for businesses to engage their customers. To do that, you just set up a “skill” or “action” on these platforms—you just configure some text input and output, and the smart speaker/voice assistant platforms will do all the heavy lifting of “understanding” your users’ verbal requests and turning your text responses into audio to read to them. You give them the data, and they give you … exposure.

This might seem like a reasonable trade—after all, you can’t build automatic speech recognition (ASR), natural language understanding (NLU), and text-to-speech (TTS) systems—that’s a ton of work and maintenance for an uncertain payoff, and systems like that aren’t anywhere near the scope of your business. So you sign up for a free developer account and make sure you have a presence on the platform. It’s not like you gave Google or Amazon credentials to your database, so what are you actually giving up in the process?

To answer that question, we have to talk a bit about how the voice systems work. By now it won’t surprise you to learn that they’re built on massive troves of data, some of it public, but much of it “proprietary”. You don’t have to know how to flow a tensor to know that machine learning, much of it in the form of neural networks, powers some of today’s most impressive software, voice tech systems among them.

I’m greatly simplifying here, but to train an ASR system, you feed a model a whole bunch of audio data (thousands of hours, if you have it) along with transcripts of that audio; thanks to the magic of math, the model learns to predict the latter from the former. Once you have a baseline system trained, you can improve it by feeding it raw audio (different from the original training data), letting it do the transcription, and correcting the transcriptions after the fact if necessary. This is what Amazon and Google do.

Next in the process, NLU models learn to turn natural language (transcripts from an ASR, which can be single words or full sentences) into something a little more structured so that software can reliably act on it, and TTS systems learn to turn text responses into audio so that they can be read to the user.

So there you have it: The data you provide in order to set up a voice app on one of the big platforms, combined with the interactions users have with your app, is all the vendors need to continuously improve their systems. That’s not so bad, though, right? After all, you want their voice tech to be as good as it can be so that your users have good experiences with your app.

Keep the cautionary tales from Amazon in mind as we go further down the rabbit hole.

A current research area in the NLU field is the so-called “end-to-end” dialogue system. It’s another neural model, but instead of translating audio into text or text into “meaning”, this one takes text input (a user request) and directly returns a text response. A quick Google search will turn up a wealth of papers on creating the models themselves and using automated systems and crowdsourcing to generate or augment training data for such systems.

But what’s better than data generated by one model under a single set of conditions? That’s right—data collected from and annotated by humans. Enter one of the newest voice assistant features.

As usual, the vendors all have a different name for this feature. Google calls it App Actions, Apple has Siri Shortcuts, and Amazon just announced Alexa for Apps. They’re all fundamentally the same thing, though, with slightly different trimmings: They allow you to surface information from deep within your app in response to a request to a voice assistant controlled by someone else.

Since I’ve spent some words focusing on Amazon already, let’s dig a little further into their implementation of this feature. Take a look at the response format for an Alexa skill that wants to connect to an app. Notice the prompts section, which lets you provide some text for Alexa to read right before it drops the user into your app to see the response. Convenient, right? Well, sort of. If a user’s started an interaction like this via voice, it might be nice to actually be able to finish it with voice inside the app (more on that later), but at least this is a smooth-ish transition between the two, and it helps the user get some answer even if the screen is locked.

But stop for a moment to think about what data Amazon has about the user interaction at this point:

audio of the user making the request
Amazon’s ASR result for that audio
the user’s “intent” (or distillation of the ASR result into an action) that you, the developer, provided Amazon with as part of creating your skill
a text response to that intent, again provided by you, an actual human

In other words, they’d have a pretty good pipeline for collecting training data for an end-to-end dialogue system, like this one that some of their engineers experimented with last year. What could they do with such a model? For starters, they could learn to answer initial user queries well enough to keep those users inside Alexa instead of delivering them to third-party apps at all. If they were so inclined, they could also aggregate data about popular queries and responses to prioritize development of new business lines for Amazon.

I don’t have any evidence to make concrete accusations about such things, but it doesn’t seem outside the realm of possibility. Keep in mind that encouraging developers to let Alexa access individual app features might also encourage those developers to provide data for features that aren’t a good fit for the smart speaker medium or were “too valuable to put on Alexa”. In other words, a feature that deep links into apps can potentially give them access to data that they can’t just mine from their Alexa Skills Store.

And text prompts aren’t the only data collection boon here. Alexa for Apps also “lets” you provide an app store ID so that Alexa can link a user directly to your app’s install page if they don’t already have it installed. This might boost your app’s discoverability if your skill somehow happens to be more popular than your mobile app, but it also lets Amazon link natural language requests to app store offerings, which is an interesting bit of data to have if you’re building, say, a recommendation engine.

Yeah, but what can I do about it?

Given all this information, what’s the app developer to do? It’d be easy enough for me to recommend total abstinence, the business equivalent of deleting your Facebook account after the Cambridge Analytica debacle. An isolationist approach might not be fair to you, though. People use these voice assistants, so not participating in them or their advanced features might make your app appear to lag behind similar apps that aren’t as cautious.

Instead, I suggest instead that you think carefully about the data you’re providing in your user responses—the text that gets read to the user in a smart speaker app or before the voice assistant drops the user into your mobile app. For any given interaction, can you respond in a way that’s helpful to the user but also vague, redirecting them into an experience that you control for more information?

“But what if the user wants or needs a voice-only experience?” you might ask. That’s a fair question—smart speakers have a distinct accessibility advantage, and some requests are just easier to speak than to tap. Finally, I have some good news for you: It’s getting easier to put voice interactions inside your app and stop relying on the ASR and other components provided by giant companies.

This is Spokestack’s mission: We exist to enable businesses to take back their voice presence. Voice is our business, not advertising or retail. We provide native mobile libraries that bring a full suite of ASR and natural language tools directly to your app. This, in turn, helps you keep your customer interactions private and prevents your thoughtful responses to user queries from being turned into training data that can be used against you.

In fact, we go a couple steps further than what the major voice platforms currently let you do. Spokestack can help you:

Give your app its own wake word (so a user can start a voice-only interaction from within the app, as if they’d said “Hey Google/Siri” to their phone).
Create your own voice to use for responses, so users can differentiate your brand from a generic AI voice.

That’s all well and good for the mobile use case, but what about the smart speakers themselves, where voice is the only medium? You have to give detailed data in those responses, right? Maybe, but this is where your brand’s custom voice comes into play. You don’t have to provide text responses that can be immediately appropriated by the smart speaker platform; you can use a separate TTS service to synthesize your response and just provide the smart speaker with audio. If they still want to take your data, they could of course run this through their own ASR systems, but that injects more error and cost into the process for them. You can do something similar with the NLU systems provided by the smart speakers too, but that’s a topic for another post.

Don’t concede the entire medium of voice interaction to your competitors (current or potential) just because they’ve built a stack of audio and language processing tools to “offer” to you. Don’t give them any more data than necessary. Let your brand be its own mouthpiece. Ask us how.

Jovo Support for Spokestack Enables Mobile App Voice Assistants

Wed, 15 Jul 2020 00:00:00 GMT

This new integration helps developers build cutting edge conversational experiences for mobile apps using their Jovo interaction models and Spokestack’s mobile ASR, NLU, and TTS services.

Like so many in the voice community, we’re big fans of the Jovo Framework! We share their love of open-source software as well as their “write once, run everywhere” approach to conversational assistants. That’s why we’re so excited to announce Spokestack integration with Jovo.

How does the Jovo and Spokestack integration work?

Starting today, Jovo developers can take the following steps to begin building mobile voice experiences based on their Jovo interaction model with Spokestack:

Create a Spokestack account to gain access to a Spokestack API key: https://spokestack.io/create.
Add your Spokestack API key and secret to your Jovo project’s configuration file.
Run the jovo build command to create a platforms/spokestack directory.
Run the jovo deploy command to upload your model to Spokestack.
Spokestack will then train the imported model for use with Spokestack’s embedded NLU solutions for iOS, Android, and React Native.
From there, developers can either follow our port a smart speaker app to mobile tutorials or build whatever mobile voice experience they have in their heads!

Why would I import my Jovo model to Spokestack?

If you’ve experienced the power of using Jovo to build a conversation that works across the Alexa, Google, and Bixby smart speaker platforms, you may have wondered about transferring that conversational experience to a mobile app. With Spokestack, you can.

We take the interaction model that powers your conversation on smart speakers and port it to an embedded model that will work on a smartphone. So now users can say, “Siri, open {your app name}” to begin a mobile app conversational experience just like the one they have on smart speakers.

New to mobile development? Want to see how Spokestack works on mobile?

We realize that the barrier to building conversations on mobile is lacking the skills to build apps that work on iOS and Android. Sometimes it’s thinking through how voice would work on mobile. We’ve tried to address both issues.

On iOS, we have Spokestack Studio, which was built to show developers exactly how wake words, speech recognition, text-to-speech, and intent classification work on mobile. It even includes a port of the Alexa Minecraft Helper skill to iOS.

Spokestack Studio isn’t available for Android yet, but Spokestack does work on Android. We have another tutorial for porting an Alexa Skill to Android that walks you step-by-step from Alexa skill to Android app. After going through our tutorial, you should be able to port your smart speaker skill to the mobile platform of your choice.

The future with Jovo

Like Jovo, we believe in sharing open-source software and building great developer tools. By working together, we can extend the capabilities of both conversational and mobile developers on mobile. We’ll continue working with Jovo to improve the Spokestack integration, including adding our ASR and TTS services options for Jovo developers. Besides the technical integration, we’re also announcing our financial support of Jovo through their Open Collective initiative. We plan on being a long-term partner and supporter of the Jovo community.

We’re excited to see what you’ll build using Jovo and Spokestack for mobile voice apps! Tell us about it at hello@spokestack.io.

Wake Words for Mobile Apps

Thu, 09 Jul 2020 00:00:00 GMT

Amazon added the “Hey Alexa” wake word to its mobile app. This is going to change user perceptions of mobile voice apps.

One of the main hurdles I’ve encountered when explaining mobile voice apps to people is that they don’t seem to understand how to start talking to mobile apps. For some reason, it’s difficult for people to get their heads around the idea that a mobile app is the same as a smart speaker such as an Alexa Skill or a Google Assistant Action.

If you want our smart speaker skill The Bartender to recommend a cocktail and help you make it, you have to say the following, depending on which smart speaker assistant you’re talking to:

“Hey Alexa, ask the Bartender for a drink.”

“Hey Google, open the Bartender.”

After invoking the Bartender, the user can start a search for a drink by name or ingredient. Pretty simple, right?

So why is it such a leap to think the same interaction can’t happen on mobile?

Because users haven’t been shown enough examples of mobile apps carrying out tasks by voice yet.

We think that if Alexa trains users to understand that they can say

“Hey Siri, open Alexa”

to have a conversation, it will change users’ expectations of mobile voice conversations.

Hey #voicefirst folks, pls read this piece by @sarahintampa on TC: https://t.co/xqq9KM8Adw

1/ Amazon just gave the Alexa mobile app the "Hey Alexa" wakeword. This is a big deal. Teaching users to say "Hey Siri, open Alexa" to start a conversation will kick off #mobilevoice
— Spokestack (@spokestack) July 8, 2020

That’s because the Alexa mobile app will help users make the mental connection that, in addition to Siri and Google Assistant, mobile apps can carry on conversations with them on their mobile device. Specifically, they can have robust conversations that help them complete tasks they wouldn’t otherwise be able to accomplish as fast. It might be because they are driving or in some sort of hands-free situation. Or might just be because voice is faster and easier than tapping and typing for certain tasks. Either way, convenience tends to drive consumer adoption once they see it.

For example, did you know you can open any app on your phone by asking for it? Think of your favorite app, then ask Siri or Google to open it for you. Seriously, try it now!

That’s convenient and definitely faster than scrolling through pages of apps to get to the service or content you need. So the next step is adding voice to your app so you can talk with your customer, just like Alexa is going to do.

To be fair, Alexa isn’t the first mobile app to add a voice interface or wake word. Spotify, Home Depot, Sephora, Pandora, and Snapchat have all added voice to their apps with varying degrees of interaction. Most focus on initiating a product search, which makes sense. It’s just that the interactions, sans the ones built by Houndify, are not that robust.

We think all of that will change which is why we built Spokestack. So if you’re looking for a wake word, on-device NLU, or custom synthetic voice to speak to your customer, email us at hello@spokestack.io.

Today, we export to independence!

Mon, 29 Jun 2020 00:00:00 GMT

With apologies to President Bill Pullman, let’s declare our independence from smart speakers. The voice interface, and the skills you’ve developed using it, are wonderful resources that are currently stuck in kitchens and dens all over the world. It’s time to free them to be carried with your users wherever they are, not wherever Amazon and Google allow them to be. Declare skill independence with Spokestack!

What’s independence?

NLU-capable computers are everywhere — your smartphone, your smartwatch, your laptop, your TV — but nobody is building NLU-enabled voice assistants for them. Smart speakers, on the other hand, are essentially NLU-only skill/action platforms that artifically limit what your voice assistants can accomplish. Why do you have to choose between running free of platform restrictions, or having an NLU? That’s why we built Spokestack. Spokestack runs your independent voice assistants on mobile platforms like Android and iOS, along with providing integrated ASR and TTS services, in convenient, cross-platform, open-source libraries using a simple, consistent API. You can take your independent voice assistants with you as you walk away from the demolished alien mothership, instead of having it stuck in the house on a smart speaker!

We built Spokestack when we realized that NLU, ASR, and TTS on mobile platforms were all siloed between service providers, none of them considering the developer experience when creating complete,independent voice assistants. Not only that, all the service providers have business models that incentivize them to force everything to the cloud. So we built the Spokestack NLU service, using state of the art intent and slot TensorFlow machine learning that is familiar to Alexa and DialogueFlow developers, capable of running entirely on the mobile device to deliver speedy, privacy-preserving results. Originally, Spokestack was just used in our own multi-modal, cross-platform independent voice assistants, but since January we’ve been focused on creating a simple way for all developers — mobile, voice, smart speaker, and front-end — to make their own independent voice assistants for mobile platforms.

Why do I want to declare independence?

Unlike smart speakers, independent voice assistants can take advantage of multiple interfaces modalities to allow your users to get the job done using whatever interaction they find convenient. Visual, haptic, and now voice user interfaces are all accessible to apps using Spokestack.

What do you gain from that? You control and learn from the NLU classification of user utterances, not the smart speaker platform. When you’re stuck on Alexa, do you get the raw utterance that your user uses when they’re in your skill, even misclassified or misunderstood ones? With Spokestack, you control your data and your users’ data.

If you’ve ever read the EULA for smart speaker platforms, you know how rapacious they are with your data. With Spokestack, you keep the most useful bits of your data instead of feeding it (for free) to the FAANG.

What do I have to do to declare independence?

To further that mission to create a complete developer experience for developing voice apps, we’re excited that now you can export your smart speaker skill to run on-device in the major mobile platforms, and declare your independence as an independent voice assistant from smart speakers!

You can leverage existing ASR and TTS services, which are the parts that actually benefit from scale and are difficult to DIY. Spokestack provides a seamless, unified API across mobile platforms (iOS, Android, and React/React Native) that makes converting your skill into a indepedent voice app easy.

Finally, like President Bill Pullman, you must have the hubris to believe.

We are fighting for our right to live. To exist. And should we win…the day the world declared in one voice…Today we celebrate our Independence Day!

Nota bene

While it’s fun to cheekily reference a cheesy scifi movie to help illustrate what our technology company can help with, please take a bit of time to learn about celebrating Independence Day during the eras of Emancipation and Reconstruction!

Choosing the Right iOS Wake Word Service

Tue, 16 Jun 2020 00:00:00 GMT

Spokestack offers two wake word services in iOS: appleWakeword and tfLiteWakeword. Which should you use?

The answer, of course, is up to you! Spokestack always gives you options, because one size does not fit everyone, and because, like you, we hate vendor lock-in.

Wait, there’s a wake word?

Indeed! If you want your app to be controllable purely by voice, you need a wake word — a word (or short phrase) that tells your app “the next thing the user says is meant for you”. The wake word detection component in Spokestack is responsible for detecting any of a user-defined set of keyword phrases in soft real time. Once detected, the Spokestack pipeline activates, providing you with an activation event and triggering the configured speech recognition service. The accuracy, speed, and flexibility of wake word detection depends on which service you configure.

Ok, so which one should I use?

In general, tfLiteWakeword will have better accuracy and faster activation. appleWakeword is intended for quick demos where you want to try out different wake words for UX research and not have the overhead of having to build a TensorFlow model for every option. Let’s discuss the guidelines for using them in more detail by asking two questions.

What wake word do we want our app to use?

What wake word do we want our app to use? Tricky question! Answering the question means UX research, brand identity discussions, and even linguistics consulting. Of course Spokestack is here to help with that.

Prefer

Spokestack’s appleWakeword uses Apple’s free on-device ASR to transcribe all speech heard while the Spokestack pipeline is running, and then filters that speech for the wake word(s) that you specify in the SpeechConfiguration.wakewords configuration. This allows you to have a tight testing cycle when trying out the UX of different wake words. Boss doesn’t like your favorite “HAL” wake word idea? Just change a single line and try out their “GUNTER” idea. Just want your app to get an event when a user says “take a selfie”, or even allow your users to choose their own wake words? No need to muck with fancy embedded computing models — just change one line of code!

Avoid

appleWakeword should not be used in any published app, both because its performance will always be slower than tfLiteWakeword and because it depends on running constant ASR over all speech while the pipeline is running, which is both a privacy concern and an overuse of resources for the actual task.

How do I distribute my app with a fast, accurate, efficient, on-device wake word?

Spokestack’s tfLiteWakeword fits the bill for you! You’ll gain fast, accurate, efficient, wake word activation that runs entirely on-device. It features a state-of-the-art machine learning pipeline using attention-based models; they operate continuously, each feeding output into the next, for both efficiency and accuracy.

To get you started, Spokestack provides pretrained TensorFlow models that enable on-device wake word detection. These free models, however, only recognize the word “Spokestack”; in order to have your app respond to a different word or phrase, you’ll need your own custom models.

Prefer

Work with us to develop a custom TensorFlow model, and then distribute that model with your app configured to use tfLiteWakeword. When testing your app using tfLiteWakeword, be sure to consult the Spokestack pipeline model hyperparameter configuration guide.

Avoid

Don’t use tfLiteWakeword without a TensorFlow model of the wake word you want to use, or when you don’t know what wake word you want to activate your app.

Announcing the Export to Independence Contest

Mon, 15 Jun 2020 00:00:00 GMT

Today we are announcing our first contest!

Let’s get right to the details.

What

Export to Independence is a community contest to see who can best port a smart speaker voice app to a mobile voice assistant using Spokestack. The contest encourages participants to export an Alexa or Dialogflow interaction model to create a mobile voice app. Participants will be judged on their creativity and innovation on creating intelligent mobile voice app experiences.

Prizes

Grand prize winner will receive \$3,000 for best overall voice experience on iOS or Android
4 runners up will receive \$500
\$5,000 will be donated to the Southern Poverty Law Center on behalf of all participants who may choose to be listed or anonymous in the donation.
All teams that submit a final app will receive a Spokestack t-shirt for up to 3 team members
Every person who enters and creates an account will receive a Spokestack Smiley Facemask

Dates - Extended!

Contest officially starts Monday, June 15th
Submissions must be submitted by Tuesday, September 15th
Winners will be announced Thursday, October 1st

Requirements

All submissions must use Spokestack’s iOS, Android or React Native libraries in their applications.
Developers are not required to use an existing Alexa or Google Assistant interaction model for submission. We realize many folks use other NLU or custom interaction models for their conversational experiences.
The Spokestack Export to Independence Developer Contest is open only to individuals who are legal residents of the United States of America or the District of Columbia who are eighteen (18) years of age or older at the time of entry.

Support

Tutorials
Documentation
Community Forum
Live Video Workshops: Dates to be announced

Legal

All contestants must agree to the terms of the contest.

If you have any further questions, please contact us. We’re incredibly excited to begin bringing together the smart speaker and mobile developer communities! We believe this will be the start of a completely new type of mobile app and the realization of voice-driven user experiences promised all the way back in 2011 when Siri first launched.

We look forward to seeing what you build, and we’re thrilled to help you!

Good luck!

Implementing Reprompts with Spokestack

Tue, 09 Jun 2020 05:00:00 GMT

Reprompts are a convenient feature provided by most smart speaker platforms. In plain English, a reprompt is a special message given to the user under the following circumstances:

The app requests information from the user, leaving the mic open to listen for the answer
The user remains silent for a pre-set length of time

Platforms differ in the number of reprompts they allow an app to give the user before the platform itself takes control and shuts off the mic — some only allow one, some will give you the chance to deliver up to four different prompts before finally giving up on the user. With Spokestack, you can choose that number for yourself. In an app where the user is expected to switch back and forth between voice and gesture input frequently, reprompts might be inappropriate altogether, but in one that’s designed to be used hands-free, you might want to give the user a couple chances to answer a question.

In Spokestack, reprompting is a matter of responding to the timeout event sent by the speech pipeline. In Android, this event is received by the OnSpeechEventListener, in Swift by the SpeechEventListener, or in React Native by attaching a listener to the onTimeout event.

The specifics will vary based on how you’ve set up your app, but the basic pseudocode for the process would be:

if event is TIMEOUT:
  if dialog_manager.is_waiting_for_response():
    reprompt = get_next_reprompt()
    if reprompt:
      Spokestack.synthesize(reprompt)
      // re-open the microphone to listen for the
      // answer again, but don't do it here since
      // you don't want to listen to the reprompt itself

And that’s it! For the reprompts, at least. Configuring the dialog manager to know when reprompts are appropriate is left as an exercise to the reader. As always, if you have any issues, don’t hesitate to reach out via Discourse, Stack Overflow, or a GitHub issue.

Porting the Alexa Minecraft Skill to Android Using Spokestack

Tue, 09 Jun 2020 00:00:00 GMT

This post is part of the Porting a Smart Speaker Voice App to Mobile series, which discusses how to turn an Alexa skill into a mobile app using Spokestack as a replacement for Amazon’s voice services. Other articles in the series can be found at the following links:

Part 1: Voice Apps on Smart Speakers
Part 2: Voice Apps on Mobile
Part 3: Import an Alexa or Dialogflow Interaction Model
Tutorial: Create an Alexa-Compatible Dialog Manager in Swift
Tutorial: Porting the Alexa Minecraft Skill to iOS Using Spokestack
Tutorial: Porting the Alexa Minecraft Skill to Android Using Spokestack (You are here!)
Tutorial: Porting the Alexa Minecraft Skill to React Native Using Spokestack (Coming Soon)

This tutorial will walk you through the details of porting an Alexa skill to Android. To be specific, we’ll be recreating a skill that lets the user ask for a recipe in Minecraft. The full code for our finished app is here; feel free to download it for reference as you follow along. We’ll go through it step by step in the tutorial for sake of discussion.

First, a quick note on language choice. Android development has been moving toward Kotlin as its preferred language. Our other Android guides are written with that in mind, but we’re going to switch gears for this one. Amazon provides a Java SDK for Alexa development, so in order to ease friction between the two ecosystems, we’ll port the skill to Java instead of Kotlin. The example code here shouldn’t be too complex to translate into Kotlin if you’re familiar with it.

With that pretext out of the way, let’s get to coding. Our first job will be to establish the mobile-specific stuff you don’t have to do when setting up an Alexa skill.

App Scaffolding

On Android, the Activity is one of the fundamental building blocks of an app. Roughly speaking, an Activity is the code for a single scene’s behavior (minus a description of the visual layout, which is done elsewhere). If you’re using Android Studio — and you probably should be for this — you’ll want to open a new project and use the “Empty Activity” template. This will create all the boilerplate we’ll need, along with a MainActivity we’ll be referring to throughout the rest of this guide. When you’re setting up a new project, Android Studio also asks for a minimum SDK. We chose 21 for this guide.

To avoid cluttering this guide too much, we’ll omit the full build.gradle files and the detailed code for requesting microphone permissions in MainActivity. See the example files for copy/paste-able examples. Here are the important things you’re looking for in app/build.gradle:

The native-dependencies plugin application and the block associated with it that retrieves Spokestack’s native library (at the top of the file)
The ndkVersion line in the android block. See here for information on installing the NDK, and make sure your version number in build.gradle matches the one you install.
The compileOptions block, also under android. These options allow some of our dependencies to build properly.
The Spokestack library dependency and others associated with it (at the bottom of the file)

And in the build.gradle in your root directory:

A classpath dependency that retrieves the native-dependencies plugin

OK, time to dive into MainActivity and set up Spokestack to handle user speech.

Can you hear me now?

In order to turn your phone into a (better) smart speaker, you need to take control of the microphone and process user input from it. That’s why we requested the permissions in the previous section. The Spokestack component used to actually do something with that data is called SpeechPipeline. It handles collecting audio from the user and turning it into text (automatic speech recognition, or ASR), and there are other components for extracting meaning from that text and for generating audio in response to the user. These exist separately in the Spokestack library so that an app can pick and choose which ones it needs. We need them all for this app, so let’s make a class to contain and control them all. Let’s call it … I don’t know … Spokestack.

Since we’re building all three components there, this is another file we won’t discuss in its entirety; you can find the full version in the demo project. For the purpose of this guide, we’re just going to pretend it exists and talk about how to interact with it in MainActivity.

public class MainActivity extends AppCompatActivity {

    private Spokestack spokestack;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        this.viewBinding = ActivityMainBinding.inflate(getLayoutInflater());
        setContentView(viewBinding.getRoot());
        // see if we were granted the microphone permission
        // during a previous session; if so, go ahead and build
        // Spokestack components
        if (checkMicPermission()) {
            buildSpokestack();
        }
    }

    // other code...

    private void buildSpokestack() {
        // extract the models from the asset bundle if we need to
        checkForModels();
        this.spokestack = new Spokestack(getApplicationContext(), this);
        try {
            // start the speech pipeline
            this.spokestack.start();
            this.spokestack.launch();
        } catch (Exception e) {
            Log.e(logTag, "Problem starting Spokestack", e);
        }
    }

}

This is all we need to interact with the components we’ve built — the start method gives Spokestack control of the microphone via its SpeechPipeline so that we can start hearing the user. If you look at how we’ve set the pipeline up for this project, though (in Spokestack), you’ll notice a line that sets the pipeline’s “profile” to PushToTalkAndroidASR. This means that we’re not using a wake word (e.g., “Alexa”) to tell the app to start actively listening to the user. Spokestack does support this, and you can see an example configuration in our Android cookbook, but we’re going to use a button here for sake of simplicity. That means we’ll need a microphone button and a handler that starts sending audio through speech recognition when the button is tapped:

// still in MainActivity
    void activateAsrTapped(View view) {
        this.spokestack.activateAsr();
    }

That wasn’t so bad, was it? We’re manually activating the ASR, but the configuration we’ve set up in the Spokestack class will handle deactivating it after speech stops.

Note: We’re using the Android ASR here because it’s the easiest way to demo ASR, but it’s not available on all devices. See our ASR documentation for more information on it and the other ASR providers Spokestack integrates with.

Also note that the Android emulator cannot record audio. You’ll need to test ASR on a real device.

With those caveats behind us, our next job is to make an effort to actually understand the user…

Integrating NLU

Part 3 of this series includes a refresher on the concept of natural language understanding (NLU) as well as instructions for converting your Alexa model into a format usable by Spokestack, so we’ll just cover the Android-specific parts here.

The configuration for the Spokestack NLU is in the Spokestack file, just like our ASR setup. It requires three external files — a TensorFlow Lite model, a JSON metadata file that describes its output, and a vocabulary file used to transform ASR results into input for the model. These files aren’t huge (typically < 20 MB total), but they’re big enough that you probably want to distribute them compressed to keep your app download size down.

To do that, place all three files in the src/main/assets directory under your app’s root directory. You may have to manually create assets, but the others should have been created along with your project. Files in the assets directory have to be decompressed at runtime to be used. We typically do that by decompressing them to the application’s cache directory on startup, then on subsequent starts checking if the files exist and decompressing again if the user has cleared the cache or the app has been updated to a new version. This is the checkForModels() method from the previous section. In the demo app, this code is in MainActivity, but you could put it elsewhere for sake of cleanliness:

private void checkForModels() {
    // PREF_NAME and VERSION_KEY are static Strings set at the top of the file;
    // we want PREF_NAME to uniquely refer to our app, and VERSION_KEY to be
    // unique within the app itself
    if (!modelsCached()) {
        decompressModels();
    } else {
        int currentVersionCode = BuildConfig.VERSION_CODE;
        SharedPreferences prefs = getSharedPreferences(PREF_NAME, Context.MODE_PRIVATE);
        int savedVersionCode = prefs.getInt(VERSION_KEY, NONEXISTENT);

        if (currentVersionCode != savedVersionCode) {
            decompressModels();

            // Update the shared preferences with the current version code
            prefs.edit().putInt(VERSION_KEY, currentVersionCode).apply();
        }
    }
}

private boolean modelsCached() {
    String nluName = "nlu.tflite";
    File nluFile = new File(getCacheDir() + "/" + nluName);
    return nluFile.exists();
}

private void decompressModels() {
    try {
        cacheAsset("nlu.tflite");
        cacheAsset("metadata.json");
        cacheAsset("vocab.txt");
        cacheAsset("minecraft-recipe.json");
    } catch (IOException e) {
        Log.e(logTag, "Unable to cache NLU data", e);
    }
}

private void cacheAsset(String assetName) throws IOException {
    File assetFile = new File(getCacheDir() + "/" + assetName);
    InputStream inputStream = getAssets().open(assetName);
    int size = inputStream.available();
    byte[] buffer = new byte[size];
    inputStream.read(buffer);
    inputStream.close();
    FileOutputStream fos = new FileOutputStream(assetFile);
    fos.write(buffer);
    fos.close();
}

The configuration in our Spokestack class expects the files to be located in the cache directory (and named as listed above). If you change the files’ locations, you’ll need to update the file paths in Spokestack.java as well.

Other than that, though, you’re all set with NLU. Doing the actual utterance classification happens each time an ASR transcript is available, via the following code in Spokestack. Note that it’s an overridden method; Spokestack implements the Spokestack library’s OnSpeechEventListener interface to receive events from the SpeechPipeline.

@Override
public void onEvent(SpeechContext.Event event, SpeechContext context) throws Exception {
    switch (event) {
        // other events
        case RECOGNIZE:
            // the RECOGNIZE event signifies that a result is available from the selected ASR.
            String utterance = context.getTranscript();
            NLUResult result = this.nlu.classify(utterance).get();
            Response response = this.dialogManager.handleIntent(result);
            speak(response);
            break;
    }
}

Notice what comes after the NLU does its thing. That’s our next step: putting another small piece of Amazon’s backend right in the app.

Recreating Dialog Management

A dialog manager takes the intent and slots from the NLU along with the current context (or “state”) of the conversation and decides what sort of response the system should give. In the Alexa SDK, this means that the system consults, in order, a series of RequestHandlers that have been registered to a SkillStreamHandler.

In Java terms, this means you have a series of classes that all implement an interface for handling user intents, and a manager class somewhere that has an ordered list of these handlers. When a request comes in, the first handler capable of responding to the request is chosen. That’s all easy enough to replicate without the help of Lambda or the SDK, so let’s do it.

First, the handler interface. We’ll stick closely to the names from the Alexa SDK for sake of analogy even though our objects will be slightly different.

public interface RequestHandler {

    public boolean canHandle(HandlerInput handlerInput);
    public Response handle(HandlerInput handlerInput);
}

We’re not returning an Optional from handle like Amazon does just to maintain compatibility with a wider range of Android devices (it wasn’t introduced until API 24).

And now the manager itself. It’s not much to look at either.

public class DialogManager {

    private List<? extends RequestHandler> requestHandlers;
    private Session session;

    public DialogManager(List<? extends RequestHandler> handlers,
                         Cookbook cookbook) {
        super();
        this.session = new Session(cookbook);
        this.requestHandlers = new ArrayList<>(handlers);
    }

    public Response handleIntent(NLUResult nluResult) {
        HandlerInput handlerInput = new HandlerInput(nluResult, this.session);
        Response response = null;
        for (RequestHandler handler : this.requestHandlers) {
            if (handler.canHandle(handlerInput)) {
                response = handler.handle(handlerInput);
                break;
            }
        }
        updateSession(nluResult, response);
        return response;
    }
}

The main addition here is the Cookbook, which we use to look up Minecraft recipes. It’s related to the conversation state, so we’re putting it right in the Session, but if we had a more complex response generation system, it would probably belong there instead. In the Amazon SDK, the Session object is nested inside a RequestEnvelope, but we don’t have a need for most of the other things their SDK exposes via objects like RequestEnvelope and AttributesManager, so we’ve flattened out the API.

That’s it for the guts of the dialog management system. Setup is done, once again, in the Spokestack class:

private void buildDialogManager() {
    String cacheDir = this.appContext.getCacheDir().getAbsolutePath();

    List<? extends RequestHandler> handlers = Arrays.asList(
          new LaunchHandler(),
          new RecipeHandler(cacheDir),
          new HelpHandler(),
          new RepeatHandler(),
          new ExitHandler(),
          new ErrorHandler()
    );
    this.dialogManager = new DialogManager(handlers);
}

You can tell what most of those handlers do from their names; their rather simple code is available in the demo project. Remember that this is a Minecraft skill whose main job is to look up “recipes” for different in-game items. Hence, most of the interesting work is done in RecipeHandler. A discussion of its business logic is outside the scope of this tutorial, but do take a look through the source code. It’s heavily commented to offer some tips for voice search, an important consideration for many apps.

We’ve now dealt with the input side of the equation, so all that’s left is to make the app talk back. It’s not an easy task, but Spokestack makes the implementation simple.

Speaking our mind

Before the app speaks, we have to figure out what it should say. This example skill has a simple call-and-response-style conversation, so there aren’t many different states the conversation can be in. An app designed to carry on a longer conversation will probably want a different structure, but we can get away with putting our prompts in a simple enum and selecting from it based on intent at runtime:

public enum Responses {

    WELCOME("Welcome to %s. You can ask a question like, what's the recipe for a %s? ... Now, what can I help you with?"),
    ERROR("Sorry, I can't understand the command. Please say it again."),

    // other responses...

    private String prompt;

    Responses(String prompt) {
        this.prompt = prompt;
    }

    public String formatPrompt(String ... params) {
        return String.format(this.prompt, params);
    }
}

Notice that the responses are templates, allowing us to inject data at runtime. Again, we’re taking advantage of our app’s simplicity; you might want a more robust templating solution than plain String.format() for yours. We’ve also gotten rid of Alexa-style reprompts, which is the name for an additional prompt that can be delivered if the app asks a question but doesn’t receive an answer for a pre-set amount of time. There’s nothing stopping you from implementing reprompts in Spokestack; it’s just once again outside the scope of this guide. We have a separate tutorial with more information if you’re interested in including reprompts.

The Responses enum covers looking up and formatting our responses; all that’s left is to turn them into audio and play them to the user. Unfortunately, dealing with media players on mobile platforms isn’t very straightforward. If your app already deals with audio playback, you may wish to approach TTS differently, but Spokestack can handle both generation and playback. This is how the TTS component is set up in our trusty Spokestack class:

private void buildTTS() throws Exception {
    if (this.tts == null) {
        this.tts = new TTSManager.Builder()
              .setTTSServiceClass("io.spokestack.spokestack.tts.SpokestackTTSService")
              .setOutputClass("io.spokestack.spokestack.tts.SpokestackTTSOutput")
              .setProperty("spokestack-id", "YOUR-ID-HERE")
              .setProperty("spokestack-secret", "YOUR-SECRET-HERE")
              .addTTSListener(this)
              .setAndroidContext(this.appContext)
              .build();
    }
}

It’s the SpokestackTTSOutput class that’s responsible for playback; if you want to manage that yourself, you’ll want to configure a TTSListener to receive and play the audio URLs that are returned by the TTS service. Here, Spokestack is set up as a listener, but it only recognizes error events so they can be logged appropriately.

Note also the spokestack-id and spokestack-secret configuration properties: these credentials are available from the account section of the Spokestack website.

With the TTS component established, generating audio is as simple as turning our prompt text into a SynthesisRequest and sending it along:

private void speak(String text) {
    SynthesisRequest request = new SynthesisRequest.Builder(text).build();
    this.tts.synthesize(request);
}

Each request handler pulls back a prompt template from the Responses enum and calls formatPrompt on it to fill it with dynamic data, so by the time the prompt is in a Response object, it’s already in its final form, and all we need to do is pull it out and send it through TTS.

Some of the prompts end in questions, which imply that the microphone should be left open (and ASR activated) after the prompt plays. This is done in Spokestack by listening for the PLAYBACK_COMPLETE event from the TTS component and calling pipeline.activate() if the last response in the Session indicates expected user input.

You’re done!

With ASR, NLU, and TTS in place, you’ve effectively recreated the Alexa experience…without Alexa! On a mobile device, you’ll have much more freedom to make your touch-based UI match your voice experience, and of course you can manage user accounts your way instead of Amazon’s.

This tutorial has been a little involved, and we know we haven’t gone over every aspect of the experience with a fine-tooth comb, so if you run into any issues, don’t hesitate to reach out!

Porting the Alexa Minecraft Skill to iOS Using Spokestack

Mon, 08 Jun 2020 00:00:04 GMT

This tutorial is part of the Porting a Smart Speaker Voice App to Mobile series.

Spokestack makes it easy to convert a smart speaker voice app like a Google Action or Alexa Skill to a mobile app. In this tutorial, we’ll go through the entire process of converting a sample Alexa skill into an iOS app using SwiftUI.

We’ve provided some components and boilerplate code so you can follow along more easily. You can download the starting and final versions of the code here.

This tutorial is a direct port of the Alexa sample skill from Alexa’s GitHub repository. We stayed as true as possible to the original and didn’t try to improve on it too much at this point.

Installation and Setup

The easiest way to install Spokestack on iOS is using CocoaPods as described in more detail in the getting started docs. Download and unzip the example code. In the “start” folder, you’ll see a Podfile with the following contents:

platform :ios, '13.0'

target 'Minecraft Skill Demo' do
    use_frameworks!
    pod 'Spokestack-iOS'
    pod 'Fuse'
end

Run the command pod install to download and install the dependencies. Next, open Minecraft Skill.xcworkspace using XCode.

Make the App Listen

If you run the app now, you’ll see it doesn’t do much. It just displays a screen with a microphone button.

Next, you’re going to start the Spokestack speech pipeline so that your app can hear you. You’re going to configure the app so that it starts listening when you press a button.

The example code contains a class PipelineStore that implements the delegates in the Spokestack speech pipeline. The init method of our ContentView already calls the pipelineStore.start() method to start background processing. To begin actually listening, add a new function to PipelineStore:

func activatePipeline() {
    print("[\(mode)] manually activating pipeline")
    self.pipeline.activate()
    DispatchQueue.main.async {
        self.isListening = true
    }
}

This new function does two things. First, it calls the activate() method on the pipeline to start actively listening for voice input. Second, it sets the variable isListening to true. Since isListening is @Published, changes to the variable will automatically be reflected in the UI.

Let’s also modify the SpeechEventListener implementation in PipelineStore so that you can update the UI with what the user said:

func didRecognize(_ result: SpeechContext) {

    print("[\(mode)] didRecognize \(result.isSpeech) and transscript \(result.transcript)")
    DispatchQueue.main.async {
        self.appSays = nil;
        self.userSays = result.transcript
    }
}

Using the ContentView class, you can modify your button so it will toggle between listening and not listening when pressed:

    Button(action:{
        if (self.pipelineStore.isListening) {
            self.pipelineStore.deactivatePipeline()
        } else {
            self.pipelineStore.activatePipeline()
        }

    }) {
    ListeningIcon(isListening: $pipelineStore.isListening)
        .background(Color.blue)
        .foregroundColor(Color.white)
        .cornerRadius(40)
    }.padding()

Try running the app now. At this point, you should be able to press the button, say something, and see what you said appear in the UI.

Make the App Understand

For this tutorial, we built an Alexa skill that allows you to ask for instructions on how to build an item in Minecraft. To do this, we needed to understand what the user is asking about and respond accordingly.

This tutorial includes Alexa’s interaction model JSON file. To make our app understand, you’ll need to import this interaction model into your Spokestack account and get an on-device NLU model back. The NLU model consists of two files. For convenience they are already included in the Resources folder of the tutorial iOS project:

nlu.tflite - a TensorFlow Lite model file
metadata.json - the model metadata

There is also a third file vocab.txt that is the BERT Wordpiece vocabulary. This is the same for every NLU model and can be downloaded here.

The sample code has a convenience class NLUService that loads these files and creates a NLUTensorflow instance. Now you can add an instance of the NLU service to PipelineStore:

    private var nluService = NLUService("nlu")

Let’s use this service to update our didRecognize() function again. The following code will pass the results from speech recognition to your NLU model for classification. For now, you should see results in your console.

func didRecognize(_ result: SpeechContext) {

    print("[\(mode)] didRecognize \(result.isSpeech) and transscript \(result.transcript)")
    DispatchQueue.main.async {
        self.appSays = nil;
        self.userSays = result.transcript
    }

    let _ = nluService.nlu!.classify(utterances: [result.transcript])
    .subscribe(on: DispatchQueue.global(qos: .userInitiated))
    .sink(receiveCompletion: { completion in
        switch completion {
        case .failure(let error):
            // respond appropriately to an error in classification
            print("nlu failure \(error)")
            break
        case .finished:
            // respond appropriately to finished classification
            print("nlu finished")
            break
        }
    }, receiveValue: { results in
        let _ = results.map({ result in
            print("nlu result")
            print(result)
        })
    })
}

Make the App Respond

The process of taking classification results and converting them into a response is the job of a dialog manager. Previously, we discussed how to create a dialog manager in Swift that mimics the Alexa SDK for Node.js syntax. The example code includes a partially complete dialog manager SkillDialogManager.

If you compare the code for the SkillDialogManager to the Alexa Node.js handler, you will see it is shockingly similar. Our dialog manager is just missing the RecipeHandler. Let’s add that to SkillDialogManager now.

struct RecipeHandler: RequestHandler {
    func canHandle(handlerInput: HandlerInput) -> Bool {
        return (handlerInput.requestEnvelope.request.type == "IntentRequest" && handlerInput.requestEnvelope.request.intent!.name == "RecipeIntent")
    }

    func handle(handlerInput: HandlerInput) -> Response {
        //return HandlerOutput(speak: "")

        let item:Slot? = handlerInput.requestEnvelope.request.intent!.slots?["Item"]

        print("got item \(item)")
        let repromptSpeech = responses["RECIPE_NOT_FOUND_REPROMPT"]!

        if (item != nil) {
            let itemValue:String = item!.value! as! String
            print("got value \(itemValue)")

            let fuse = Fuse(threshold:0.3)
            let keys = Array(recipes.keys)

            //let's do a fuzzy match to deal with an imperfect ASR
            let results = fuse.search(itemValue, in: keys)

            results.forEach { item in
                print("score: \(item.score) - \(keys[item.index])")
            }

            var recipe:String? = nil;
            if (results.count > 0) {
                recipe = recipes[keys[results[0].index]]
            }

            guard let recipeContent = recipe else {
                let speak = String(format: responses["RECIPE_NOT_FOUND_WITH_ITEM_NAME"]!,itemValue)
                return handlerInput.responseBuilder.speak(speak + repromptSpeech).reprompt(repromptSpeech).getResponse()
            }

            return handlerInput.responseBuilder.speak(recipeContent).getResponse()
        } else {
            return handlerInput.responseBuilder.speak(responses["RECIPE_NOT_FOUND_WITHOUT_ITEM_NAME"]! + repromptSpeech).reprompt(repromptSpeech).getResponse()
        }
    }
}

RecipeHandler is set up to handle RecipeIntent. The code looks to see if there is an “item” slot type that holds the name of the item we want to return a recipe for. The Alexa skill fulfillment logic is very straightforward. It just looks up the item name from a JSON dictionary and returns the recipe that matches. We can do the same in Swift. In fact, we copied the JSON file straight into XCode as recipe.json.

However, we improved this experience. ASR is not perfect and in some cases it really can’t be. For example, if I ask for a “recipe for red dye,” ASR might register that as “recipe for red die.” While “red dye” is in the dictionary, “red die” is not. There are a few strategies for resolving this that are beyond the scope of this tutorial, but a simple one is to implement a fuzzy match. Our RecipeHandler uses the Swift Fuse library to find the best similar match instead of an exact match.

The next step is to register the new RecipeHandler with the dialog manager:

class SkillDialogManager {
    var session:[String:Any] = [:]
    var handlers:[RequestHandler] = [LaunchRequestHandler(),RecipeHandler(),HelpHandler(),RepeatHandler(),ExitHandler(),ErrorHandler()]

...
}

Lastly, update didRecognize() again to run NLU results through your dialog manager:

let _ = results.map({ result in
    print("nlu result")
    print(result)
    self.currentResponse = try! self.dialogManager.turn(type: "IntentRequest", nluResult: result)
    print(self.currentResponse.speak)
    DispatchQueue.main.async {
        self.appSays = self.currentResponse.speak
    }
})

At this point your app should be able to understand and answer questions.

Make the App Speak

Up to this point, your app has recognized user speech, interpreted intent using an on-device NLU, and handled dialog just like the Alexa skill. Now, you want to make your app respond.

To add TTS capabilities, add this to your PipelineStore:

lazy private var tts: TextToSpeech = {
    return TextToSpeech(self, configuration: SpeechConfiguration())
}()

func speak(_ request:Response) {
    DispatchQueue.main.async {
        self.appSays = request.speak
        self.isSpeaking = true
    }
    self.currentResponse = request
    self.tts.speak(TextToSpeechInput(request.speak))
}

Then, make a final modification to didRecognize():

let _ = results.map({ result in
    print("nlu result")
    print(result)
    self.currentResponse = try! self.dialogManager.turn(type: "IntentRequest", nluResult: result)
    print(self.currentResponse.speak)
    DispatchQueue.main.async {
        self.appSays = self.currentResponse.speak
        self.isSpeaking = true
    }
    self.tts.speak(TextToSpeechInput(self.currentResponse.speak))

})

These changes make the app respond with recipe instructions and update the UI accordingly while the app is speaking.

To make the app behave more like a smart speaker skill, you can also handle a LaunchRequest when the app starts. The app will call the dialog manager to get intro text and read it to user when the app launches.

To achieve this, modify ContentView as follows:

var body: some View {
    VStack {
        ...
        if (!pipelineStore.isSpeaking ) {
            Button(action:{
                if (self.pipelineStore.isListening) {
                    self.pipelineStore.deactivatePipeline()
                } else {
                    self.pipelineStore.activatePipeline()
                }

            }) {
            ListeningIcon(isListening: $pipelineStore.isListening)
                .background(Color.blue)
                .foregroundColor(Color.white)
                .cornerRadius(40)
            }.padding()
        }
    }.onAppear{
            let speak = try! self.dialogManager.turn(type: "LaunchRequest")
            self.pipelineStore.speak(speak)
    }

}

The onAppear event handler fires when the view first appears. The !pipelineStore.isSpeaking logic hides the mic button while the app is speaking.

Handling Reprompts

If the user doesn’t say anything while the app is listening, we should reprompt. Again in SpeechPipeline modify the didTimeout() function.

func didTimeout() {
    print("[\(mode)] didTimeout")
    currentResponse = try! self.dialogManager.turn(type: "TimeoutRequest")
    print(currentResponse.speak)
    DispatchQueue.main.async {
        self.userSays = nil
        self.appSays = self.currentResponse.speak
        self.isSpeaking = true
    }
    self.tts.speak(TextToSpeechInput(currentResponse.speak))
}

This implementation is not very clever and will continue reprompting forever or until the user taps the mic button to deactivate the pipeline. A smarter implementation might only reprompt three times before giving up.

Wrapping Up

This tutorial gives you a blueprint for converting any Alexa skill to a mobile app. Of course, mobile apps also have a screen. This only scratches the surface of what you could do to customize a skill. You can also display images, videos, or complex visual user interfaces as part of a response to a voice request.

If you get stuck at any point during this tutorial, please reach out to us on the Spokestack community forum. And if you build something cool, please let us know!

Porting a Smart Speaker Voice App - Part 3

Mon, 08 Jun 2020 00:00:02 GMT

Import an Alexa or Dialogflow Interaction Model

In Part 1 and Part 2 of this series we covered what smart speaker voice apps are and how you can port them to mobile using Spokestack. This post describes how you can easily bring your voice app’s natural language understanding over to a smartphone.

Voice apps running on smart speakers use the platform’s natural language understanding (NLU) to infer intent from a user’s utterance. When moving an Alexa Skill or a Google Action to mobile, you’ll need a different way to perform NLU. Fortunately, Spokestack provides custom NLU models that run on device using TensorFlow Lite’s interpreter. Even better, you can export your existing Alexa or Dialogflow interaction model and let our system build a custom on device model for you.

NLU is a critical component of modern voice user interfaces (VUI). This process takes what the user says, or utterance, and classifies it as something the voice application can understand. For smart speaker voice apps, this classification results in an intent with optional slots. For example, the utterance “Turn off the hallway lights” might result in an intent called TOGGLE_LIGHTS with slots for mode:off and location:hallway. A similar utterance “Turn on the bedroom lights” could also give the intent TOGGLE_LIGHTS, but with slots mode:on and location:bedroom.

The exact name of an intent is not important except as a mnemonic for the programmer. The same goes for the slot names and values. What this is doing is scoping the huge domain of natural language down to a few actionable intents with predefined possible values. As a voice interface designer, it’s your job to define which intents the interface can respond to just like a GUI designer chooses which visual elements a user can interact with. The voice app can choose to ignore any utterance that doesn’t match with a known intent or prompt for further clarification. For more technical information on Spokestack’s NLU system, take a look at our NLU concepts guide in the documentation.

If you’ve built a Google Action or Alexa Skill, you’ve already defined a natural language interaction model using Dialogflow or Alexa Skills Kit. Behind the scenes, these services generate an NLU model for you and make it available to your skill using their cloud services. Spokestack can also generate an NLU model for you, except you get to run it in your app!

I want to pause to point out that you don’t have to use the Spokestack on-device NLU for your embedded assistant. Spokestack’s speech pipeline provides you with text from the ASR service. You could choose to classify this text using a hosted NLU service’s API like Google’s Dialogflow or Amazon Lex. However, running on device provides some advantages. First, it eliminates a network round trip that could result in faster performance depending on connection speeds. Second, it allows NLU to work even offline or with a poor signal. Finally, on-device NLU enhances privacy for your users by not sharing transcripts of their voice interactions with third party services.

To get started with a free NLU model that’s compatible with Spokestack, create a free account and upload your existing Alexa or Dialogflow interaction model. The following instructions for that process are paraphrased from our export guide:

Log into the Amazon developer console, find your skill, click the “Build” tab at the top, and look for “JSON Editor” listed under your intents and slots on the left side (at the time of this writing).
Copy the entire contents of the JSON editor and paste them into a new file on your computer. Save it as <YOUR-MODEL-NAME-HERE>.json.
Log in to your Spokestack account, click on “Language Understanding” in the navigation, and upload your JSON file using the “Import” button.
Watch the email address you used to sign up for the account; we’ll email you when your files are ready, and you can download them from your account page.

Next Steps: Tutorials

Spokestack has libraries for iOS, Android, and React Native.

Tutorial: Create an Alexa-Compatible Dialog Manager in Swift
Tutorial: Porting the Alexa Minecraft Skill to iOS Using Spokestack
Tutorial: Porting the Alexa Minecraft Skill to Android Using Spokestack (Coming Soon)
Tutorial: Porting the Alexa Minecraft Skill to React Native Using Spokestack (Coming Soon)

Porting a Smart Speaker Voice App - Part 2

Mon, 08 Jun 2020 00:00:01 GMT

Voice Apps on Mobile

In Part 1 of this series we covered covered what smart speaker voice apps are and some of their limitation. Part 2 describes what we need to do to port a smart speaker voice app to a smartphone.

If you’ve authored an Alexa Skill or Google Action, you’ve already mixed the ingredients needed to produce a voice app. The smart speaker hardware and virtual assistant software provide the ASR, NLU, and TTS. You create the fulfillment and dialog management with the help of the platform SDKs.

To move your smart speaker skill or action to a mobile device, you need to mix the same 5 ingredients we introduced in Part 1.

Without Spokestack

Modern smartphones have integrated microphones and speakers you can access. Both Android and iOS devices allow app to access audio input and output using their development kits.
Android provides programmatic access to Google’s ASR. iOS apps have access to Apple’s ASR.
You can write a dialog manager to handle conversation turns in your app.
You can train an NLU model using a cloud service like Amazon Lex or Google’s Dialogflow.
You can integrate a service like Amazon Polly or Google Cloud’s Speech-to-Text.

If you’re highly competent with native Android and iOS SDKs as well as with integrating third-party web services, you could write code to mix four of the five ingredients necessary for a mobile voice app. But, it’s not easy. We’ve done it. This experience is what drove us to create Spokestack. Our libraries for iOS, Android, and React Native consolidate mobile ASR, NLU, and TTS into a single, easy-to-use API.

With Spokestack

✅ Spokestack managed hardware control
✅ Spokestack managed ASR
Custom Dialog Mananger
✅ Import Alexa or Dialogflow conversation model for Spokestack NLU
✅ Spokestack TTS

The Spokestack libaries simply make it easier for you to focus on the custom portions of your voice assistant experience like the conversation flow and the content. You have to worry much less about platform-specific implementation details.

Part 3 of this series will cover how to import an interaction model from Alexa or Dialogflow to create an on-device Spokestack NLU model. Following that we have a tutorial on how to write a dialog manager that mimics how your Alexa Skill or Google Action works on a smart speaker. Finally, we’ll go through some complete tutorials on how to add Spokestack to a mobile app on various platforms to recreate an Alexa Skill as an embedded assistant.

Porting a Smart Speaker Voice App to Mobile - Part 1

Mon, 08 Jun 2020 00:00:00 GMT

Voice Apps on Smart Speakers

Over the last several years, intelligent virtual assistants such as Siri, Alexa, and Google Assistant have become increasingly more important in our everyday lives. Smart speakers like Amazon Echo and Google Home brought convenient, hands-free access to assistants into our homes. One way independent developers have improved these assistants is by creating add-ons collectively known as voice apps. Users access these voice apps through virtual assistant platforms on smart speakers.

Spokestack introduces a new way to build and access voice apps independent from major virtual assistant platforms. These articles cover what you need to move a smart speaker voice app onto a mobile device.

Part 1: Voice Apps on Smart Speakers
Part 2: Voice Apps on Mobile
Part 3: Import an Alexa or Dialogflow Interaction Model
Tutorial: Create an Alexa-Compatible Dialog Manager in Swift
Tutorial: Porting the Alexa Minecraft Skill to iOS Using Spokestack
Tutorial: Porting the Alexa Minecraft Skill to Android Using Spokestack
Tutorial: Porting the Alexa Minecraft Skill to React Native Using Spokestack (Coming Soon)

What’s Wrong With Smart Speaker Voice Apps?

Voice apps like Alexa Skills and Google Actions have been an exciting new frontier for developers over the last few years. Voice apps let virtual assistants delegate some interactions to third-party applications running on the platform. Voice apps are to smart speakers what mobile apps are to smartphones.

Unfortunately, consumer adoption of voice apps has not been as successful as adoption of mobile apps. Discovery and retention has been a long-standing challenge. Monetization remains elusive for most skill developers. As a result, 2019 saw the lowest number of new Alexa skills released since 2016.

This doesn’t mean consumers are moving away from voice as an interface. Voice assistant usage is on track to triple in the next few years. Consumers have embraced voice as an interface. However, the model for third-party skills and actions has not been a success. Fortunately, there’s a way to free these voice apps and promote them as independent voice assistants embedded in smartphone apps.

Transform Your Skill to a Mobile Voice Assistant

Moving voice apps off of smart speakers and into an app on a smartphone resets the conversation. You gain full control of your app’s branding and visual UI. This gives you the flexibility to integrate images, videos, and traditional GUI components as you see fit. You’re free from the commerce limitations Amazon and Google place on smart speaker voice apps.

Smart speakers and virtual assistants provide key components that are used to build voice apps. In order to move a voice app to a smartphone, you have to make sure you still have all the ingredients. Let’s take a look at the ingredients needed to make a general voice app, regardless of platform.

Universal Voice App Ingredients

A device with a microphone and speaker
A way to recognize human speech using the device’s microphone and a way to convert that speech to text
A dialog management system that takes the speech transcript and formulates a text response
A way to infer computer-readable meaning from the speech transcript
A way to convert the text back to speech and play it through the device’s speaker

If you are coming from a background in voice apps, chat bots, or virtual assistants, these ingredients will be very familiar to you. For those who aren’t, some of these topics are worth a quick review.

Speech Recognition & Text-to-Speech

Converting speech to text is often called automated speech recognition (ASR). Converting text back to speech is known as text-to-speech (TTS). Both of these are memory-intensive operations usually performed using a cloud service. Smart speakers have built-in access to cloud services to perform both ASR and TTS.

Dialog Management System

A dialog management system takes the text transcript as input and formulates a response. This is known as a dialog turn. Developers are responsible for providing content for the responses in a process called fulfillment. Some dialog management systems will automatically perform tasks such as slot filling or re-prompting for misunderstood utterances.

Natural Language Understanding

In order to write logic for turns, developers need a way to infer meaning from text. This is known as natural language understanding (NLU). The state-of-the-art way of performing NLU is using deep learning models to infer “intent” from “utterances.” Both Amazon and Google have cloud-based services where a developer can train an NLU model by giving example utterances for various intents. The developer decides which types of intents a voice app should respond to. Then he or she provides example phrases for each. Google and Amazon make their resulting NLU services available for voice apps to use.

Voice Apps on Smart Speakers

If you’ve authored an Alexa Skill or Google Action, you’ve already mixed the ingredients needed to produce a voice app. The smart speaker hardware provides the microphone and speaker. Each smart speaker is integrated behind the scenes with a cloud service to perform ASR, NLU, and TTS. Amazon and Google provide SDKs to create a fulfillment web service that includes dialog management.

In this introduction we covered what smart speaker voice apps are and some of their limitation. We also covered what components are needed to build one and how the smart speaker hardware provides them. In Part 2 of this series we explore how to find these ingredients on a different device, a mobile phone.

Test Your Voice Assistant with Real People

Fri, 05 Jun 2020 00:00:00 GMT

Setting aside time to test your voice assistant(s) with actual users is a worthwhile investment in usability. At Spokestack, I’ve conducted tests using prototypes and fully-realized voice user interfaces (VUI). How I test products is similar in execution to user interviews with a few exceptions. Your goal should be to gauge whether users are able to complete tasks using their voice. Refer to my previous article on how to find candidates within your network. What will change is what you ask and look for. Here’s how I test voice assistants throughout the development process.

Come prepared with research questions

Write a test script to guide usability testing that includes instructions, tasks, and questions. Here’s a sample test script I adopted from Steve Krug’s Rocket Surgery Made Easy that incorporates my suggestions. I start every test script with a list of research questions. Think of these as internal measurement tools for your observations. These will include any uncertainties you want to put to rest without directly asking users these questions. Here are some examples.

Do users know what to ask or say?
Are they running into any errors?
(If applicable) Do they understand that your app is wake word enabled? Did they use this feature? If not, why?

Keep context in mind when setting up your test environment

This is where survey demographics will come into play. Decide where you want candidates to test usability. For example, if your voice assistant will be used primarily outside where there’s lots of background noise, make sure you test outside. If you’re unable to replicate the environment, start each task by framing context. For example, if your target demographic uses voice commands frequently while driving, start a task with something like “Imagine you’re commuting to work.” Remind them of context as it changes. With the previous example, if they’re supposed to complete a task while they’re no longer commuting, include something like “You’ve reached your destination and are out of the car.”

Write tasks focused on usability, not opinions

Refer back to your research questions for these. You’ll want to test features that are likely going to be most used as well as features you’re the most unsure about. Here are my tips for writing tasks that effectively test voice assistants:

Remind subjects to share what they’re doing and thinking out loud as much as possible.
Ask users for their first impression after they’ve completed their first task. Encourage constructive feedback when warranted and avoid asking if they “like” or “dislike” something.
If you included the last research question from above in your script, don’t explicitly tell users your app is voice-enabled. You want to understand whether or not they can determine this on their own. Avoid including trigger or wake words in your tasks. If a user mistakenly activates their device, include phrasing for how to stop an interaction and continue.
Don’t use technical terms.
Include a few negative questions to reframe their thinking.

Moderate some, but mostly listen & observe

If you see a user struggle to complete a task, try not to interrupt them unless they’re unable to continue. Ask follow-up questions directly after a related task has been completed. Otherwise, subjects are likely to forget any insights they had in the moment. Try to answer these by observing rather than explicitly asking them. Their actions will speak volumes. Here are some things to be on the lookout for:

Do they know your app is voice-enabled?
Do they understand what they can say? What they can access using gestures?
Do they understand how to use voice commands moving forward?
Are they asking the right questions? If not, what are they asking?
Are they ever at a loss for words?
Do they default to touch instead of voice? If so, how often? It could be out of habit or personal preference. It could also be a sign that there were too many speech recognition or understanding errors.
If they use the wake word more than once, what is their reaction after repeated usage? Did they find it useful or cumbersome to use?
Do they understand when the app is listening to them?
If your app supports voice output, how are they reacting to different responses? Are you providing too much information or just enough for them to move forward?
What is their reaction to your synthetic voice and any sounds used? Are they too robotic? Too personable? If their reaction is negative, what would they prefer?

Incorporate what you observed into your product road map

Your colleagues will have likely written brief notes during the test. After testing, I go back and watch recordings to document takeaways and discuss with fellow observers by doing the following:

Write down insights from your observations without providing a solution. Include direct quotes that stood out to you after listening to your discussion a second time.
Discuss and group overlapping ideas with your team.
Write a brief problem statement that accurately captures each group. Discuss and document possible solutions. This could include things like bug fixes, tweaks to existing features, new features, etc.
Look back at your solutions list and let your observations guide the level of priority for each. Incorporate these into your existing product road map.

Continue to iterate and test throughout your product’s evolution. For more help designing independent voice assistants, visit the design section of our documentation.

What to Include in a Survey

Fri, 29 May 2020 00:00:00 GMT

Recruiting folks to test your Voice User Interface (VUI) or give insights on a concept is no easy task. Finding candidates in your target demographic takes time and prowess. You need to know where your users are in order to reach them in the first place. Relying on your network is a great way to quickly get feedback. However, even with old acquaintances, I’ve found myself (maybe not wanting to, but) needing a tool to capture basic demographic information. For this, I recommend using a survey.

Really…You Want Me to Use a Survey?

The value of surveys has come into question in the greater design community, and with good reason. I urge you to take your results with a grain of salt. For one thing, they can be inherently biased. Use them as a starting point rather than a means to an end. These results are not definitive nor are they statistically accurate.

Leverage them to help you gain some perspective, but don’t let them dictate the whole picture. For example, in one study, I wanted to interview people who used voice assistants while driving. One user said in the survey he used Google Assistant on Android daily and spent a decent amount of time commuting in his car. However, during our interview, he admitted he wouldn’t use voice commands while he had a sleeping baby in the back seat which was most of his time spent in the car. The survey results provided a good starting point for our discussion that lead me to discover a new demographic I hadn’t considered.

Ask the Right Questions Up Front & Save Time Later

Start by your letting subjects know what to expect. Be transparent about the length. Use a mix of quantitative and qualitative research questions Limit your questions to no more than ten to increase engagement. In general, stick with multiple choice or questions that can be answered with a short response.

Decide what information is most critical to your research. Understand how your audience already uses voice in their daily routines. For example, when testing a voice-enabled mobile audio service with users, I surveyed fifty or so candidates from my community. I wanted to test with a mix of voice assistant power users and people who didn’t use voice assistants at all. I found this was a good way to understand and, as a result, better design for both perspectives. I also asked a mix of questions about their listening habits - what they usually listen to, how, etc. Knowing these answers helped me better prepare my conversation, saving time for both me and my subjects.

Here’s an example of some survey questions used when recruiting folks to usability test Radiobrain.

Limit open-ended feedback. You’ll be able to further probe for detail when conducting user interviews. Draft your survey and get feedback from those you trust. Here’s a sample survey I created to get you started.

For more help designing independent voice assistants, visit the design section of our design documentation.

TTS, Outside the (Black) Box

Wed, 27 May 2020 00:00:00 GMT

At Spokestack, we process a lot of, well, speech. Our libraries help you both listen to and talk to your users. Both parts are tricky, requiring sophisticated machine learning models and lots of data. On the text-to-speech side, we’re constantly working to improve our models to give our voices higher fidelity and smoother prosody, and do it all even faster.

It’s easy to focus on that sort of improvement exclusively and forget that there’s more to reading text than a left-to-right (or right-to-left) translation of characters into sounds. I recently posted on Medium about some of the edge cases we’ve encountered while building our system. There are quite a few ways to trip over yourself, so if you’re interested in responding to your users naturally — with a voice unique to your brand — get in touch!

Tools and Tips for Mapping Multimodal Products Remotely

Tue, 19 May 2020 00:00:00 GMT

At Spokestack, we use experience mapping as a tool to formulate multimodal flows for our products based on researched observations and ideas. It’s a great way to get everyone on the same page before designing or building anything. Plus, it forces us to consider what exactly we want users to ask and how we’ll respond, both conversationally and visually. When our team committed to being 100% distributed over two years ago, I wanted to recreate this exercise digitally. What I came up with follows many of the same steps outlined in our design docs. However, the tools and set-up are a bit different. Here are a few considerations to keep in mind when adopting this exercise.

Tools We Use

I use Zoom to screen share and record conversations with my team. Meeting recordings are a good reference point if you or a teammate needs to be reminded of how decisions were made and why. You’ll still need to designate someone to lead and record the map. That person will need to screen share documentation while mapping and record the meeting. For our team, I’m usually the designated leader. I use one of two tools (both free) for mapping:

Stickies.io

Stickies.io does a great job mimicking the tactile nature of mapping with post-it notes (and, without the clean up). Create a new “board” for each project. If your project requires more than one experience map (i.e. if you’re mapping for new and return users), you’ll need to create a new sheet for each in the top nav. Use stickies here like you would post-it notes in a conference room. List actors along the left axis. The bottom axis will represent time. Pick a starting point and list each action on a separate sticky in the same color as the corresponding actor.

Here's an example of an experience map for Radiobrain created using Stickies.io

Shared Spreadsheets

These are equally great if not more so for complex maps that capture lots of actions and actors. Create a new spreadsheet for each project. Use tabs to keep track of multiple experience maps where necessary. Use cells here like you would post-it notes same as you would above and make sure that they are color coded accordingly. Unlike stickies.io, if an action is reoccurring across the x axis, you won’t need to repeat that cell.

Here's an example of the same experience map from Stickies.io recreated using Google Sheets.

Not sure where to start? Here’s a template using Google Sheets to get you started. Make sure to duplicate this document before making any changes.

For more tips on designing multimodal voice experiences, visit the design section of our documentation.

User Research for Voice Experiences

Fri, 15 May 2020 00:00:00 GMT

Over the last three years, I’ve worked on designing voice and chat conversations paired with mobile UIs including The Bartender and Tasted. One thing has remained clear: design for verbal input is a very different UX problem. The number of actions a user can take is not constrained by buttons and navigational menus. That’s why it’s important to start every project by identifying who exactly you’re designing for. This will help you better understand how voice can improve your overall experience.

“What Can I Help You With?”

If you’ve ever said “Hey Siri” to an iOS device, you’re likely familiar with the assistant’s opening screen. “What can I help you with?” can be intimidating. You might not be sure what or how Siri can help you, if at all. Siri serves a broad audience (every iPhone user in the world) and an even broader domain of knowledge similar to a search query, which can be overwhelming. Independent Voice Assistants (IVA) serve a more limited domain of knowledge. The actions a user can take, or “intents,” focus on specific tasks in your app like buying movie tickets, recording a run, or finding a cocktail recipe.

With fewer intents, your assistant can provide a better experience without having to know how to respond if a user asks something unrelated like “How many miles to the moon?” But, what intents do your users care about? To answer that question, you need to start by understanding who you’re building for. Even if you already have an idea of who your target audience is, adding a voice assistant to your mobile app may shift your product’s reach.

Identify How Users Talk About What’s Important to Them

Take a step back at start of a project and consider what’s most important to your intended audience. What are they trying to accomplish? How can voice make tasks easier? This will help you decide how a voice assistant can improve your app’s experience.

If you don’t know the answers to these questions, it’s ok. I’m going to help you seek these out before diving into product specifics by gathering anecdotal insights. And, if you’re starting a new project that hasn’t been built yet, that’s ok too. This process can be adopted to gain insights for concepts, prototypes, and/or real-world examples. It’s important you start this discussion early in the process and keep it ongoing.

Above are some questions I brainstormed with my team for an initial proof of concept. We wanted to better understand the pain points of managing AirBnBs.

Recruit from Your Network

Relying on your network can be a great way to quickly get initial feedback. Use a survey to gather demographic information on candidates, even old friends. Here are some tips that have helped me during recruitment:

Allow yourself at least a week to schedule one-on-one time with each person. Following Jakob Nielsen’s advice, I’ve found success limiting this to around five people.
Share a calendar link. I’ve found Calendly and Google Calendar to work well for this. Be sensitive of their time and schedule. Make it easy for them to talk to you.
Offer your subjects an incentive if you’re able. We often give our participants gift cards for example.

How to Guide the Discussion

This will be a guided discussion that you will take charge of. It’s important to come prepared. Here are a three things I make sure to do prior to an interview:

Create a document for each person you interview that includes background information from an email correspondence or survey results. Leave room to jot down questions during your interview. Getting feedback from your users usually happens fast during an interview or usability test. You’ll have time to take more in-depth notes later. Here’s a template I usually use for this.
Write a script that provides a framework for your conversation, but improvise as you see fit (see the next section). Include instructions and questions. Here’s a sample script I adopted from Steve Krug’s Rocket Surgery Made Easy that helps me stay on task during interviews.
It will feel silly, but it really helps to rehearse your script by doing a dry run. Have someone pretend they’re the user and read the script aloud to them. This will help iron out any kinks that may remain (and ease any pre-interview jitters).

Don’t Forget Who Your Real Users Are (Psst, It’s Not You!)

You may be more savvy about voice assistants or tech in general than your audience. It’s hard to step back and really hear what users are saying when you’re eager to start designing or building a product. Ask interviewees how they would normally complete certain tasks. Observe their language including nouns, verbs, and sentiment. Your users’ natural speech will inform which utterances and intents your assistant supports and how you’ll prompt these. Be on the look out for any contextual clues that might reveal how or where they’re completing these tasks. Voice interfaces allow certain tasks to be completed while multitasking. Use this to your advantage.

Here’s an example from a user interview I did for a virtual host concept for AirBnb hosts. An owner I spoke with had a complicated coffee maker in one of his units. He had made a binder full of useful tips for his guests that explained how to use his appliances. However, housekeeping often misplaced the binder making it hard for his guests to find. A simple “how does the coffee maker work?” to a voice assistant followed by step-by-step instructions could satisfy the problems of both the owner and guest.

If candidates feel comfortable, stream and record the discussion (we use Zoom). This will allow your team to ask questions in real time.

Post-Interview

After the interview, I like to use recording to document takeaways by doing the following:

Write down observations and transcribe any statements that stood out to you after listening to your discussion a second time.
Discuss and group reoccurring insights with your team.
Label each group to reflect a product feature or intent.
Let your observations guide the level of priority for each. Which intents will most ease the user’s experience? This will dictate focus when you start to map your multimodal flow.

After sharing my observations from Airbnb hosts with the team, here are some trends we identified.

Don’t forget to re-visit this discussion as needed throughout the design process, especially when testing a product’s usability, to determine if what you’re building remains relevant to your intended audience. In further posts, I will show you how to take this research and turn it into a great voice user experience.

For more help designing independent voice assistants, visit the design section of our documentation.

What Is an Independent Voice Assistant?

Tue, 12 May 2020 00:00:00 GMT

“A voice assistant is a digital assistant that uses voice recognition, speech synthesis, and natural language processing (NLP) to provide a service through a particular application.”

If you search ”what is a voice assistant?” on Google, the definition above appears as the first result. This definition works because it lists the technologies required to provide human-to-computer voice interactions, a.k.a. a “voice interface”. Voice recognition, speech synthesis, and natural language processing technologies together create a software stack that enables voice assistants to operate (a “Spokestack” you might say).

So does providing a voice interface to a company’s digital services make it a voice assistant?

The answer is yes. Now let’s discuss the difference between a smart speaker (e.g. Alexa) smart speaker apps (e.g. The Jeopardy Skill), and independent voice assistants.

An independent voice assistant, sometimes referred to as an “owned assistant,” is a type of voice interface that enables companies to directly communicate with their customers. With independent assistants, there are no intermediaries (e.g. Alexa, Siri, Google Assistant) standing between companies and their customers. The conversation is heard, understood, and responded to directly by the company. This provides multiple benefits for both the consumer and the company.

Mutual Benefits for Customers and Companies

Building an independent voice assistant offers more control over the customer experience, thereby creating the experience a company wants their customer to have with their brand. Benefits include:

Easier access to customer account and payment information.
A voice that best represents their brand over the platform’s voice (e.g. Siri’s voice)
Data on intents and experiences that the company can use to inform product and service decisions.

How Are Independent Voice Assistants Used?

Voice assistants can be integrated with existing mobile apps and/or websites. These can entail something as simple as offering voice search using a touch-to-talk mic button. They can also be more dynamic experiences that involve things like wake words to activate listening and content that is broken out into conversational pieces like step-by-step instructions.

Independent voice assistants can also co-exist with third party voice platforms (e.g. Siri, Google, Alexa). Currently, iOS and Android prefer using deep links (Siri Shortcuts for iOS; App Actions for Android) to interface with Siri and Google Assistant. Deep links require developers to link to parts of their application based on what the user says to Siri and Google Assistant. Developers who add voice to their app can have deep links resolve to a listening screen that asks the user “How may I help you?” and take over the conversation.

There is another scenario where companies can retain more control of the conversation on smart speakers by Alexa, but that’s a bit more complicated. We’ll save that for another post.

Different Types of Voice Assistants

The difference between the types of voice assistants comes down to who controls the experience and how the voice assistant is accessed. The following is how we think about the different types of voice assistants based on control and access.

Platform Assistants handle natural language processing, speech synthesis, and a wake word that activates listening (“Hey Siri”), and are tightly integrated with hardware such as smart speakers and mobile devices. These assistants are usually general AIs that try to answer every question a user may have. They also provide marketplaces for third party “voice apps” that make platform assistants smarter.

Examples: Siri, Alexa, Google Assistant, Bixby, Mycroft, Snips (now Sonos).

Voice Apps exist within a platform assistant. They wrap content and services into an API that platform assistants can access and distribute. These apps offload natural language processing, speech synthesis and wake word control to platform assistants. They’re usually described in relation to the platforms where they operate, such as Alexa Skills, Google Actions, Bixby Capsules, etc.

Examples: Jeopardy! Skill, Pizza Hut Action, iTranslate Capsule

Independent Assistants are voice assistants independent of platforms meaning that customer conversations are controlled by companies and developers without help from a platform assistant. This means companies are able to control natural language processing, speech synthesis, and wake word(s). These assistants can be integrated within existing products such as mobile apps, websites, and/or proprietary hardware.

Examples: Spotify (voice search), Home Depot (powered by Google’s Dialogflow for voice search), Pandora (powered by Soundhound).

Spokestack provides technology and services to enable developers to build independent voice assistants. Please email us at hello@spokestack.io if you are interested in hearing more about how we can help you build an independent voice assistant.

Keeping Perspective

Fri, 20 Mar 2020 00:00:00 GMT

I watch this presentation by Wilson Minor at the 2011 Build Conference at least once a year. Last week, I rewatched it trying to refocus my mind from pandemic to product development work. Watching Wilson’s talk reminds me why we make things, both the impact and consequences. If you can spare 38 minutes, it’s worth your time.

Nine years ago, before we knew the full impact of mobile and social apps would have on our lives, Wilson used a quote from Marshall McLuhan and adapted by Winston Churchill to explain the future:

This quote is a reminder that the things we build impact the world. Even if it looks like a toy at first, like Chris Dixon said 10 years ago. At the beginning of a project, you may not understand all potential uses for what you are building. But, if you keep your eyes and ears open, some of those uses may find you.

At Spokestack, we build tools and services that make your app fully voice-enabled. The idea started with “wouldn’t it be cool if you could talk to apps?”. That came from frustration. If you ask Siri a question, you’ll likely hear, “this is what we found on the web”. The answers are already on our phones but locked inside our apps. Why can’t they answer our voice?

One common request when talking to customers is improved mobile accessibility. Especially for those who need more than a touch-only experience. Existing companies already build hardware for the visually impaired. There are also companies like Tecla that build devices to help those with limited upper-body mobility to use mobile devices. Voice technology has the power to increase usability for millions of people.

Startups like ours are always looking for a product-market fit, but times like these compel us to reexamine how we think of the “market”. What we make should provide for societal needs in addition to making a great business.

If you have specific problems a voice-enabled interface could solve, please reach out. Help us shape Spokestack’s tools for everyone.

Why We’re Building Spokestack

Tue, 10 Dec 2019 00:00:00 GMT

Over 58% of Americans already use their smart phone as a voice assistant - more than smart speaker, smart watch and desktop voice assistant users combined. That’s because we have them with us at all times.

Smart phones are smart because apps like yours make them so. In order for apps to become smarter, they need a voice.

Here’s an example: try saying “Hey Siri, open {your app name}” or “OK, Google, open {your app name}” depending on your phone.

Did your app say anything? Did it ask what you needed? If not, Spokestack can help. At Spokestack, it’s our mission to give every company a unique voice, one that reinforces their brand’s relationship with customers. We believe every company should have direct conversations with their customers rather than broker with intermediaries. Customers should know who they’re talking to when they ask for your brand. Voice enables a new way to interact with customers that will grow your business by making your app more useful.

Finally, we believe in open source software, best practices around privacy, and making sure you have full control over conversations with your customers. We’re at the forefront of the next great leap in user experiences. We’re excited to partner with you to build that future together.