Écriture/lecture/modification de fichiers dans un projet Universel app/UWP et cordova

Dans nos applications, nous avons pris le parti d’écrire nos fichiers en JSON. Nous avons donc conçu une abstraction (en WinJS, avec un système de promises) qui nous permet d’utiliser des méthodes claires et simples sur toutes les plateformes (WinRT/UWP ou Cordova (iOS/Android) )

Ce module « DataContainer » est disponible dans notre libraire WinJSContrib (Github)

Dans WinJSContrib.DataContainer, nous avons 4 fichiers qui exposent les mêmes méthodes avec des implémentations différentes :

  • read : lecture d’un élément
  • save : enregistrement d’un élément
  • remove : suppression d’un élément
  • list : liste des fichiers d’un container
  • child : création/accès container enfant

Dans une application WinRT/UWP nous allons naturellement utiliser la couche WinRT/UWP pour écrire/lire les fichiers. Pour cela il suffit d’inclure le fichier « winjscontrib.datacontainer.winrt.file.js »

Dans une application cordova IOS ou Android, nous avons plusieurs choix :

  • Utiliser une base de données : winjscontrib.datacontainer.cordova.database.js
  • Utiliser le système de fichiers avec le plugin file : « winjscontrib.datacontainer.cordova.file.js »
  • Utiliser le localStorage : « winjscontrib.datacontainer.localstorage.js » (utilisable en WinRT aussi)

Ensuite, il faut instancier un container parent et l’utiliser partout dans l’application.

Dans le JS des applications Windows 8/Phone il suffit d’instancier le container (dossier) parent :

MyApp.Data.container = new WinJSContrib.DataContainer.WinRTFilesContainer(undefined, { logger: WinJSContrib.Logger });

Et changer WinRTFilesContainer par notre choix (CordovaDatabaseContainer par exemple) pour l’application cordova en faisant attention de ne l’appeler qu’après l’enclenchement de l’évènement deviceready.

Et c’est tout, la magie s’opère à l’intérieur de notre librairie 🙂

Quelques exemples d’utilisation :

  • Pour lire un fichier :
  • Data.container.read("objKey").then(function (data) { }, function (error) { });
    
  • Supprimer un fichier :
  • Data.container.remove("objKey").then(function () { }, function (error) { });
    
  • Enregistrer un fichier :
  • Data.container.save("objKey",obj).then(function () { }, function (error) { });
    
  • Accès à un container enfant (sous dossier) avec la lecture d’un fichier fichier :
  • Data.container.child(folderid).read("subObjKey").then( function (subObjInFolderID) { }, function (error) { }));
    
  • Liste des fichiers dans un container :
  • Data.container.list().then(function (res) {}, function (error) { });
    
SpeechAPI OxfordProject

Exploring Microsoft Speech APIs

This article introduces the speech APIs, one of the services updated in Windows 10 and powered by the project Oxford. This article is divided into several parts:


I.Speech APIs features 

These APIs provide two major features :

  • Speech Recognition : Speech To Text (STT)

It converts spoken audio to text. The APIs can recognize audio coming from the microphone in real-time, or from an audio file.

  • Speech Synthesizer : Text To Speech (TTS)

It converts text to spoken audio when applications need to talk back to their users.

Microsoft Speech Platform

 

The Speech APIs are included in Windows 10 libraries, and also provided by the project oxford services, which require an Azure subscription.

II. Project Oxford Speech APIs 

These APIs are in a beta version, they work by sending data to Microsoft servers in the cloud, to use them we must have an Azure account. They also offer the Intent Recognition feature which can convert spoken audio to intent.

To use these APIs, follow the steps below:

  1. Using an Azure account, go to the Market Place to purchase the speechAPIs Service (which is for free 😉 ), then retrieve the primary or secondary key. Just in case you are using the Azure DreamSpark subscription, don’t be surprised if you don’t find this service. Unfortunately this type of account does not give access to Oxford Services.
  2. Download the Speech SDK of project oxford from here! if you are targeting another platform rather than Windows, have a look here you will find what you are looking for.

 

  • Speech To Text (STT):

The oxford version of Speech APIs offers two choice to make the STT:

  1. REST API
  2. Client library

When using the REST API, we only get one recognition result back at the end of the session, but in the case of a client library, we also get partial result before getting the final recognition.

Setting up speech recognition begins with the Speech Recognition Service Factory. By using this factory, we can create an object which can make a recognition request to the Speech Recognition Service. This factory can create two types of objects:

  1. A Data Recognition Client : used for speech recognition with data (for example from an audio file). The data is broken up into buffers and each buffer is sent to the Speech Recognition Service.
  2. A Microphone Recognition Client : used for speech recognition from the microphone. The microphone is turned on, and data is sent to the Speech Recognition Service.

When creating a client from the factory, it can be configured in one of two modes:

  1. In ShortPhrase mode, an utterance may only be up to 15 seconds long, As data is sent to the server, the client will receive multiple partial results and one final multiple N-best choice result.
  2. In LongDictation mode, an utterance may only be up to 2 minutes long. As data is sent to the server, the client will receive multiple partial results and multiple final results.

Also the client can be configured for one of the following several languages:

  • American English: « en-us »
  • British English: « en-gb »
  • German: « de-de »
  • Spanish: « es-es »
  • French: « fr-fr »
  • Italian: « it-it »
  • Mandarin: « zh-cn »

Now, time to code 😀 you can implement the code below in a WPF app


string stt_primaryOrSecondaryKey = ConfigurationManager.AppSettings["primaryKey"];

// We have 2 choices : LongDictation or ShortPhrase
SpeechRecognitionMode stt_recoMode = SpeechRecognitionMode.LongDictation;

// For a Speech recognition from a Microphone
MicrophoneRecognitionClient stt_micClient = SpeechRecognitionServiceFactory.CreateMicrophoneClient(stt_recoMode, "fr-fr",
stt_primaryOrSecondaryKey);

// For a Speech recognition from a data like wav file
DataRecognitionClient stt_dataClient = SpeechRecognitionServiceFactory.CreateDataClient(stt_recoMode, "fr-fr",
stt_primaryOrSecondaryKey);

Then we must subscribe some events to get the result of the recognition. The Microphone Recognition Client & Data Recognition Client have the same Events as follow:

  • OnConversationError : Event fired when a conversation error occurs
  • OnIntent : Event fired when a Speech Recognition has finished, the recognized text has
    been parsed with LUIS for intent and entities, and the structured JSON result is available.
  • OnMicrophoneStatus : Event fired when the microphone recording status has changed.
  • OnPartialResponseReceived : Event fired when a partial response is received
  • OnResponseReceived : Event fired when a response is received

Inside the events, we can do whatever we want, displaying the result in a textBox for ex. and more…


// Event handlers for speech recognition results
sst_micClient.OnResponseReceived += OnResponseReceivedHandler;
sst_micClient.OnPartialResponseReceived += OnPartialResponseReceivedHandler;
sst_micClient.OnConversationError += OnConversationErrorHandler;
sst_micClient.OnMicrophoneStatus += OnMicrophoneStatus;

// Data Client event from an audio file for ex.
sst_dataClient.OnResponseReceived += OnResponseReceivedHandler;
sst_dataClient.OnPartialResponseReceived += OnPartialResponseReceivedHandler;
sst_dataClient.OnConversationError += OnConversationErrorHandler;

Now how do we start or stop the speech recognition? It’s simple, we just need to make a method call


// Turn on the microphone and stream audio to the Speech Recognition Service
sst_micClient.StartMicAndRecognition();

// Turn off the microphone and the Speech Recognition
sst_micClient.EndMicAndRecognition();

To convert an audio file to text, it’s easy, we just need to convert the file into a byte array and send it to the server for the recognition, like shown below:


if (!string.IsNullOrWhiteSpace(filename))
using (FileStream fileStream = new FileStream(filename, FileMode.Open, FileAccess.Read))
{
int bytesRead = 0;
byte[] buffer = new byte[1024];

try
{
do
{
bytesRead = fileStream.Read(buffer, 0, buffer.Length);
// Send of audio data to cloud service.
sst_dataClient.SendAudio(buffer, bytesRead);
} while (bytesRead > 0);
}
finally
{
m_dataClient.EndAudio();
}
}

  • Text To Speech (TTS)

The TTS feature of project oxford can be used only through the REST API, and we have a complete example here.

The end-point to access the service is: https://speech.platform.bing.com/synthesize

The API uses HTTP POST to send audio back to the client. The maximum amount of audio returned for a given request will not exceed 15 seconds.

For any question about using this API, please refer to TTS through REST API documentation

III. Windows 10 Speech APIs

Windows 10 Speech APIs support all Windows 10 based devices including IoT hardware, phones, tablets, and PCs.

The Speech APIs in Windows 10 are represented under this two namespaces :

Requirement 

  1. Windows 10
  2. Visual Studio 2015
  3. Make sure that Windows Universal App Development Tools are installed in VS2015.

First of all we have to create a Windows 10 Universal application project in visual studio : New Project dialog box, click Visual C# > Windows > Windows Universal > Blank App (Windows Universal).

With Windows 10, applications don’t have the permission to use the microphone by default, so you must at first change the parameters of the universal application as follows:

Double click on the file Package.appxmanifest > Capablilites > Microphone > select the check box.

Note: The Windows 10 Speech APIs are using the languages installed in the Operating System.

  • Speech To Text (STT)

The STT feature using Windows 10 APIs works in online mode, if we want to make it available in offline mode we have to provide the necessary grammar manually.

To make this feature works we have 3 steps:

  • Create a SpeechRecognizer object,
  • Create an other object from SpeechRecognitionConstraint type and add it to the SpeechRecognizer object already created,
  • Compile the constraints.

SpeechRecognizer supports 2 types of recognition sessions:

  1. Continuous recognition sessions for prolonged audio input. A continuous session needs to be either explicitly ended or automatically times out after a configurable period of silence (default is 20 seconds).
  2. Speech recognition session for recognizing a short phrase. The session is terminated and the recognition results returned when a pause is detected by the recognizer.

Like shown in the code below


SpeechRecognizer speechRecognizer = new SpeechRecognizer();
// Here we choose a simple constraints scenario of dictation
var dictationConstraint = new SpeechRecognitionTopicConstraint(SpeechRecognitionScenario.Dictation, "dictation");
speechRecognizer.Constraints.Add(dictationConstraint);
SpeechRecognitionCompilationResult result = await speechRecognizer.CompileConstraintsAsync();

A continuous recognition session can be started by calling SpeechRecognizer.ContinuousRecognitionSession.StartAsync() method and can be stoped by calling speechRecognizer.ContinuousRecognitionSession.StopAsync(). The SpeechRecognizer.ContinuousRecognitionSession object provides two events :

  • Completed : Occurs when a continuous recognition session ends.
  • ResultGenerated : Occurs when the speech recognizer returns the result from a continuous recognition session.

We have another event tied to the speechRecognizer object, which is the HypothesisGenerated event, occurs when a recognition result fragment is returned by the speech recognizer.

The code below show how to start the recognition:


public async void StartRecognition()
{
// The recognizer can only start listening in a continuous fashion if the recognizer is urrently idle.
// This prevents an exception from occurring.
if (speechRecognizer.State == SpeechRecognizerState.Idle)
{
try
{
await speechRecognizer.ContinuousRecognitionSession.StartAsync();
}
catch (Exception ex)
{
var messageDialog = new Windows.UI.Popups.MessageDialog(ex.Message, "Exception");
await messageDialog.ShowAsync();
}
}
}

To stop the recognition :


public async void StopRecognition()
{
if (speechRecognizer.State != SpeechRecognizerState.Idle)
{
try
{
await speechRecognizer.ContinuousRecognitionSession.StopAsync();

TXB_SpeechToText.Text = dictatedTextBuilder.ToString();
}
catch (Exception exception)
{
var messageDialog = new Windows.UI.Popups.MessageDialog(exception.Message, "Exception");
await messageDialog.ShowAsync();
}
}
}

 

  • Text To Speech (TTS)

This feature is available in offline and online mode, to make it works we have to create a SpeechSynthesizer object, then we set the speech synthesizer engine (voice) and generate a stream from the speechSynthesizer.SynthesizeTextToStreamAsync method by passing the text we want to read in parameter.

To read the stream we have to use a MediaElement object, like shown in the code below:


SpeechSynthesizer speechSynthesizer = new SpeechSynthesizer();
speechSynthesizer.Voice = SpeechSynthesizer.DefaultVoice;
//Init the media element which will wrap tthe text to speech
MediaElement mediaElement = new MediaElement();
//We have to add the mediaElement to the Grid otherwise it won't work
LayoutRoot.Children.Add(mediaElement);

var stream = await speechSynthesizer.SynthesizeTextToStreamAsync("Hello World!");
mediaElement.SetSource(stream, stream.ContentType);
mediaElement.Play();

Managing voice commands using the STT and TTS features

We can make the applications implementing these APIs more interactive, by passing some commands using voice. Once the command is executed, the app will confirm this, using the TTS feature. To do that, we can use the STT events, like shown in the code below:


private async  void ContinuousRecognitionSession_ResultGenerated(SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
{
// We can ignore the generated text using the level of the conversion confidence (Low,Medium,High)
if (args.Result.Confidence == SpeechRecognitionConfidence.Medium || args.Result.Confidence == SpeechRecognitionConfidence.High)
{
// The key word to activate any command
//ex. user says : text red, the key word is text and the command is red
string command = "text";

if (args.Result.Text.ToLower().Contains(command))
{
string result = args.Result.Text.ToLower();
string value = result.Substring(result.IndexOf(command) + command.Length + 1).ToLower();
//The generated text may ends with a point
value = value.Replace(".", "");
switch (value)
{
case "in line": case "line": case "in-line":
dictatedTextBuilder.AppendFormat("{0}", Environment.NewLine);
await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
ReadSpecilaCommandToUser("Carriage return command is activated");
});
break;
case "blue":
await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
TXB_SpeechToText.Foreground = new SolidColorBrush(Colors.Blue);
ReadSpecilaCommandToUser("Blue color command is activated");
});
break;
case "red":
await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
TXB_SpeechToText.Foreground = new SolidColorBrush(Colors.Red);
ReadSpecilaCommandToUser("Red color command is activated");
});
break;
case "green":
await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
TXB_SpeechToText.Foreground = new SolidColorBrush(Colors.Green);
ReadSpecilaCommandToUser("Green color command is activated");
});
break;
case "black":
await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
TXB_SpeechToText.Foreground = new SolidColorBrush(Colors.Black);
ReadSpecilaCommandToUser("Black color command is activated");
});
break;
}
}
else
{
dictatedTextBuilder.Append(args.Result.Text + " ");
}

await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
TXB_SpeechToText.Text = dictatedTextBuilder.ToString();
});

}

}

private async void ReadSpecilaCommandToUser(string text)
{
if (!string.IsNullOrWhiteSpace(text))
{
using (SpeechSynthesizer speech = new SpeechSynthesizer())
{
speech.Voice = SpeechSynthesizer.AllVoices.FirstOrDefault(item => item.Language.Equals(language.LanguageTag));
SpeechSynthesisStream stream = await speech.SynthesizeTextToStreamAsync(text);

mediaElement.SetSource(stream, stream.ContentType);
mediaElement.Play();
}
}
}

 


IV. Demo

The video below shows how to edit text by voice, the app is using the Windows 10 Speech APIs:

Going further

 

Speech APIs – Universal Windows Platform:

SpeechAPIs – Project Oxford: https://github.com/Microsoft/ProjectOxford-ClientSDK/tree/master/Speech

Project Oxford : https://www.projectoxford.ai

Hey Cortana, do you speak JavaScript?

In this article I will try to describe every steps to use Cortana with your JavaScript UWP app.

The first step is to create an xml file that represents the Voice Command Definition :

<?xml version="1.0" encoding="utf-8" ?>
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
  <CommandSet xml:lang="fr-fr" Name="VDM_fr-fr">
    <CommandPrefix> VDM, </CommandPrefix>
    <Example> Affiche les dernières VDM </Example>
    
    <Command Name="showlast">
      <Example> Affiche les dernières VDM </Example>
      <ListenFor RequireAppName="BeforeOrAfterPhrase"> Affiche [les] dernières </ListenFor>
      <ListenFor RequireAppName="BeforeOrAfterPhrase"> Affiche [mes] dernières </ListenFor>
      <ListenFor RequireAppName="BeforeOrAfterPhrase"> Ouvre [les] dernières </ListenFor>
      <ListenFor RequireAppName="BeforeOrAfterPhrase"> montre [moi] [les] dernières </ListenFor>
      <Feedback> affichage des dernières VDM </Feedback>
      <Navigate />
    </Command>
    <Command Name="showcategorie">
      <Example> Affiche les VDM de travail</Example>
      <ListenFor RequireAppName="ExplicitlySpecified"> ouvre les {builtin:AppName} [de] {cat}</ListenFor>
      <ListenFor RequireAppName="ExplicitlySpecified"> affiche les {builtin:AppName} [de] {cat}</ListenFor>
      <Feedback> ouverture des VDM de {cat}</Feedback>
      <Navigate />
    </Command>

    <PhraseList Label="cat">
    </PhraseList>
  </CommandSet>
  <CommandSet xml:lang="en-us" Name="VDM_en-us">
    <CommandPrefix> FML </CommandPrefix>
    <Example> Show me the latest </Example>

    <Command Name="showlast">
      <Example> Show me the latest </Example>
      <ListenFor RequireAppName="AfterPhrase"> show [me] the latest </ListenFor>
      <ListenFor RequireAppName="AfterPhrase"> open the latest </ListenFor>
      <ListenFor RequireAppName="AfterPhrase"> display the latest </ListenFor>
      <Feedback>  display of the last FML</Feedback>
      <Navigate />
    </Command>

    <Command Name="showcategorie">
      <Example> Displays the FML of love </Example>
      <ListenFor RequireAppName="ExplicitlySpecified"> Opens the  {builtin:AppName} [of] {cat}</ListenFor>
      <ListenFor RequireAppName="ExplicitlySpecified"> Displays the {builtin:AppName} [of] {cat}</ListenFor>
      <Feedback> opening FML of {cat}</Feedback>
      <Navigate />
    </Command>

    <PhraseList Label="cat">
    </PhraseList>
  </CommandSet>

</VoiceCommands>
  • In this file the root element is the VoiceCommands Element, it’s contains a list of commandSet elements. Each commandSet is for a language.
    • An commandSet contains a list of command (and others things …)
      • A command is a “command” and contains an example, and multiple elements of ListenFor, a feedback element, and an instruction element (navigate in the first sample) that explicitly specifies that this command will navigate to your app.
        • ListenFor is the command phrase, it has a RequireAppName attribute that specifies where the app name can appear in the voice command.
      • A PhraseList that contains multiple item, each Item specifies a word or phrase that can be recognized to initiate the command that references the PhraseList (optional). You can optionnaly load dynamically a list of items from your code.

You have to be very careful, when you write this file! If you don’t respect the structure or If you miss an element, the Cortana API will fall in error without any information*.

The final step.
Instantiate your new XML file of VCD, in your JavaScript code.

In this following code, I call initCortana function to initialize the VCD file using de VoiceCommandDefinitionManager API located in the Windows.ApplicationModel.VoiceCommands namespace.

You have to pass your xml file to the « installCommandDefinitionsFromStorageFileAsync » function. If everything it’s not OK the callback of the promise returns a error and you pass by:

console.error(‘error file vdmvoicecommands.xml’, er);

In this case: check and re check an re re re check your VCD file 🙂

If the file was initialized correctly, you could add a list of items to populate the phrase lists.



    var wap = Windows.ApplicationModel.Package;
    var voiceCommandManager = Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager;

    var initCortana = function (categories) {
        categories = categories || [];
        return wap.current.installedLocation.getFileAsync("vdmvoicecommands.xml").then(function (file) {
            return voiceCommandManager.installCommandDefinitionsFromStorageFileAsync(file);
        }, function (er) {
            console.error('error file vdmvoicecommands.xml', er);
        }).then(function () {
           var language = window.navigator.userLanguage || window.navigator.language;

            var commandSetName = "VDM_" + language.toLowerCase();
            var commansets = Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager.installedCommandDefinitions;
            if (commansets.hasKey(commandSetName)) {
                var vcd = commansets.lookup(commandSetName);
                var phraseList = [];
                categories.forEach(function (c) {
                    phraseList.push(c.title);
                });
                return vcd.setPhraseListAsync("cat", phraseList).then(function () {
                    console.log("VCD loaded !");
                 }, function (er) {
                    console.error('error set phraseList', er);
                })
            } else {
                console.warning("VCD not installed yet?");
            }
        }, function (ee) {
            console.error("installCommandDefinitionsFromStorageFileAsync error", ee);
        });
    }

Now you have to handle the activation event that will get sent to your app, and parse arguments.

  app.addEventListener("activated", function (args) {
        var appLaunchVoiceCommand = activation.ActivationKind.voiceCommand || 16;
        if (args.detail.kind === appLaunchVoiceCommand) {
            return handleVoiceCommand(args);
        }
    });

The handleVoiceCommand function parse the args passed from the activated app event and do the navigation to the right place

    var app = WinJS.Application;
    var commands = {
        "showlast": function (commandresult) {
            return WinJS.Navigation.navigate(VDM.Pages.VDMList, { viewType: 'last', viewLabel: WinJS.Resources.getString('appbar_views_last') });
        },
        "showcategorie": function (commandresult) {
            var categorie = commandresult.semanticInterpretation.properties.cat[0];
            return WinJS.Navigation.navigate(VDM.Pages.VDMList, { viewType: categorie.toLowerCase(), viewLabel: categorie });
        }
    }
    var handleVoiceCommand = function(args) {
        if (args.detail && args.detail.detail) {
            var voicecommand = args.detail.detail[0];
            if (voicecommand) {
                var result = voicecommand.result;

                if (result) {
                    var commandname = result.rulePath ? result.rulePath[0] : '';
                    var properties = result.semanticInterpretation ? result.semanticInterpretation.properties : {};
                    console.log("voice activation (" + commandname + ") confidence: " + result.rawConfidence, result);
                    var cmd = commands[commandname];
                    if (cmd) {
                        return cmd(result).then(function () {
                            VDM.Utils.loadMainAds();
                        });
                    }
                }

            }
        }
    }
    app.addEventListener("activated", function (args) {
        var appLaunchVoiceCommand = activation.ActivationKind.voiceCommand || 16;
        if (args.detail.kind === appLaunchVoiceCommand) {
            return handleVoiceCommand(args);
        }
    });

Let’s talk more with the app (with appService)

If you need a deeper integration with Cortana, you could also « talk » with her using an app service.

An app service is a Background task that Cortana could call when you use a command. You will have to explicitely declare which service Cortana must call in your command file.

    <Command Name="getFML">
      <Example> tell me a FML </Example>
      <ListenFor RequireAppName="ExplicitlySpecified"> tell me a {builtin:AppName} </ListenFor>
      <Feedback> Here an FML </Feedback>
      <VoiceCommandService Target="FMLVoiceCommandService"/>
    </Command>

Now let’s implement the Application Service. You must add it to your application manifest by pointing to JavaScript file, and give it the name you use in the command file :

 <uap:Extension Category="windows.personalAssistantLaunch"/>
 <uap:Extension Category="windows.appService" StartPage="js/voiceCommandService.js">
    <uap:AppService Name="FMLVoiceCommandService"/>
 </uap:Extension>

Beware the visual studio appxmanifest editor, it removes this entry if anything changes in it (like a upgrade of version when you generate a new appx package) this bug will certainly be corrected in the update 1 of Visual Studio.

Now let’s create the javascript file and implement the service itself.

When you are using JavaScript App Services are a lot like independant web workers. You can import all the JS file you need to run your code by using importScripts

importScripts("/WinJS/js/base.js");
importScripts("/js/FML.API.js");

The service is loaded by cortana, so when is loaded the doWork function is called.
If the Windows.UI.WebUI.WebUIBackgroundTaskInstance.current.triggerDetails is an instance of Windows.ApplicationModel.AppService.AppServiceTriggerDetails, we can get voice command used to launch this task and do things like:

  • Send an waiting response message
  • displays items
  • Send a final response message
var appService = Windows.ApplicationModel.AppService;
var backgroundTaskInstance = Windows.UI.WebUI.WebUIBackgroundTaskInstance.current;
var details = backgroundTaskInstance.triggerDetails;
var deferral = backgroundTaskInstance.getDeferral();

if (details instanceof appService.AppServiceTriggerDetails) {
    voiceServiceConnection = voiceCommands.VoiceCommandServiceConnection.fromAppServiceTriggerDetails(details);
    voiceServiceConnection.addEventListener("voiceCommandCompleted", onVoiceCommandCompleted);

    voiceServiceConnection.getVoiceCommandAsync().then(function completed(voiceCommand) {

    // here you can check the voiceCommand, call an API (or read a file) and send messages to Cortana UI

            var userMessage = new voiceCommands.VoiceCommandUserMessage();
                    userMessage.spokenMessage = "I'm Cortana and I read this awesome message";
                    userMessage.displayMessage = "I'm Cortana and I read this awesome message";
            var response = voiceCommands.VoiceCommandResponse.createResponse(userMessage);
            return voiceServiceConnection.reportSuccessAsync(response);

    });
}

The displayMessage string must not exceed 256 character

And now, with this, you can ask Cortana: « hey cortana, tell me a FML »

Writing Windows 10 App Services in JavaScript

What is an App Service ?

Windows 10 introduce a bunch of new ways for applications to communicate with each others. One way is to implement « App Services ». App Services are a request/response model where one app can call a service located within another app. App Services enable communication between apps, but also with the system. If you want to implement interactive scenarios with Cortana, you will have to implement an App Service to provide data to Cortana.

If you’re the kind of people who prefer code than blabla, you may head directly to the github repository with sample applications.

App Services use the same infrastructure as BackgroundTasks, and most of the logic and implementation details still applies. It means that when your service is called, you don’t have the whole application running, but only your service. It also means that your application don’t communicate directly with your App Service. For example, your application does not get notify when your service is called, or terminated.

In all resources I can found (Build session, Channel 9 videos, samples, …), App Services are implemented in C#. Those resources are really helpfull (especially this one on Channel 9), but if (like me) you are writing apps in HTML and JavaScript, it is likely that you prefer writing those services in JavaScript and share business code with the rest of your application. Porting C# resources to JavaScript is actually very easy. In this post, we will dive into implementing an App Service in Javascript, based on a C# sample from Microsoft Virtual Academy.

Show me some code !

In a Windows Web Application (Windows application written in HTML and JavaScript), a background task, therefore an App Service, should be thought of as a special Web Worker (no postMessage with it unfortunately). It’s a standalone JavaScript file that will get caught independently by the system.

The first step to implement your App Service is to create this file. As with web workers, you could use « importScripts » to reference any code you want to share between your app and the service. Be aware that, like with Web Workers, there is no « window » or « window.document » objects inside your background task or app service. The global context points to a completely different beast, and there is no DOM.

Inside your task or service, you will access to the current instance of the BackgroundTask object using WinRT APIs, and get a deferral on it to control the lifecycle of your service. As with background task, your service can also be canceled by the system if it needs to recover memory, or if battery is running low on the device. Your task instance provide a « cancel » event that will get caught is such cases.

A minimalistic background task/app service would look like this

var backgroundTaskInstance = Windows.UI.WebUI.WebUIBackgroundTaskInstance.current;
var triggerDetails = backgroundTaskInstance.triggerDetails;
var bgtaskDeferral = backgroundTaskInstance.getDeferral();

function endBgTask() {
    backgroundTaskInstance.succeeded = true;
    bgtaskDeferral.complete();
    bgtask.close();
}

backgroundTaskInstance.addEventListener("canceled", function onCanceled(cancelEventArg) {
    return endBgTask();
});

Now we must declare this file as an App Service. For that, we must add an extension to our application in its manifest, pointing to our javascript.

<Applications>
    <Application Id="App" StartPage="default.html">
        ...
        <Extensions>
            <uap:Extension Category="windows.appService" StartPage="js/appservice.js">
                <uap:AppService Name="MyJavascriptAppService"/>
            </uap:Extension>
        </Extensions>
    </Application>
</Applications>

As you can see, we provide the path to our JavaScript file, and we are giving a name (MyJavascriptAppService) to the App Service.

Now we must implement the service logic. To do that, we will check the trigger details on our background task, and register for a request event. When the event gets activated, we received an App Service request. This request will contains a message (with request arguments), and a sendResponseAsync method to reply to the caller. On both sides, the values in the request and in the response are provided with a ValueSet object.

//check that the app service called is the one defined in the manifest. You can host
//multiple AppServices with the same JavaScript files by adding multiple entries in the application manifest
if (triggerDetails && triggerDetails.name == 'MyJavascriptAppService') {
    triggerDetails.appServiceConnection.onrequestreceived = function (args) {
        if (args.detail && args.detail.length) {
            var appservicecall = args.detail[0];
            //request arguments are available in appservicecall.request.message
            var returnMessage = new Windows.Foundation.Collections.ValueSet();
            returnMessage.insert("Result", 42);
            appservicecall.request.sendResponseAsync(returnMessage)
        }
    }        
}

Calling your App Service

The app calling your service can be any app. If you want to restrict access, you will have to implement your own security mecanism. As you may have understood, the caller and the callee doesn’t have to be written in the same language. You can call a service written in C++ from a JavaScript app. All data are passing throught Windows APIs.

Calling the app service require some arguments, the caller should provide the package family name (PFN) of the target application, and the name of the App Service (as declared in the target’s app manifest). If you don’t know your PFN, you can get it through WinRT APIs by calling « Windows.ApplicationModel.Package.current.id.familyName » in your service application.

Using the PFN and service name, you will first get a connection to your App Service, and register to the « serviceclosed » event to be notified if your service terminate.

function getServiceConnection(){
    var connection = new Windows.ApplicationModel.AppService.AppServiceConnection();
    connection.appServiceName = "MyJavascriptAppService";
    connection.packageFamilyName = "...your PFN here...";

    return connection.openAsync().then(function(connectionStatus){
        if (connectionStatus == Windows.ApplicationModel.AppService.AppServiceConnectionStatus.success) {
            connection.onserviceclosed = serviceClosed;
            return connection;
        }
        else {
            return WinJS.Promise.wrapError({ message: 'service not available' });
        }
    });
}

Once you get a valid connection, you will be able to send requests to the service

function callService(){
    return getServiceConnection().then(function (connection) {
        var message = new Windows.Foundation.Collections.ValueSet();
        message.insert("Command", "CalcSum");
        message.insert("Value1", 8);
        message.insert("Value2", 42);

        return connection.sendMessageAsync(message).then(function (response) {
            var e = response;
            if (response.status === Windows.ApplicationModel.AppService.AppServiceResponseStatus.success) {
                document.getElementById('result').innerHTML = 'calculated ' + response.message.Result;
	                
                return response.message;
            }
        });
    });
}

And voilà ! you’re ready to go. A picture is worth a thousand words, so I put a sample with service and caller apps on github for you.

Debugging your service

If you grab the sample, you can see how easy it is to debug your service. If you configure the solution to run both caller and callee on debug, you can set breakpoints in your app service. If you don’t want to run the full service app, you could also edit the properties of the project hosting the service. In the debugging section, you could set « Launch Application » to false. In that case, when you run debug for both apps, you will only see the caller application starting, but your breakpoints in the app service will get called appropriately.

Porting Windows API calls from C# to JavaScript

Windows 10 is still in preview, and as of this writing, a lot of the samples available are written in C#.
The fact that samples are not available in JavaScript does not means that the feature is not available in JavaScript. The only things that may not be available is all topics about XAML controls, like the new InkCanvas, the RelativePanel, and so on. All code related to Windows APIs may be used in JavaScript.

Porting C# code to JavaScript is actually straightforward, especially if you are familiar with WinRT development. Windows APIs (aka WinRT) are projected in all available languages (C++, C#, and JavaScript), using each language paradigms. So, moving C# code to JavaScript is just about changing variable names to lowercase (ex « connection.SendMessageAsync() » => « connection.sendMessageAsync() »), or use the « on… » syntax for event. For example « connection.RequestReceived += someEventHandlerCallback » will convert to « connection.onrequestreceived += someEventHandlerCallback ».

As you can see, this is really easy…

Build 2015 – User Data : Working with Contacts, appointments, text messages and more

Windows 10 introduit de nouvelles API pour interagir avec les entités les plus fondamentales pour un utilisateur : contacts, agenda, appels, sms, messages, etc… toutes ces entités manipulées par les applications fournies dans Windows comme People, mail ou Cortana. Les API utilisées par ces applications sont maintenant exposées dans WinRT.

L’objectif est de pouvoir manipuler et requêter ces entités et les données associées (historique des appels par exemple). Pour le moment, tout n’est pas accessible sur toutes les entités. L’entité la mieux couverte niveau API celle concernant les contacts.

Une application peut attacher des données supplémentaires sur les entités, ou créer des conteneurs (datastore) qui lui sont spécifiques. Ces données peuvent n’être visibles que par l’application, ou partagées avec les autres. Les datastores proposent des fonctions avancées comme le tracking de modifications, ainsi que des évènements particuliers, dans l’application, et pour certains types d’entités, le déclenchement de tâches de fond.

WinJS Contrib Basics

Now that we set the stage for WinJS and WinJS Contrib, it’s time to dig into some code goodness.
In this episode, we will look at the fundamental of WinJS Contrib.

interactive elements

You have interactive elements in all apps. Weither it is with mouse or touch, it is good practice to provide visual feedback on interactions, otherwise the user feel trust in your application. WinJS Contrib has an abstraction over interactive element. Using it will provide an appropriate cursor with mouse, and a visual feedback with touch. It is as simple as :

WinJSContrib.UI.tap(domElement, callback, options);

the third argument is optional and allow you to specify such thing as « disableAnimation » to avoid the « pressed » effect with touch, or « lock » to add interactivity on elements in a scrollable are (like adding a button inside an item in a listview). The beauty is that « tap » is cross platform and will act as « fastclick » on Android and iOS.

When using it, a css class « tap » will be added on the element, and when element is pressed a « tapped » will be set as long as the element is pressed. You can use this to override the default animation.

In previous post, we explained that WinJS Contrib replaces the page implementation of WinJS. Our page control allow you to declare taps in your markup. You must simply add a « tap » attribute on an element, and as a value, you pass the name of the function on your page, like this :

<div class="button" tap="someFunctionOnMyPage"></div>

When called, your function will be bound to the page, it means that when the function is called, « this » refer to your page.

You could specify arguments to tap by adding a « tap-args » attribute. The value will be passed in the argument to the function on your page. The value could be a simple string, or use the resolution pipeline from WinJS Contrib. We will expand on this pipeline in another post, for now, just note that you can inject argument from a property or a function of your page.

<div class="button" tap="someFunctionOnMyPage" tap-args="ctrl:somePropertyOrFunctionOnMyPage"></div>

Another thing that is really common in all applications is to navigate after clicking a button. You can implement this as easily as tap with a « linkto » attribute. As a value, you will specify the url of the target page.

<div class="button" linkto="/pages/home/home.html"></div>

In case you want more, linkto can have arguments, just like tap. The arguments will be passed to the navigation

<div class="button" linkto="/pages/home/home.html" linktoargs="obj:{ index: 3}"></div>

interacting with DOM

When writing a WinJS page, you end up writing tons of DOM selector to get instances of the controls, or DOM element that you will interact with. Your page end up looking a bit ugly with things like this all around the place :

var listView = page.element.querySelector('#myListView').winControl;

If you are wandering why scoping selector to page, go have a look at some WinJS best practices.

WinJS Contrib page introduce a mecanism to map elements and controls as properties on your page by adding a « member » attribute on the node. You could leave it blank, and the name of the property will be the « id » on the node, or you could specify the name of the property.

<div class="button" member="btnClose"></div>

If the element has a control attached on it, the property will hold the control, otherwise it will be the DOM element :

<div member="listview" data-win-control="WinJS.UI.ListView"></div>

then you will use it magically in your page :

page.listview.itemDataSource = new WinJS.Binding.List().dataSource;

If you prefer having the declarations in your page, we still have some goodness for you. The page control has « q » and « qAll » functions for running querySelector and querySelectorAll from your page element, which is at least shorter than the full syntax. The « qAll » function is also kind enougth to map the result as an array, so you can use forEach on the result.

page.qAll('.button').forEach(function(buttonElement){
    //do something
});

preventing leaks

The first cause of memory leaks is unreleased event listeners having a closure on some global variable. To ensure releasing your listener without adding some (ugly) extra code to your page, WinJS Contrib provide an event tracker object. The page control holds an instance of it that you can use. You use this object to attach functions to event listeners, and when you dispose the tracker, it will dispose all attached events automatically. If you use the page instance, the page will take care of releasing the tracker.

instead of

ctrl.showBinded = ctrl.show.bind(ctrl);    		 
ctrl.searchInput.addEventListener('focus', ctrl.showBinded);
...
ctrl.searchInput.removeEventListener('focus', ctrl.showBinded);

you have :

ctrl.eventTracker.addEvent(ctrl.searchInput, 'focus', ctrl.show.bind(ctrl));    

does all this really help ?

Well, see for yourself by looking at the small sample application that we made. You have one version with WinJS only, and another with WinJS Contrib.

This application is for searching Flickr. If you look at the code for the search bar control and compare the pure WinJS and the WinJS Contrib version, you will see a real difference (it removed 45 lines of code on 154…)

Next time we will see other features enabled by the WinJS Contrib page