NAV Navbar

Nuance Mix Runtime Event Logs

The Mix ASR, NLU, TTS, and Dialog runtime services generate event logs that can be used by Nuance customers to troubleshoot deployments, monitor operational data, produce reports, generate billing records, track usage, and so on.

Mix offers two REST APIs to help customers access and use this data:

Note: Before you can use the Mix logging APIs, Nuance must enable event logs in your application. Please contact your Nuance representative for more information.

Kafka API essentials

The Nuance Kafka Consumer API uses the standard Kafka concepts and terminology. This section assumes that you have a basic understanding of Kafka.

In particular, the Kakfa Consumer API uses the following concepts.

Messages/events

A Kafka message corresponds to an event that occurred in Mix. There are two main types of Mix events:

Each message has a key that is composed of the following two fields:

Field Description
service Name of the runtime service that generated this message. Valid values are: TTSaaS, ASRaaS, NLUaaS, DLGaaS.
id Randomly-generated UUID that uniquely identifies the message.

Topics

Example of topic in Mix

"topic": "DEMO-OMNICHANNEL-APP-DEV"

Kafka messages are organized into topics. In Mix, topic names correspond to app IDs. Therefore, a Mix topic stores messages for a specific app ID.

Partitions

Example of Get partitions endpoint:

{
    "topic": "DEMO-OMNICHANNEL-APP-DEV",
    "partitions": [
        0
    ]
}

Kafka topics are divided into partitions.

Topics and partitions are created by Nuance. To get the list of partitions for a topic, use the Get Partitions endpoint.

This endpoint returns the topic name and partition numbers that correspond to the client credentials used when creating the access token.

Consumers

Consumers pull the messages from topics. You create your own consumers using the Create Consumer endpoint.

In Mix, consumer names follow this nomenclature:

consumer-UUIDv4

where UUIDv4 is a Version 4 UUID. For example:

consumer-9872ec64-50d9-4dcd-9f4f-e8b5099a7f4f

Consumer groups

A consumer group is a set of consumers that cooperate to consume data from topics. Partitions are assigned to a consumer group. Each consumer in a consumer group receives messages from a different subset of the assigned partitions.

You create your own consumer groups using the Create Consumer endpoint.

In Mix, consumer group names follow this nomenclature:

appID-appID/topic-clientName-clientName-two-digit number

Where:

For example:

appID-DEMO-OMNICHANNEL-APP-DEV-clientName-default-00

Consuming messages

Consumers pull the messages from topics. A consumer must be assigned to one or more partitions to pull messages. There are two ways to assign partitions:

Note: Use only one of the above methods.

The following diagram summarizes some of the concepts covered so far. Topic A includes two partitions, and each partition stores messages as they are generated. Consumers are part of consumer groups and read the messages.

Kafka concepts summary

To determine which messages to pull, Kafka uses offsets.

Using offsets

Offsets specify a position in a partition and determine where a consumer should start pulling messages.

Each consumer tracks its position (that is, offset) in a partition. By default, when a consumer is created, it is set to start reading the newest message in a partition. For example, in the diagram above, a new consumer assigned to partition 0 will start reading the next message coming at offset 9.

Kafka uses the following terms when talking about offsets:

Offsets are attached to a consumer group, so consumers should use the same consumer group to maintain the offsets.

Authorization

Nuance Mix uses the OAuth 2.0 protocol for authorization. All client applications must provide an access token to access event logs and audio files for an app ID.

The access token is created using client credentials for an app ID. Specify a scope of log to get access to the event logs and audio files, as shown in the following example:


$ export CLIENT_ID="appID%3ADEMO-OMNICHANNEL-APP-DEV%3Ageo%3Aqa%3AclientName%3AScriptUser"
$ export SECRET="riAbk888CC2B.97D7eUklVe6pD"
$ export TOKEN="`curl -s -u "$CLIENT_ID:$SECRET" "https://auth.crt.nuance.com/
oauth2/token" -d 'grant_type=client_credentials' -d 'scope=log' | python -m 
json.tool |  python -c 'import sys, json; print(json.load(sys.stdin)["access_token"])'`"

The access token returned will provide access to event logs for the app ID linked to the client credentials specified. For example, the credentials in the sample code above will provide access to the event logs for the DEMO-OMNICHANNEL-APP-DEV app ID:

This access token must be included in every API request. Specify the access token as a bearer token in the request header.

For more information about the authorization process, see Authorization in Mix.

You must reuse the same access token across the different consumer processes and not acquire a new one at each API call. See Reusing access tokens for details.

Quick start

This section takes you through the process of getting event logs for a Mix application using Postman.

It assumes that:

Import the Nuance-Kafka-REST-API collection into Postman

You can get up and running quickly by importing the Nuance-Kafka-REST-API collection into Postman.

This documentation provides the Nuance-Kafka-REST-API_Postman_collection.zip file, which includes the following files:

  1. Download and unzip the Nuance-Kafka-REST-API_Postman_collection.zip file.
  2. In Postman, select File->Import.
  3. Click the Folder tab.
  4. Click Choose folder from your computer, navigate to the directory that contains the Nuance-Kafka-REST-API_Postman_collection folder, select it, and click Select folder.
  5. Click Import.

The Kafka Postman collection and environment variables are imported.

Configure the consumer name and configure group name variables

Example of NUANCE-KAFKA environment variables
(Click the image for a close-up view)

Most requests require a valid consumer name and consumer group name.

The Nuance-Kafka-REST-API collection uses variables for these values. To configure the variables for the endpoints:

  1. In Postman, click Environments (left side of the Postman window).
    The list of environments appears.
  2. Click NUANCE-KAFKA.
    The variables for the NUANCE-KAFKA environment are displayed.
  3. Set a valid consumer name in the Current value field for the consumer_name variable.
  4. Set a valid consumer group name in the Current value field for the consumer_group variable.
  5. Leave all other fields as is.
  6. Click Save to save the consumer variable values.

Make sure to click the Save button, otherwise your updates will not be saved.

Select the NUANCE-KAFKA environment

Selecting the NUANCE-KAFKA environment

The Nuance-Kafka-REST-API collection requires variables that are set in the NUANCE-KAFKA environment. Always make sure that the NUANCE-KAFKA environment is selected when you run endpoints from the Nuance-Kafka-REST-API collection:

  1. In Postman, click Collections (left side of the Postman window) and expand the Nuance-Kafka-REST-API collection.
  2. In the top-right corner of the Postman window, to the left of the eye icon eye icon, select NUANCE-KAFKA.

You can now run the endpoints.

Authorize your application

Authorize your application: Postman Authorization tab

To access the Mix Kafka API, you need a valid access token. This procedure describes how to create a valid access token using the Authorization endpoint.

The credentials that you use to get the access token will determine the event logs that you have access to. For more information, see Authorization.

  1. In the Nuance-Kafka-REST-API collection, select the Authorization request.
  2. Click the Authorization tab and specify the following information:
  3. Click Send.

Response

The access token is returned in the response body; for example:


{
    "access_token": "eyJhbGciOiJSUzI1NiIsImtpZ...YOE",
    "expires_in": 899,
    "scope": "log",
    "token_type": "bearer"
}

This token is saved in the logaccesstoken NUANCE-KAFKA environment variable and will be used automatically for the following requests in this environment.

Note that the access token is valid for 15 minutes. You will need to get a new one after it expires.

Get the topic name and partition numbers

First get the topic name and partition numbers with the Get Partitions endpoint:

To get the topic and partitions:

  1. In the Nuance-Kafka-REST-API collection, expand the Partitions folder and select Get Partitions.
  2. Click Send.

Response

The topic name and partition numbers are returned in the response body; for example:


{
    "topic": "DEMO-OMNICHANNEL-APP-DEV",
    "partitions": [
        0
    ]
}

Create a consumer in a consumer group

Create a consumer: Postman Body tab

You create a consumer in a consumer group with the Create Consumer endpoint.

By default, when you create a consumer, it is set to start reading the newest message in a partition. In this quick start, we want to retrieve the event logs that have already been created, so we will set the auto.offset.reset parameter to earliest. This will ensure that the new consumer instance will start reading logs from the beginning of the partition.

Note: If the consumer group name already exists, the consumer is added to this group. Otherwise, the group is created.

To create a consumer in a consumer group:

  1. Expand the Consumers and Groups folder and select Create Consumer.
  2. Click the Body tab and set the auto.offset.reset parameter to earliest.
  3. Click Send.

Response

A 204 No Content code is returned, indicating that the consumer and consumer group were created successfully.

Subscribe the consumer to a partition

Assign partitions dynamically: Postman Body tab

The next step is to subscribe the consumer to one or more partitions to get messages. There are two ways to subscribe to a partition:

In this quick start, we'll assign the partition dynamically, using the Subscribe endpoint. This endpoint dynamically assigns the consumer to the partitions of the specified topic.

To assign the consumer to a specific partition:

  1. Expand the Subscriptions folder and select Subscribe.
  2. Click the Body tab and specify the topic name obtained in Get the topic and partitions.
  3. Click Send.

Response

A 204 No Content code is returned, indicating that the partition was assigned successfully.

Get event log records

You fetch event log records for a consumer with the Get Records endpoint.

To get event log records:

  1. Expand the Records folder and select Get Records.
  2. Click Send.

Response

Event logs are returned; for example:


[
    {
        "topic": "DEMO-OMNICHANNEL-APP-DEV",
        "key": {
            "service": "DLGaaS",
            "id": "45206957-3770-4883-87c8-3b8350e39611"
        },
        "value": {
            "specversion": "1.0",
            "service": "DLGaaS",
            "source": "NIIEventLogger",
            "type": "NIIEventLog",
            "id": "45206957-3770-4883-87c8-3b8350e39611",
            "timestamp": "2021-02-08T16:48:12.330Z",
            ...

Delete the consumer

For a graceful shutdown of your application, delete the consumer with the Delete Consumer endpoint.

To delete the consumer:

  1. Expand the Consumers and Groups folder and select Delete Consumer.
  2. Click Send.

Response

A 204 No Content code is returned, indicating that the consumer was deleted successfully.

Consumer lifecycle

Consumer lifecycle for a consumer instance

Consumer lifecycle

The Quick Start section provides a very simple use case to introduce the main concepts and steps required when using the Kafka API.

This section describes recommended practices for using the API in a real-life scenario.

The example in this use case assumes:

  1. Create a consumer instance and specify:
    • A new consumer instance (c1)
    • A consumer group (g1)
    • Parameter auto.offset.reset: earliest
      This will ensure that the new consumer instance will start reading logs from the beginning of the partition. Note that this parameter is ignored if the group already exists and an offset was committed.
  2. Get the list of available partitions.
    This will determine the number of consumer instances that need to be created. Nuance recommends that the number of consumer instances that you create should be less than or equal to the number of available partitions in a topic.
  3. Call the Assign Partitions endpoint to manually assign partition 0 to consumer instance c1.
  4. Call the Get Records endpoint in a loop to fetch the records for consumer c1, but do not process them at this point to avoid consumer performance issues.
  5. Call the Get Assigned Partitions endpoint to get the list of partitions assigned to consumer c1.
  6. Call the Commit Offsets endpoint to commit the offset for the last fetched record.
  7. Create a consumer instance as described in step 1, using c2 as the consumer instance.
    Creating and managing consumer instance c2 can be done in parallel with creating and managing consumer instance c1.
  8. Call the Assign Partitions endpoint to manually assign partition 1 to consumer instance c2.
  9. Call the Get Records endpoint in a loop to fetch the records for consumer c2, as described in step 4.
  10. When completed, do a graceful shutdown of the consumers:
    1. Delete consumer c1.
    2. Delete consumer c2.

Retarting consumers

To restart consumers, repeat all the steps above. The consumers will read new records from the last commit offsets.

Recommendations

Creating clients

Nuance recommends that you create a client that is dedicated to collecting logs. In addition, if multiple organizations need to access event logs for the same app ID, Nuance recommends that you create one client per organization.

For example, let's say that two different groups in your company (research and professional services) need to access event logs. Create two different clients in the Manage Tab, one for each group, as follows:

Manage tab

For more information on creating clients, see Create a client ID .

Creating customer groups

If the same client must be shared between multiple organizations, Nuance recommends that you create different client groups, one per organization. Offsets are attached to a consumer group, so creating different groups will ensure that committing offsets will not impact other users.

Reusing access tokens

You must reuse the same access token across the different consumer processes. Do not acquire a new token at each API call. The same token can be used between the Kafka and AFSS APIs.

For example, if you are running 20 consumer processes for a specific app ID, generate a single access token and share it across the 20 consumers.

Nuance also recommends that you use the token until it expires. The expected token expiration validity period is returned in the token response.

Please note that the application may be limited to a certain maximum number of access token requests over a period of time.

Number of consumer instances

The number of consumer instances that you create should be less than or equal to the number of available partitions in a topic.

Nuance recommends that, for production applications, you use one consumer instance per partition.

Retention period

Messages

The retention period for messages is 28 days. To avoid any loss of data, consumers should make sure that the data is consumed within the retention period.

Consumers should use the same consumer group to maintain the offsets (all offsets are attached to the consumer group).

Audio files

The retention period for audio files is 14 days.

Graceful shutdown of consumers

Consumers should call the Delete Consumer and Unsubscribe endpoints for a graceful shutdown of the consumer.

Handling incoming event logs

Event logs are generated as your customers use your application; as such, they are often generated in waves and usually consist of a combination of peaks and slower periods.

Nuance recommends that you write your client so that it pulls data continuously; however, keep in mind that the client may not always be able to consume logs at the same rate as they are being produced. For example, a client may keep up with data collection during off-peak hours, fall behind in peak hours, and then catch up again in off-peak hours.

If your peak periods are so busy that your client cannot keep up, Nuance does not recommend that you use the log API for real-time monitoring.

Troubleshooting

This section describes some of the most common issues when working with the Kafka API.

117: Invalid consumer name

Error message

{
    "error_code": 117,
    "message": "Invalid consumer name"
}

The consumer name specified is not valid.

See Consumers for more information.

121: Invalid client name

Error message

{
    "error_code": 121,
    "message": "Invalid client name."
}

The client name specified in the consumer group name is not valid.

See Consumer groups for more information.

401: Unauthorized

Error message

{
  "error":
  {
    "code": 401,
    "status": "Unauthorized",
    "message": "The request could not be authorized"
  }
}

The credentials specified are incorrect or expired.

Make sure that you have generated the access token and that you are providing the credentials as described in Authorization. Note that the token needs to be regenerated regularly. See Access token lifetime for details.

40403: Consumer instance not found

Error message

{
    "error_code": 40403,
    "message": "Consumer instance not found."
}

The default inactivity time for a consumer instance is 2 minutes. If the consumer is not active for more than 2 minutes, the consumer is deleted. This ensures that customer instances that are not used do not take server resources.

To solve this issue:

40903: Illegal state: No current assignment for partition

Error message

{
  "error_code": 40903,
  "message": "Illegal state: No current assignment for partition ..."
}

This error indicates that the consumer is trying to access a partition that is not assigned to it.

To solve the issue, call the Get Assigned Partitions endpoint to get the list of partitions assigned to the consumer instance.

40903: Illegal state: Subscription to topics, partitions and pattern are mutually exclusive

Error message

{
  "error_code": 40903,
  "message": "Illegal state: Subscription to topics, partitions and pattern are mutually exclusive"
}

This error indicates that both the Subscribe endpoint and the Assign Partitions endpoint were used to assign partitions for the consumer.

Use either the Subscribe endpoint (to assign partitions automatically) or the Assign Partitions endpoint (to manually assign partitions).

Consumer gets stuck or times out

This may occur if the consumer instance is not deleted during shutdown. A consumer instance remains alive in the server until the consumer times out. In this scenario, the orphan consumer instance keeps a partition blocked until the timeout period is reached and the consumer is deleted. Any new consumer created during this period may get stuck since the partition is blocked by the orphan consumer.

Make sure to manage the lifecycle of consumer instances as recommended in Consumer lifecycle.

About payloads

Each event log message has the same top-level structure, which is based on standard Kafka messages.

The top-level value field contains high-level information about the event that generated the event log and follows the CloudEvents - Version 1.0 specification.

In particular, it includes the data field, which provides the actual contents of the payloads. This field includes a set of standard fields, which are available for all payloads, as well as a set of service-specific fields.

This section first describes the top-level structure of the event logs. The remainder of this document describes the specifics of each service payloads.

Top-level Kafka structure

Top-level Kafka structure

[{
        "topic": "DEMO-OMNICHANNEL-APP-DEV",
        "key": {
            "service": "DLGaaS",
            "id": "ca6e4281-47c1-4cc8-a3e2-48d0bcda29de"
        },
        "value": {
          // Contents of message. See Value field.
        },
        "partition": 0,
        "offset": 1057
    }
]

All Kafka messages have the following top-level structure:

Field Description
topic Name of the topic. In Mix, this is the app ID.
key Key for this message. Messages with the same key go to the same partition in the topic. The key includes two fields:
  • service: Name of the runtime service that generated this message. Valid values are: TTSaaS, ASRaaS, NLUaaS, DLGaaS.
  • id: Randomly-generated UUID that uniquely identifies the message.
value Contents of the Kafka message. See Value field.
partition Partition number for this topic. See Partitions.
offset Integer number representing the last record that Kafka has sent to a consumer. See Using offsets.

Value field

Sample value field structure

...
"value": {
    "specversion": "1.0",
    "service": "DLGaaS",
    "source": "nuance.dlg.v1.DialogService/Start",
    "type": "Start",
    "id": "892e1679-065d-4bdd-8112-ff12f64e70cf",
    "timestamp": "2021-03-15T14:39:00.393Z",
    "appid": "DEMO-OMNICHANNEL-APP-DEV",
    "datacontenttype": "application/json",
    "data": {
        ...
    }
},
...

The top-level value field contains high-level information about the event that generated the event log. It follows the CloudEvents - Version 1.0 specification and has the following structure:

Field Description
specversion Version of the specification.
service Name of the runtime service that generated this message. Valid values are: TTSaaS, ASRaaS, NLUaaS, DLGaaS.
source Context (endpoint) in which the event was generated. Possible values are service-specific. See Source and type fields.
type Type of the event. Possible values depend on the source. See Source and type fields.
id Randomly-generated UUID that identifies the event. The combination of the source and id fields is unique for each distinct event.
timestamp Timestamp of when the event occurred.
appid App ID for this event.
datacontenttype Content type of data field. Always application/json.
data Contents of the payload. See Data field.

Source and type fields

The source field specifies the context in which the event was generated.

Service source field type field
ASRaaS nuance.asr.v1.Recognizer/Recognize Recognize
DLGaaS nuance.dlg.v1.DialogService/Start Start
DLGaaS nuance.dlg.v1.DialogService/Update Update
DLGaaS nuance.dlg.v1.DialogService/Execute Execute
DLGaaS nuance.dlg.v1.DialogService/ExecuteStream ExecuteStream
DLGaaS nuance.dlg.v1.DialogService/Stop Stop
DLGaaS NIIEventLogger NIIEventLog
NLUaaS nuance.nlu.v1.Runtime/Interpret Interpret
TTSaaS nuance.tts.v1.Synthesizer/Synthesize Synthesize
TTSaaS nuance.tts.v1.Synthesizer/UnarySynthesize Synthesize

Data field

"data": {
  "dataContentType": "application/x-nuance-dlg-interaction-summary.v1+json",
  "traceid": "547d2679e93eb9631fca96e1adf36dd1",
  "requestid": "f30758a8-536f-9239-ad31-8cce730eeeb0",
  "userid": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "locale": "en-US",
  "clientData": { // for TTS and NII logs
    "company": "Aardvark",
    "location": "Montreal"
  },
  "processingTime": {
    "startTime": "2021-03-15T14:40:59.196Z",
    "durationMs": 70
  },
  "request": {
    // Service-specific fields
    "clientData": { // for ASR, DLG, and NLU logs
      "company": "Aardvark",
      "location": "Montreal"
    }
  },
  "response": {
    "status": {
      "code": 200,
      "message": "OK",
      "details": ""
    }
    // Service-specific fields
  }
}

The data field specifies the actual content of the payload. It includes a set of standard fields, which are available for all payloads, as well as a set of service-specific fields, which depend on the value of the dataContentType field.

The following table lists the standard fields:

Field Description
dataContentType Defines the type of content that will be available in the data field. See dataContentType field.
traceid Currently not used.
requestid Nuance-generated UUID that identifies an entire API request through all the services. This field can be used for debugging and to correlate all the event logs that relate to a specific interaction. See Correlate logs for a specific API interaction.
clientRequestid Client-provided ID that identifies an entire API request through all the services. See Client-provided request ID.
userid Contents of the user_id field provided in the API. For example, for DLGaaS, this is provided as part of the StartRequest. The user ID is hashed using SHA256 and is provided as a 64-character hexadecimal string. See User ID.
locale Locale used for this interaction.
processingTime Included in the final response payload, when duration can be calculated. Includes the following fields:
  • startTime: Time that the request is received.
  • firstAudioBufferTime: Time that the first audio buffer is received. Applies to TTSaaS only.
  • durationMs: Duration of the request in milliseconds. The duration is defined as the number of milliseconds between the time that the request has arrived and the time that the response it sent. It includes the time required by the other invoked services, such as when the dialog engine invokes NLU.
clientData Contents of the client_data map provided in the API. See Client data. For TTSaaS and NII logs, this is provided at the top level of the data field. For DLGaaS, NLUaaS, and ASRaaS, the client_data field is provided in the request object.
request Contents of the request, if applicable. Value is service-specific.
response Contents of the response, if applicable. Can contain a status code, message, and details. Value is service-specific.

dataContentType field

The dataContentType field specifies the content type of the data field. In particular, it determines which additional service-specific fields are included in the payload.

The dataContentType value has the following structure:

application/<i>contenttype</i>.<i>version</i>+json

Where:

The following table links to the sections that document each dataContentType payload:

Service Field See
ASRaaS application/x-nuance-asr-callsummary.v2+json asr-callsummary payload
application/x-nuance-asr-finalresultresponse.v2+json asr-finalresultresponse payload
application/x-nuance-asr-finalstatusresponse.v2+json asr-finalstatusresponse payload
application/x-nuance-asr-partialresultresponse.v2+json asr-partialresultresponse payload
application/x-nuance-asr-recognitioninitmessage.v2+json asr-recognitioninitmessage payload
application/x-nuance-asr-startspeechresponse.v2+json asr-startspeechresponse payload
application/x-nuance-asr-statusresponse.v2+json asr-statusresponse payload
DLGaaS application/x-nuance-dlg-interaction-summary.v1+json dlg-interaction-summary payload
application/x-nuance-dlg-nii-logs.v1+json dlg-nii-logs payload
NLUaaS application/x-nuance-nluaas-interpretation.v1+json nluaas-interpretation payload
TTSaaS application/x-nuance-tts-callsummary.v3+json tts-callsummary

About payload versions

The version number (for example, v1 in application/x-nuance-dlg-nii-logs.v1+json) specifies the version of the specification for this data content type. Non-breaking changes may be applied to the specification without updating the version number. Non-breaking changes include updates such as adding fields, adding new events, adding new values, and so on.

The client application that consumes the payloads should take this into account and should not assume that the structure and content are fixed.

Should a breaking change occur (for example, an event or a field is no longer available), then the version number will be increased to indicate this change.

Dialog payloads

There are two types of dialog event logs:

A complete dialog session contains a combination of these two types of event logs.

dlg-interaction-summary payload

Sample payload: DLGaaS-Start-End event

"data":
{
  "dataContentType": "application/x-nuance-dlg-interaction-summary.v1+json",
  "traceid": "1f16b4bd07b2f1f9c548acaedc487c56",
  "requestid": "01b73d0f-bfd6-9921-a160-ad037d2e6036",
  "sessionid": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13",
  "channel": "default",
  "sessionTimeoutSec": 900,
  "locale": "en-US",
  "processingTime":
  {
    "startTime": "2021-03-15T14:41:46.313Z",
    "durationMs": 110
  },
  "events": [
    {
      "event": "DLGaaS-Start-End",
      "time": "2021-03-15T14:41:46.424Z"
    }
  ],
  "request":
  {
    "selector":
    {
      "channel": "default",
      "language": "en-US",
      "library": "default"
    },
    "payload":
    {
      "modelRef":
      {
        "uri": "urn:nuance-mix:tag:model/A1880_C1/mix.dialog"
      },
      "data": {}
    },
    "sessionTimeoutSec": 900
  },
  "response":
  {
    "status":
    {
      "code": 200,
      "message": "OK",
      "detail": ""
    },
    "payload":
    {
      "sessionId": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13"
    }
  }
}

Sample payload: DLGaaS-ExecuteStream-End event

"data":
{
  "dataContentType": "application/x-nuance-dlg-interaction-summary.v1+json",
  "traceid": "362ce333dfa6d3009c716ca40ca4cfd5",
  "requestid": "d69cb4e2-15c5-9b2b-9827-45bc6ad073e8",
  "sessionid": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13",
  "processingTime":
  {
    "startTime": "2021-03-15T14:41:52.607Z",
    "durationMs": 6815
  },
  "events": [
    {
      "event": "DLGaaS-ExecuteStream-End",
      "time": "2021-03-15T14:41:59.423Z"
    }
  ],
  "request":
  {
    "sessionId": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13",
    "selector":
    {
      "channel": "default",
      "language": "en-US",
      "library": "default"
    },
    "payload": {}
  },
  "response":
  {
    "status":
    {
      "code": 200,
      "message": "OK",
      "detail": ""
    },
    "payload":
    {
      "messages": [
        {
          "visual": [
            {
              "text": "Perfect, a double espresso coming right up!"
            }
          ],
          "view": {}
        }
      ],
      "endAction":
      {
        "data": {},
        "id": "End dialog"
      }
    }
  }
}

Sample payload: DLGaaS-Stop-End event

"data":
{
  "dataContentType": "application/x-nuance-dlg-interaction-summary.v1+json",
  "traceid": "c04aca32c263b1a1cbf467b7182e6863",
  "requestid": "d7823d68-f9bb-9a8e-a819-66499a3df1ac",
  "sessionid": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13",
  "processingTime":
  {
    "startTime": "2021-03-15T14:41:59.550Z",
    "durationMs": 8
  },
  "events": [
    {
      "event": "DLGaaS-Stop-End",
      "time": "2021-03-15T14:41:59.558Z"
    }
  ],
  "request":
  {
    "sessionId": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13"
  },
  "response":
  {
    "status":
    {
      "code": 404,
      "message": "NOT_FOUND",
      "detail": "Could not find session for [c475ff51-d60c-4ad0-b1df-7fb4aae43b13]"
    }
  }
}

Current version: v1

The application/x-nuance-dlg-interaction-summary payloads provide an overview of the Runtime API interactions that occur in a DLGaaS session.

In addition to the standard fields described in Data field, messages with the application/x-nuance-dlg-interaction-summary dataContentType include the following fields:

Field Description
sessionid ID of the session. It is always included for all dialog events, except for DLGaaS-Start-Start. For DLGaaS-Start-Start, the session ID is included only when it is passed in the StartRequest.
events Array of event and time values for this transaction. See table below for details.
channel Channel used for the interaction
sessionTimeoutSec Session timeout value (in seconds), after which the session is terminated
request Contents of the request. See table below for details.
response Contents of the response. See table below for details.

Note: The DLGaaS-Start-Start event will only include the sessionid when it is passed in on the Start request. If the session is not passed in, the dialog service will create one and return it in the DLGaaS-Start-End event. All other requests will include it.

Request and response objects

The request and response objects available in the event logs depend on the dialog events that generated the logs.

The following table describes the contents of the request and response objects according to the event that generated the event log:

API Method Possible events Contents of request Contents of response1
Start DLGaaS-Start-Start for StartRequest N/A N/A
DLGaaS-Start-End for StartResponse Content provided in StartRequest StartResponse payload
Update DLGaaS-Update-Start for UpdateRequest N/A N/A
DLGaaS-Update-End for UpdateResponse Content provided in UpdateRequest payload N/A
Execute DLGaaS-Execute-Start for ExecuteRequest N/A N/A
DLGaaS-Execute-End for ExecuteResponse Content provided in ExecuteRequest ExecuteResponse payload
Execute
Stream
DLGaaS-ExecuteStream-Start for ExecuteRequest in StreamInput N/A N/A
DLGaaS-ExecuteStream-End for ExecuteResponse in StreamOutput Content provided in the ExecuteRequest ExecuteResponse payload
ASRaaS-reco-Start when ASRaaS is invoked N/A N/A
ASRaaS-reco-End when the result is received from ASRaaS N/A N/A
NLUaaS-interpret-Start when NLUaaS is invoked N/A N/A
NLUaaS-interpret-End when the result is received from NLUaaS N/A N/A
TTSaaS-synthesize-Start when TTSaaS is invoked N/A N/A
TTSaaS-synthesize-End when the result is received from TTSaaS N/A N/A
Stop DLGaaS-Stop-Start for StopRequest N/A N/A
DLGaaS-Stop-End for StopResponse Content provided in StopRequest N/A

1. In addition to default response content described in Data field.

The following diagram shows at which point in a dialog interaction each event gets generated:

(Click the image for a close-up view)

dlg-nii-logs payload

Sample payload: dlg-nii-logs

{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key": {
    "service": "DLGaaS",
    "id": "977f87ae-5923-4281-8e91-1f1968baea7b"
  },
  "value": {
    "specversion": "1.0",
    "service": "DLGaaS",
    "source": "NIIEventLogger",
    "type": "NIIEventLog",
    "id": "977f87ae-5923-4281-8e91-1f1968baea7b",
    "timestamp": "2021-01-19T19:06:07.113Z",
    "appid": "DEMO-OMNICHANNEL-APP-DEV",
    "datacontenttype": "application/json",
    "data": {
      "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
      "traceid": "8bbedbb2353cdc1d",
      "requestid": "2c06142d-71b7-4077-bf4b-e3892811b3d8",
      "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
      "locale": "en-US",
      "clientData": {},
      "seqid": "1",
      "events": [{
          "name": "session-start",
          "value": {
            "runtime": {
              "api": "v1"
            },
            "project": {
              "name": "TestMixClient",
              "namespace": "alex.smith@company.com",
              "deployed": "2020-12-10T19:52:25.224Z",
              "contextTag": "A1880_C1",
              "id": "8655"
            },
            "selector": {
              "channel": "default",
              "language": "en-US"
            },
            "version": {
              "asr": {
                "en_US": "2"
              },
              "nlu": {
                "en_US": "2"
              },
              "dlg": "2"
            },
            "user": {
              "systemID": null,
              "userChannelID": null,
              "userAuxiliaryID": null,
              "userGlobalID": null,
              "location": null
            },
            "timeout": 900
          }
        }
      ]
    }
  },
  "partition": 0,
  "offset": 312
}

Current version: v1

The application/x-nuance-dlg-nii-logs payloads provide detailed information about events related to the behavior of the dialog application (for example, the dialog was started, input was received, a transition occurred, and so on).

In addition to the standard fields described in Data field, messages with the application/x-nuance-dlg-nii-logs.v1+json dataContentType include the following fields:

Field Description
sessionid ID of the dialog session
seqid Sequence ID of the NII log; starts at 1 and increments for each NII message in a dialog session. The seqid ensures that you can determine the sequence of the NII logs in a dialog session, since these logs may not be received in the order in which they are created and sent.
events Global and custom events in the project; see NII log events for details. A single NII message may contain one or more events.

NII log events

NII logs can include the following global and custom events. Each event has two fields:

Event name Description
session-start Indicates the start of an interaction between a user and a dialog application.
selector Provides the channel and language combination used for an interaction.
decision Provides the next node in the dialog flow after a decision. Relates to a decision node.
transition Indicates that the dialog flow encountered a transition.
message Provides the contents of the message payload sent to the client application. Relates to a message node.
qa-config Indicates that the dialog flow encountered a question and answer node.
input-required Indicates that input is required from the user.
input-received Indicates that input was received from the user.
intent Indicates that an intent was set.
input-processed Indicates that the input received from the user has been processed.
question-router Indicates that the dialog flow encountered a question router node.
session-update Provides session variables that were updated.
data-required Indicates that the dialog flow encountered a data access node and requires data.
data-received Indicates that data was received.
transfer-initiated Indicates that an external actions node of type "Transfer" was encountered.
transfer-completed Indicates that a transfer action is completed.
component-invoked Indicates that a component was invoked.
component-returned Indicates that the dialog flow has returned after having invoked a component.
event Indicates that client application returned an event.
error Indicates that an error occurred in the dialog flow.
application-ended Indicates that the application has ended. Relates to an external actions node of type "End".

session-start

Sample payload: "session-start" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "8bbedbb2353cdc1d",
  "requestid": "2c06142d-71b7-4077-bf4b-e3892811b3d8",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "1",
  "events": [{
      "name": "session-start",
      "value": {
        "runtime": {
          "api": "v1"
        },
        "project": {
          "name": "TestMixClient",
          "namespace": "alex.smith@company.com",
          "deployed": "2020-12-10T19:52:25.224Z",
          "contextTag": "A1880_C1",
          "id": "8655"
        },
        "selector": {
          "channel": "default",
          "language": "en-US"
        },
        "version": {
          "asr": {
            "en_US": "2"
          },
          "nlu": {
            "en_US": "2"
          },
          "dlg": "2"
        },
        "user": {
          "systemID": null,
          "userChannelID": null,
          "userAuxiliaryID": null,
          "userGlobalID": null,
          "location": null
        },
        "timeout": 900
      }
    }
  ]
}

This event indicates the start of an interaction between a user and a dialog application.

Field Description
runtime Provides information about the runtime environment:
api Version of the DLGaaS API protocol
project Provides details about the project:
name Name of project
namespace Organization for this project
deployed Date and time that the project was deployed
contextTag Context tag of the application configuration used for this session
id ID of project
selector Channel and language used for this interaction:
channel Channel name
language Language for this interaction, identified by a language code
version Versions of the ASR, NLU, and dialog builds used for this session:
asr Language and version of the ASR build. This is the version of the build deployed in the Mix environment.
nlu Language and version of the NLU build. This is the version of the build deployed in the Mix environment.
dlg Version of the dialog build. This is the version of the build deployed in the Mix environment.
user userData values configured for the user; see userData schema in the Mix.dialog documentation for details
timeout The session timeout used. This can be either the default timeout value or a value specified in the StartRequest.

selector

Sample payload: "selector" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "752e56c6ded0f281",
  "requestid": "059f0fdc-bf24-4314-a00d-687ff65dacdd",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "3",
  "events": [{
      "name": "selector",
      "value": {
        "channel": "default",
        "language": "en-US"
      }
    }
  ]
}

This event provides the selector used for this interaction.

Field Description
channel Channel used for the interaction
language Language used in this interaction, identified by language code

decision

Sample payload: "decision" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "752e56c6ded0f281",
  "requestid": "059f0fdc-bf24-4314-a00d-687ff65dacdd",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "4",
  "events": [{
      "name": "decision",
      "value": {
        "next": "WelcomeMessage",
        "label": "default"
      }
    }
  ]
}

This event indicates that a decision was executed and specifies the next node in the dialog flow. This event relates to a decision node.

Field Description
next Name of the next node in the dialog flow
label Descriptive label for the decision

transition

Sample payload: "transition" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "97acfc8f98bc0de",
  "requestid": "cf26af47-a23c-4337-92fc-c51c19f21996",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "16",
  "events": [{
      "name": "transition",
      "value": {
        "from": {
          "name": "Get Intent",
          "uuid": "8e31e30b-bfcc-4878-b2a4-f2e1789d6f07"
        },
        "to": {
          "name": "Intent Mapper",
          "type": "intentmapper",
          "uuid": "6075a7ea-6171-43ec-8e0a-92da7b3e5ba5"
        }
      }
    }
  ]
}

This event represents a transition from a source node to a target node in the dialog flow.

Field Description
from Source node for this transition:
name Name of the source node
uuid UUID of the node
to Target node for this transition:
name Name of the target node
type Type of node; values include decision, playprompt, dataaccess, intentmapper, message, questionanswer, questionrouter, transfer, end
uuid UUID of the node

message

Sample payload: "message" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "2ef850949bb32b9b25744cc7d2f79091",
  "requestid": "a8f1df04-f804-9155-9ef9-5f1376c8b4d2",
  "sessionid": "e19ae4f8-f4c2-4f4a-b8db-401e30cbda69",
  "locale": "en-US",
  "clientData": {},
  "seqid": "6",
  "events": [{
      "name": "message",
      "value": {
        "nlg": [{
            "text": "Welcome to the Mix Bank Demo."
          }
        ],
        "visual": [{
            "text": "Welcome to the Mix Bank Demo.  Rich Text VA."
          }
        ],
        "audio": [{
            "text": "Welcome to the Mix Bank Demo. ",
            "audio": "en-US/prompts/default/default/welcome_to_mix_bank_demo_.wav?version=1.0_1614834167198"
          }
        ]
      }
    }
  ]
}

This event provides the contents of the message field returned to the client application.

See Message in the DLGaaS documentation for details.

qa-config

Sample payload: "qa-config" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "362ce333dfa6d3009c716ca40ca4cfd5",
  "requestid": "d69cb4e2-15c5-9b2b-9827-45bc6ad073e8",
  "sessionid": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13",
  "locale": "en-US",
  "clientData": {},
  "seqid": "20",
  "events": [{
      "name": "qa-config",
      "value": {
        "confirmationMode": "NEVER",
        "role": "validation",
        "confidence": {
          "low": "0.150",
          "high": "0.550"
        },
        "name": "COFFEE_SIZE",
        "entity": {
          "confirmationMode": "NEVER",
          "confidence": "1.000",
          "name": "COFFEE_SIZE",
          "value": "lg",
          "required": "false",
          "literal": "large",
          "status": "validated"
        }
      }
    }
  ]
}

This event indicates that the dialog flow encountered a question and answer node.

Field Description
confirmationMode Confirmation strategy for the question and answer node. Possible values are 'ALWAYS', 'IF_NECESSARY', or 'NEVER'.
role Specifies the current role of the question and answer node. Possible values are:
  • collection: No values are filled so the node needs to collect them
  • confirmation: A yes/no confirm question was asked (currently not supported)
  • validation: Input is currently being validated
confidence Low and high confidence thresholds used
name Name of the question and answer node
entity If the entity is collected, the following information is provided:
  • confirmationMode: Confirmation strategy used for the entity. Possible values are 'ALWAYS', 'IF_NECESSARY', or 'NEVER'.
  • confidence: Confidence level of the interpretation.
  • name: Name of the entity.
  • value: Value of the entity.
  • required: Indicates whether this entity must be collected.
  • literal: Exact words used by the user.
  • status: Indicates the status of collecting the entity. Possible values are empty, prefilled, collected, confirmed, validated.
Otherwise, the field is null. See Status field for details.

Status field

The following diagram shows how the status field value is determined:

Mix.dialog nodes

input-required

Sample payload: Simple "input-required" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "a9f419174dfe1e13d0fb70943e099892",
  "requestid": "be824d02-938e-946f-ad8f-6134f60c729a",
  "sessionid": "b15302ac-be35-4d9c-b593-3789b091e83f",
  "locale": "en-US",
  "clientData": {},
  "seqid": "11",
  "events": [{
      "name": "input-required",
      "value": {
        "data": {}
      }
    }
  ]
}

Sample payload: "input-required" event with data and interactive elements

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "752e56c6ded0f281",
  "requestid": "059f0fdc-bf24-4314-a00d-687ff65dacdd",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "11",
  "events": [{
      "name": "input-required",
      "value": {
        "data": {
          "date": "now"
        },
        "selectables": [{
            "description": "",
            "display_text": "Discover",
            "display_image_uri": "",
            "value": {
              "value": "discover",
              "id": "payee"
            }
          }, {
            "description": "",
            "display_text": "American Express",
            "display_image_uri": "",
            "value": {
              "value": "amex",
              "id": "payee"
            }
          }, {
            "description": "",
            "display_text": "Visa",
            "display_image_uri": "",
            "value": {
              "value": "visa",
              "id": "payee"
            }
          }
        ]
      }
    }
  ]
}

This event indicates that input is required from the user. This event will also include data and interactive elements configured in the question and answer node, if any.

Field Description
data Data sent to the client application. See Sending data in the DLGaaS documentation for details.
selectables Interactive elements configured in Mix.dialog for this question and answer node. See Interactive elements in the DLGaaS documentation for details.

input-received

Sample payload: "input-received" event with interpretation

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "data": {
    "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
    "traceid": "97acfc8f98bc0de",
    "requestid": "cf26af47-a23c-4337-92fc-c51c19f21996",
    "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
    "locale": "en-US",
    "clientData": {},
    "seqid": "13",
    "events": [{
        "name": "input-received",
        "value": {
          "userText": "I want an espresso",
          "interpretation": {
            "isSuccess": "true",
            "confidence": "1.0",
            "inputmode": "text",
            "utterance": "I want an espresso",
            "data": {
              "COFFEE_TYPE": "espresso",
              "INTENT": "ORDER_COFFEE"
            },
            "slot_literals": {
              "COFFEE_TYPE": "espresso",
              "INTENT": "I want an espresso"
            },
            "slot_confidences": {
              "COFFEE_TYPE": "0.9515329",
              "INTENT": "1.0"
            }
          }
        }
      }
    ]
  }
}  

Sample payload: "input-received" event with selected item

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "89eba3e69ec15c0c12ab90b1901149c0",
  "requestid": "efc93c71-5046-95e0-85a9-75592a4d23f4",
  "sessionid": "e19ae4f8-f4c2-4f4a-b8db-401e30cbda69",
  "locale": "en-US",
  "clientData": {},
  "seqid": "24",
  "events": [{
      "name": "input-received",
      "value": {
        "selectedItem": {
          "id": "payee",
          "value": "discover"
        }
      }
    }
  ]
}

This event indicates that input was received from the user. This can either be an interpretation or a selected interactive element:

Field Description
userText Text collected from end user, if applicable. Only applies for interpretations.
interpretation An interpretation of the input; see Interpretresult in the DLGaaS documentation for a description of the fields.
selectedItem The item selected by the user. See Interactive elements in the DLGaaS documentation for details.

intent

Sample payload: "intent" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "97acfc8f98bc0de",
  "requestid": "cf26af47-a23c-4337-92fc-c51c19f21996",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "14",
  "events": [{
      "name": "intent",
      "value": {
        "value": "ORDER_COFFEE"
      }
    }
  ]
}

This event indicates that an intent was set.

Field Description
value Name of the intent that was set

input-processed

Sample payload: "input-processed" event, with result "collected"

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "97acfc8f98bc0de",
  "requestid": "cf26af47-a23c-4337-92fc-c51c19f21996",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "15",
  "events": [{
      "name": "input-processed",
      "value": {
        "result": "collected",
        "counters": {
          "nomatch": 0,
          "help": 0,
          "repeat": 0,
          "invalidRecoOption": 0,
          "noToConfirm": 0,
          "try": 0,
          "turn": 1,
          "noinput": 0
        }
      }
    }
  ]
}

Sample payload: "input-processed" event, with result "maxnoinputs"

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "c044de1e2c3ff3f58f5d4b43bdd7d077",
  "requestid": "b0c37992-8b68-9d65-9c07-c3b7d1267a88",
  "sessionid": "cad5b7e8-41fd-488e-a169-0cdd2017cad2",
  "locale": "en-US",
  "clientData": {},
  "seqid": "20",
  "events": [{
      "name": "input-processed",
      "value": {
        "result": "maxnoinputs",
        "counters": {
          "nomatch": 0,
          "help": 0,
          "repeat": 0,
          "invalidRecoOption": 0,
          "noToConfirm": 0,
          "try": 0,
          "turn": 2,
          "noinput": 2
        }
      }
    }
  ]
}

This event indicates that the input received from the user has been processed.

Field Description
result Indicate the results of the processing operation. Possible values are collected, maxnomatches, maxnoinputs, maxretries, maxturns, nomatch, noinput
counters Provides the number of times the following events occurred:
nomatch Number of times the user's input could not be matched
help Number of times the user asked for help in an interaction
repeat Number of times the user asked the dialog to repeat the question
invalidRecoOption Number of times the response received was not what was expected (for example, a response was received for an entity called DATE but the response was not a date)
noToConfirm Number of times the user said "no" in the confirmation phase
try Number of recognition steps (recognition request and result) to get a piece of information
turn Number of turns (that is, number of recognition and confirmation steps) in the interaction
noinput Number of times the user did not provide an input

question-router

Sample payload: "question-router" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "97acfc8f98bc0de",
  "requestid": "cf26af47-a23c-4337-92fc-c51c19f21996",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "18",
  "events": [{
      "name": "question-router",
      "value": {
        "belief": [{
            "confirmationMode": null,
            "inputMode": null,
            "confidence": "0.952",
            "name": "COFFEE_TYPE",
            "value": "espresso",
            "required": "false",
            "literal": "espresso",
            "status": "filled"
          }, {
            "confirmationMode": null,
            "inputMode": null,
            "confidence": null,
            "name": "COFFEE_SIZE",
            "value": null,
            "required": null,
            "literal": null,
            "status": null
          }
        ]
      }
    }
  ]
}

This event indicates that the dialog flow encountered a question router node and provides the information that has been accumulated so far about the entities to collect.

Field Description
belief For every entity that needs to be collected, this field provides the information that has been accumulated so far
confirmationMode Confirmation strategy used. Possible values are 'ALWAYS', 'IF_NECESSARY', or 'NEVER'.
inputMode Input mode; for example: voice, select, dtmf, text
confidence Confidence level of the interpretation
name Name of the entity
value Value of the entity
required Indicates whether this entity must be collected
literal Exact words used by the user
status Indicates the status of collecting the entity. Possible values are empty, prefilled, collected, confirmed, validated. See Status field for details.

session-update

Sample payload: "session-update" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "db3f1849cc4471cc11d8f509df07b715",
  "requestid": "e4e5563b-ca9a-933e-a997-a7a50805fbe3",
  "sessionid": "16d80b2e-8e18-4f1b-a146-90b02c3f7272",
  "locale": "en-US",
  "clientData": {},
  "seqid": "16",
  "events": [{
      "name": "session-update",
      "value": {
        "data": {
          "user_name": "****"
        }
      }
    }
  ]
}

This event provides session variables that were updated with the UpdateRequest method.

Field Description
data Session variables that were updated

data-required

Sample payload: "data-required" event, client-side integration

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "b43ba10286fed5486e875612ee55db29",
  "requestid": "64d3d58e-9052-9ddd-ba69-8588c040983c",
  "sessionid": "8a7c5d63-6983-4c5c-835a-0ffcf6b27e33",
  "locale": "en-US",
  "clientData": {},
  "seqid": "6",
  "events": [{
      "name": "data-required",
      "value": {
        "id": "GetUserDetails",
        "type": "external",
        "data": {
          "userID": "18470293"
        }
      }
    }
  ]
}

Sample payload: "data-required" event, server-side integration

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "b8fb1468bb4aa1be997dca245c4d1881",
  "requestid": "4894fcdf-8ebf-9750-a96b-5c29099f2a8f",
  "sessionid": "734339bb-c1a3-4fee-a873-b6e3122b2ec7",
  "locale": "en-US",
  "clientData": {},
  "seqid": "35",
  "events": [{
      "name": "data-required",
      "value": {
        "id": "GetCoffeePrice",
        "type": "internal",
        "data": {
          "size": "large",
          "type": "latte"
        },
        "timeout": {
          "connect": 0,
          "request": 30000
        },
        "endpoint": {
          "type": "REST",
          "method": "POST",
          "uri": "https://coffee.app.com:443/dataaccess-examples/coffee/price",
          "headers": ["API_KEY"]
        }
      }
    }
  ]
}

This event indicates that the dialog flow encountered a data access node and requires data.

The contents of the event depend on the integration type:

Field Description
id Name of the data access node that requested the data
type Integration type:
  • external: Client-side integration
  • internal: Server-side integration
data Data to be sent as configured in the data access node
timeout Timeout values specified in the data access node; applies to server-side integrations only
connect connect timeout value, in milliseconds
request request timeout value, in milliseconds
endpoint Applies to server-side integrations only
type Always REST
method GET or POST
uri The final URI called
headers List of header names sent, if any

data-received

Sample payload: "data-received" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "ee544a6130c8be935c0e461fb0d51394",
  "requestid": "ff19f364-77c0-94a0-aa27-ea03f1991acf",
  "sessionid": "8e539261-9d69-49f6-9c80-777b53e5fe95",
  "locale": "en-US",
  "clientData": {},
  "seqid": "8",
  "events": [{
      "name": "data-received",
      "value": {
        "id": "SessionData",
        "data": {
          "returnCode": "0",
          "preferred_coffee": "espresso ",
          "user_name": "****"
        },
        "returnCode": "0"
      }
    }
  ]
}

This event indicates that the requested data has been received.

Field Description
id Name of the data access node that requested the data
data Data requested
returnCode Variable indicating whether the query was successful. A returnCode of "0" indicates that the transaction was successful.
returnMessage Return message (optional)
duration Duration of the request in milliseconds (applies to server-side integrations only)

transfer-initiated

Sample payload: "transfer-initiated" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "e130302b942d312fdae6344756bedc62",
  "requestid": "f9681763-99dc-92e7-91cd-4289371c6eda",
  "sessionid": "cc69a575-b4f1-4384-a143-98ed7136af81",
  "locale": "en-US",
  "clientData": {},
  "seqid": "31",
  "events": [{
      "name": "transfer-initiated",
      "value": {
        "id": "TransferToAgent",
        "data": {
          "user_name": "****",
          "userID": "****"
        }
      }
    }
  ]
}

This event indicates that an external actions node of type "Transfer" was encountered. See Transfer actions in the DLGaaS documentation for details.

Field Description
id Name of the external actions node of type "Transfer"
data Data configured in the transfer node and sent to the client application

transfer-completed

Sample payload: "transfer-completed" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "a28a6b7fceeb088241c1abb08f21c076",
  "requestid": "8e04269a-bf84-9829-bbe5-9249ebc49507",
  "sessionid": "cc69a575-b4f1-4384-a143-98ed7136af81",
  "locale": "en-US",
  "clientData": {},
  "seqid": "33",
  "events": [{
      "name": "transfer-completed",
      "value": {
        "returnCode": "0",
        "data": {
          "status": "Call successfully completed"
        }
      }
    }
  ]
}

This event indicates that a transfer action is completed. See Transfer actions in the DLGaaS documentation for details.

Field Description
data Any data provided in the requested_data field of the ExecuteRequestPayload
returnCode Return code sent by the client application to indicate the status of the transfer
returnMessage Return message sent by the client application (optional)

component-invoked

Sample payload: "component-invoked" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "5cd7a032b6fbcce60561d3ba1359c4f",
  "requestid": "a4d34efa-3884-95e2-ad85-c2f708a11318",
  "sessionid": "e19ae4f8-f4c2-4f4a-b8db-401e30cbda69",
  "locale": "en-US",
  "clientData": {},
  "seqid": "64",
  "events": [{
      "name": "component-invoked",
      "value": {
        "name": "Goodbye"
      }
    }
  ]
}

This event indicates that a component call node was invoked. See Set up a component call node in the Mix.dialog documentation for details.

Field Description
value Name of the component that was invoked

component-returned

Sample payload: "component-returned" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "5cd7a032b6fbcce60561d3ba1359c4f",
  "requestid": "a4d34efa-3884-95e2-ad85-c2f708a11318",
  "sessionid": "e19ae4f8-f4c2-4f4a-b8db-401e30cbda69",
  "locale": "en-US",
  "clientData": {},
  "seqid": "71",
  "events": [{
      "name": "component-returned",
      "value": {}
    }
  ]
}

This event indicates that the dialog flow has returned after having invoked a component.

Field Description
value Always empty

event

Sample payload: "event" event, sent by the client application

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "441249de36ff9f682f69a63064e0b125",
  "requestid": "64f78612-552a-9351-bbd9-b89d98c87118",
  "sessionid": "8baf36f2-886d-468d-9e4c-057fdbde66e4",
  "locale": "en-US",
  "clientData": {},
  "seqid": "17",
  "events": [{
      "name": "event",
      "value": {
        "name": "NO_INPUT",
        "message": ""
      }
    }
  ]
}

Sample payload: "event" event, sent by the client application

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "c044de1e2c3ff3f58f5d4b43bdd7d077",
  "requestid": "b0c37992-8b68-9d65-9c07-c3b7d1267a88",
  "sessionid": "cad5b7e8-41fd-488e-a169-0cdd2017cad2",
  "locale": "en-US",
  "clientData": {},
  "seqid": "19",
  "events": [{
      "name": "event",
      "value": {
        "name": "maxnoinputs",
        "message": ""
      }
    }
  ]
}

This event indicates that an event was received. This can be an event sent by the client application or an event that occurred during the dialog flow.

Field Description
name Event type.
For events sent by the client application, this can be, for example: 'NO_INPUT', 'NO_MATCH', 'HANGUP', 'CUSTOM', 'SUCCESS'. See Dialog event in the DLGaaS documentation for details.
message Optional message associated with the event

error

Sample payload: "error" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "c044de1e2c3ff3f58f5d4b43bdd7d077",
  "requestid": "b0c37992-8b68-9d65-9c07-c3b7d1267a88",
  "sessionid": "cad5b7e8-41fd-488e-a169-0cdd2017cad2",
  "locale": "en-US",
  "clientData": {},
  "seqid": "23",
  "events": [{
      "name": "error",
      "value": {
        "name": "error.stateengine",
        "message": "Make sure an Event Handler has been configured. No transition defined for event[event.nuance.dialog.odm.maxnoinputs] in node[event]"
      }
    }
  ]
}

This event indicates that an error occurred in the dialog flow.

Field Description
name Name of the error
message Message associated with the error

application-ended

Sample payload: "application-ended" event

"data": {
  "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
  "traceid": "5e49e59ead2572e7",
  "requestid": "fbc5586d-178b-465c-918c-aed78d4bb264",
  "sessionid": "0c234bb6-e2c4-4198-a7f7-53020b02d734",
  "locale": "en-US",
  "clientData": {},
  "seqid": "36",
  "events": [{
      "name": "application-ended",
      "value": {
        "id": "End dialog",
        "language": "en-US"
      }
    }
  ]
}

This event indicates that the application has ended. This can occur in the following situations:

Field Description
id Name of the end node
language Language used in this interaction
data Variables and entities to be sent to an external system, if applicable

Dialog values possibly masked

When the suppress_log_user_data field in the StartRequest is set to true, entities and variables marked as sensitive in Mix.dialog are masked in application logs.

NLU payload

The x-nuance-nluaas-interpretation payload provides an overview of the interactions that occur in an NLUaaS interpretation session.

nluaas-interpretation payload

Sample payload: Single-intent interpretation

"data": {
  "dataContentType": "application/x-nuance-nluaas-interpretation.v1+json",
  "locale": "en-US",
  "traceid": "441249de36ff9f682f69a63064e0b125",
  "request": {
    "resources": [],
    "clientData": {
      "x-nuance-dialog-session-id": "8baf36f2-886d-468d-9e4c-057fdbde66e4",
      "x-nuance-dialog-seqid": "15"
    },
    "parameters": {
      "postProcessingScriptParameters": {},
      "interpretationResultType": "SINGLE_INTENT",
      "interpretationInputLoggingMode": "PLAINTEXT",
      "maxInterpretations": 0
    },
    "model": {
      "type": "SEMANTIC_MODEL",
      "uri": "urn:nuance:mix/eng-USA/TestBotInterface/mix.nlu",
      "requestTimeoutMs": 0
    },
    "userId": "",
    "input": {
      "text": "I want a double espresso",
      "inputUnion": "text"
    }
  },
  "processingTime": {
    "startTime": "2021-03-22T15:13:00.285Z",
    "durationMs": 127
  },
  "response": {
    "status": {
      "code": 200,
      "message": "OK",
      "details": ""
    },
    "result": {
      "interpretations": [{
          "singleIntentInterpretation": {
            "entities": {
              "COFFEE_TYPE": {
                "entities": [{
                    "entities": {},
                    "textRange": {
                      "startIndex": 16,
                      "endIndex": 24
                    },
                    "confidence": 1,
                    "origin": "GRAMMAR",
                    "literal": "espresso",
                    "sensitive": false,
                    "audioRange": null,
                    "stringValue": "espresso",
                    "valueUnion": "stringValue"
                  }
                ]
              },
              "COFFEE_SIZE": {
                "entities": [{
                    "entities": {},
                    "textRange": {
                      "startIndex": 9,
                      "endIndex": 15
                    },
                    "confidence": 1,
                    "origin": "GRAMMAR",
                    "literal": "double",
                    "sensitive": false,
                    "audioRange": null,
                    "stringValue": "lg",
                    "valueUnion": "stringValue"
                  }
                ]
              }
            },
            "intent": "ORDER_COFFEE",
            "confidence": 1,
            "origin": "GRAMMAR"
          },
          "interpretationUnion": "singleIntentInterpretation"
        }
      ],
      "literal": "I want a double espresso",
      "sensitive": false
    }
  }
}

Current version: v1

Messages with the application/x-nuance-nluaas-interpretation.v1+json dataContentType contain no additional service-specific fields beyond those described in Data field. Note, however, that for NLUaaS and ASRaaS the clientData field is part of the request object.

NLUaaS request and response objects

The request object describes the input, as well as the model, used for the interpretation. Parameters, extra resources, client data, and user ID may also be included to customize the interpretation. See InterpretRequest for details.

The input to interpret is either:

"input":
    {
      "asrResult":
      {
        "type_url": "type.googleapis.com/nuance.asr.v1.Result",
        "value":
        {
          "type": "Buffer",
          "data": [16, 210, ..., 46, 48]
        }
      },
      "inputUnion": "asrResult"
    }

The response object, in addition to providing a status message, describes the interpretation result, both the input (literal) and the candidate interpretations of the input. For more example payloads, see InterpretResult.

Notice that the first child under interpretations specifies the result type (type of interpretation). For example:

"result": {
   "interpretations": [{
       "singleIntentInterpretation": {
         ...
       }
     },
     ...

There are two interpretation result types:

NLU values possibly masked

When the interpretation_input_logging_mode field in the InterpretRequest is set to SUPPRESSED, input is masked or redacted as described below. See InterpretationParameters for more information.

Values in the data field that may be masked include:

For example:


...
"entities": {
  "COFFEE_TYPE": {
    "entities": [{
        "entities": {},
        "textRange": {
          "startIndex": 16,
          "endIndex": 24
        },
        "confidence": 1,
        "origin": "GRAMMAR",
        "literal": "****redacted****",
        "sensitive": true,
        "audioRange": null,
        "stringValue": "****redacted****",
        "valueUnion": "stringValue"
      }
    ]
  },
  ...

For more information, see InterpretResult.

Also, entities marked as sensitive in Mix.nlu or Mix.dialog are masked in application logs (including literals and values assigned to those entities).

TTS payload

The x-nuance-tts-callsummary payload provides an overview of the interactions that occur in a TTSaaS synthesis session.

tts-callsummary payload

Sample payload: Synthesizer/Synthesize

"data":
{
  "dataContentType": "application/x-nuance-tts-callsummary.v3+json",
  "clientName": "default",
  "traceId": "7c5bcda51d0b9cc63eabcf8f662f82df",
  "sessionId": "8070e584-de4d-4daf-ac37-d9947de5eff1",
  "userId": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "processingTime":
  {
    "startTime": "2021-03-15T18:13:19.650Z",
    "firstAudioBufferTime": "2021-03-15T18:13:19.872Z",
    "durationMs": 363
  },
  "request":
  {
    "resources": []
  },
  "clientData":
  {
    "name": "Alex Smith",
    "x-nuance-dialog-session-id": "8070e584-de4d-4daf-ac37-d9947de5eff1"
  },
  "response":
  {
    "events": [
      // List of events
    ],
    "status":
    {
      "code": 200,
      "message": "OK",
      "details": ""
    }
  }
}

Current version: v3

Nuance Vocalizer for Cloud (NVC) offers two types of synthesis response. By default, the synthesized voice is streamed back to the application (Synthesizer/Synthesize), but a unary (non-streamed, single package) response may be requested instead (Synthesizer/UnarySynthesize). The same payload is returned with the exception of the source field. For more information on streamed versus unary responses, see the TTSaaS gRPC API documentation.

In addition to the standard fields described in Data field, messages with the application/x-nuance-tts-callsummary.v3+json dataContentType include service-specific fields as detailed below.

Request object

Sample request with synthesis resources: ActivePrompt database, ruleset, and user dictionary

"request": {
    "resources": [{
            "uri": "urn:nuance-mix:tag:model/MY_APDB/mix.tts?=language=en-US",
            "type": "application/x-vocalizer-activeprompt-db"
        }, {
            "uri": "urn:nuance-mix:tag:model/MY_RST/mix.tts?=language=en-US",
            "type": "application/x-vocalizer-rettt+text"
        }, {
            "uri": "urn:nuance-mix:tag:model/MY_DCB/mix.tts?=language=en-US",
            "type": "application/edct-bin-dictionary"
        }
    ]
}

The request object describes the synthesis resources used to tune the synthesized audio, as provided in the <../tts-grpc/v1/#input" target="_blank">Input message of a SynthesisRequest. Default is blank (empty array).

Resources may include:

Response object

Sample response with NVOCcntv, NVOCcntg, NVOCinpt, and NVOCsynd events

"response":
{
  "events": [
    {
      "EVENT": "NVOCcntv",
      "VOICE_VOP": "Allison_full_vssq5f22",
      "CHARS": 50,
      "TTSTIME": "2021/03/15 14:39:01.660 UTC",
      "TTSAASTIME": "2021-03-15T14:39:01.660Z",
      "VOIC": "Allison",
      "VMDL": "full_vssq5f22",
      "LOCALE": "en-US",
      "DURS": 3347.39
    },
    {
      "EVENT": "NVOCcntg",
      "CHARS": 50,
      "TTSTIME": "2021/03/15 14:39:01.660 UTC",
      "TTSAASTIME": "2021-03-15T14:39:01.660Z"
    },
    {
      "EVENT": "NVOCinpt",
      "MIME": "text/plain;charset=utf-8",
      "TXSZ": 62,
      "TEXT": "Hello and welcome to the coffee app. What can I get you today?",
      "TTSTIME": "2021/03/15 14:39:01.660 UTC",
      "TTSAASTIME": "2021-03-15T14:39:01.660Z"
    },
    {
      "EVENT": "NVOCsynd",
      "INPT": 62,
      "DURS": 3347,
      "RSTT": "ok",
      "TTSTIME": "2021/03/15 14:39:01.660 UTC",
      "TTSAASTIME": "2021-03-15T14:39:01.660Z"
    }
  ],
  "status":
  {
    "code": 200,
    "message": "OK",
    "details": ""
  }
}

The response object, in addition to providing a status message, provides a list (array) of events. For more information about the synthesis response, see SynthesisResponse.

For a successful response, response.events will always contain:

Each event includes the following two fields:

NVOCcntv event

{
    "EVENT": "NVOCcntv",
    "VOICE_VOP": "Ava-Mls_xpremium",
    "CHARS": 36,
    "TTSTIME": "2020/09/15 10:10:20.504 UTC",
    "TTSAASTIME": "2020-09-15T10:10:20.504Z",
    "VOIC": "Ava-Mls",
    "VMDL": "xpremium",
    "LOCALE": "en-US",
    "DURS": 3490.2
}, {
    "EVENT": "NVOCcntv",
    "VOICE_VOP": "Paulina-Ml_xpremium-high",
    "CHARS": 18,
    "TTSTIME": "2020/09/15 10:10:20.504 UTC",
    "TTSAASTIME": "2020-09-15T10:10:20.504Z",
    "VOIC": "Paulina-Ml",
    "VMDL": "xpremium-high",
    "LOCALE": "es-MX",
    "DURS": 2069.93
}

The NVOCcntv event indicates the amount of Unicode input characters processed for a specific voice operating point in a single speak request. One event is logged per voice used in the request.

Field Description
EVENT NVOCcntv
VOICE_VOP Name of the voice and its operating point
CHARS Number of Unicode characters processed
VOIC Voice name
VMDL Voice model. For example, xpremium (standard) xpremium-high (enhanced). Other values will be returned for legacy voices (full_vssqf22 for the Allison voice, for example).
LOCALE Locale ID
DURS Output duration in milliseconds

NVOCcntg event

{
    "EVENT": "NVOCcntg",
    "CHARS": 54,
    "TTSTIME": "2020/09/15 10:10:20.505 UTC",
    "TTSAASTIME": "2020-09-15T10:10:20.505Z"
}

The NVOCcntg event indicates the total Unicode character count processed in a single speak request.

Field Description
EVENT NVOCcntg
CHARS Number of Unicode characters processed

NVOCinpt event

{
    "EVENT": "NVOCinpt",
    "MIME": "application/synthesis+ssml",
    "TXSZ": 373,
    "TEXT": "<?xml version\\=\"1.0\"?><speak xmlns\\=\"http://www.w3.org/2001/10/synthesis\" 
    xml:lang\\=\"en-US\" version\\=\"1.0\" xmlns:xsi\\=\"http://www.w3.org/2001/XMLSchema-instance\"
    xsi:schemaLocation\\=\"http://www.w3.org/2001/10/synthesis http://www.w3.org/TR/speech-synthesis/synthesis.xsd\">English text here. 
    <voice xml:lang\\=\"es-MX\">Texto en español aquí</voice> and now back to English.</speak>",
    "TTSTIME": "2020/09/15 10:10:20.505 UTC",
    "TTSAASTIME": "2020-09-15T10:10:20.505Z"
}

The NVOCinpt event provides the input text for the speak request and is logged at the end of each speak request. If an error occurred or the synthesis request was stopped, NVOCinpt reports the information obtained prior to the error or interruption.

Field Description
EVENT NVOCinpt
MIME MIME content type for the input text
TXSZ Input size in bytes, the raw input prior to any processing
TEXT Input text. Omitted when the suppress_input field is true in the SynthesisRequest (default is false).

NVOCsynd event

{
    "EVENT": "NVOCsynd",
    "INPT": 373,
    "DURS": 5560,
    "RSTT": "ok",
    "TTSTIME": "2020/09/15 10:10:20.506 UTC",
    "TTSAASTIME": "2020-09-15T10:10:20.506Z"
}

The NVOCsynd event is logged at the end of synthesis. If an error occurred or the synthesis request was stopped, NVOCsynd reports the information obtained prior to the error or interruption.

Field Description
EVENT NVOCsynd
INPT Input size in bytes; the raw input prior to any processing
DURS Output duration in milliseconds
RSTT Result status:
  • "error" if an error occurred
  • "ok" if synthesis succeeded
  • "stop" if the application stopped the synthesis operation

NVOCifst event

{
    "EVENT": "NVOCifst",
    "PROP": "cache.timeoutOpen=5000;inet.timeoutDownload=30000;inet.timeoutIO=30000;inet.timeoutOpen=30000",
    "TTSTIME": "2020/11/19 16:35:00.273 UTC",
    "URI": "urn:nuance-mix:tag:model/MY_APDB/mix.tts?=language=en-US",
    "TTSAASTIME": "2020-11-19T18:18:00.891Z"
}

The NVOCifst event indicates an Internet fetch has started.

Field Description
EVENT NVOCifst
PROP Semicolon-separated list of Internet fetch properties, where each property is of the form property=value
URI URI being fetched; can be a relative URI

NVOCifnd event

{
    "EVENT": "NVOCifnd",
    "FRST": "SUCCESS",
    "TTSTIME": "2020/11/19 16:35:00.327 UTC",
    "DSRC": "http",
    "SIZE": "12652",
    "MIME": "application/octet-stream",
    "URI": "urn:nuance-mix:tag:model/MY_APDB/mix.tts?=language=en-US",
    "TTSAASTIME": "2020-11-19T18:18:00.891Z"
}

The NVOCifnd indicates an Internet fetch has ended.

Field Description
EVENT NVOCifnd
FRST Fetch result: SUCCESS if the fetch succeeded; otherwise a brief error description
DSRC Data source:
  • "cache" if it was loaded from a cache
  • "file" if it was a local file
  • "http" if it was fetched from a web server
  • "validated" if it was loaded from a cache, but only after validating the cached copy with the web server
  • empty string if the fetch failed
SIZE Size in bytes. 0 if the fetch failed.
MIME MIME content type. Empty string if the fetch failed.
URI URI that was fetched. If the fetch was successful, this will be an absolute URI.

NVOCrlst event

The NVOCrlst event is written when a ruleset is used during synthesis.

Field Description
EVENT NVOCrlst
RULSET_NAME URI that was fetched for the ruleset. Example:
urn:nuance-mix:tag:model/ruleset/mix.tts?=language=en-US

NVOCdict event

The NVOCdict event is written when a user dictionary is used during synthesis.

Field Description
EVENT NVOCdict
URI URI that was fetched for the user dictionary. Example:
urn:nuance-mix:tag:model/user_dictionary/mix.tts?=language=en-US

NVOCactp event

The NVOCactp event is written when an ActivePrompt is used during synthesis. These events are written in the order in which the ActivePrompts are matched (highest to lowest precedence across the input text), not the order in which they appear in the output audio stream.

Field Description
EVENT NVOCactp
APID ActivePrompt ID of the form domain::prompt

NVOCadon event

The NVOCadon event is written when a system-level dictionary is used during synthesis.

Field Description
EVENT NVOCadon
ADDON_NAME URI that was fetched for the add-on. Example:
urn:nuance-mix:tag:model/system_dictionary/mix.tts?=language=en-US

TTS values possibly masked

When the suppress_input field in the SynthesisRequest is set to true, the following fields will be masked:

See the TTSaaS gRPC API documentation for more information.

ASR payloads

A typical ASR interaction includes a subset or all of the following events:

The code samples in this section provide all the event logs generated for recognizing the following utterance:

"My credit card number is 3141592653589793"

asr-recognitioninitmessage payload

Sample payload

"data": {
  "request": {
    "recognitionInitMessage": {
      "parameters": {
        "language": "eng-USA",
        "topic": "NTEFAST",
        "audioFormat": {
          "pcm": {
            "sampleRateHz": 16000
          }
        },
        "recognitionFlags": {
          "autoPunctuate": false,
          "filterProfanity": false,
          "suppressCallRecording": false
        }
      },
      "clientData": {
        "applicationName": "appID:test_user_nuance_com_test3:geo:us:clientName:default"
      }
    }
  },
  "asrSessionid": "31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c",
  "dataContentType": "application/x-nuance-asr-recognitioninitmessage.v2+json",
  "requestid": "3136437b-f95b-4a37-8c18-5c6ee836e212",
  "audioUrn": "urn:nuance-mix:log:service:audio/asraas/31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c"
}

Current version: v2

The application/x-nuance-asr-recognitioninitmessage payloads provide the content of the RecognitionInitMessage sent in the ASRaaS RecognitionRequest. This message specifies the parameters and resources for recognition.

In addition to the standard fields described in Data field, messages with the application/x-nuance-asr-recognitioninitmessage dataContentType include the following fields:

Field Description
request Contains the RecognitionInitMessage payload
asrSessionid Identifier of the current ASR session
audioUrn URN of the ASR resource. This field is helpful when downloading audio files of the utterances that were sent for recognition to ASRaaS. See AFSS API for details.

asr-statusresponse payload

Sample payload

"data": {
  "response": {
    "status": {
      "code": 100,
      "message": "Continue",
      "details": "recognition started on audio/l16;rate=16000 stream"
    }
  },
  "asrSessionid": "31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c",
  "dataContentType": "application/x-nuance-asr-statusresponse.v2+json",
  "requestid": "3136437b-f95b-4a37-8c18-5c6ee836e212"
}

Current version: v2

The application/x-nuance-asr-statusresponse payloads provide the content of the 100 Continue status message that is sent back to the client to indicate that recognition parameters and resources were accepted and successfully configured.

This event is usually sent after the recognitionInitMessage event is received and also indicates that recognition has started.

In addition to the standard fields described in Data field, messages with the application/x-nuance-asr-statusresponse dataContentType include the following fields:

Field Description
response Contains the 100 Continue status code. See Status messages and codes in the ASRaaS documentation for details.
asrSessionid Identifier of the current ASR session

asr-startspeechresponse payload

Sample payload

"data": {
  "response": {
    "startOfSpeech": {
      "firstAudioToStartOfSpeechMs": 360
    }
  },
  "asrSessionid": "31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c",
  "dataContentType": "application/x-nuance-asr-startspeechresponse.v2+json",
  "requestid": "3136437b-f95b-4a37-8c18-5c6ee836e212"
}

Current version: v2

The application/x-nuance-asr-startspeechresponse payloads indicate the beginning of audio as detected by Krypton. This event is triggered only when the recognition parameter enableUtteranceDetection is true (the default).

In addition to the standard fields described in Data field, messages with the application/x-nuance-asr-startspeechresponse dataContentType include the following fields:

Field Description
response Contains the StartOfSpeech payload
asrSessionid Identifier of the current ASR session

asr-partialresultresponse payload

Sample payload

"data":
{
  "response":
  {
    "result":
    {
      "resultType": 1,
      "absStartMs": 0,
      "absEndMs": 1290,
      "dataPack":
      {
        "language": "ara-XWW",
        "topic": "GEN",
        "version": "4.2.0"
      },
      "hypotheses": [
        {
          "rejected": false,
          "formattedText": "كل سنه وانت طيب",
          "minimallyFormattedText": "كل سنه وانت طيب",
          "words": [
            {
              "text": "كل",
              "startMs": 80,
              "endMs": 240,
              "silenceAfterWordMs": 0
            },
            {
              "text": "سنه",
              "startMs": 240,
              "endMs": 640,
              "silenceAfterWordMs": 0
            },
            {
              "text": "وانت",
              "startMs": 640,
              "endMs": 860,
              "silenceAfterWordMs": 0
            },
            {
              "text": "طيب",
              "startMs": 860,
              "endMs": 1200,
              "silenceAfterWordMs": 0
            }
          ]
        }
      ]
    }
  },
  "asrSessionid": "6cdfce4b-76c6-41c1-9dad-76500323ffcb",
  "dataContentType": "application/x-nuance-asr-partialresultresponse.v2+json",
  "requestid": "3136437b-f95b-4a37-8c18-5c6ee836e212"
}

Current version: v2

The application/x-nuance-asr-partialresultresponse payloads provide the partial recognition results.

In addition to the standard fields described in Data field, messages with the application/x-nuance-asr-partialresultresponse dataContentType include the following fields:

Field Description
response Contains the Result payload for the partial result
redactedReason If content was redacted, provides the reason. For example, the "suppressCallRecording" value indicates that the suppress_call_recording RecognitionFlags field was set to true. See ASR values possibly masked for details.
asrSessionid Identifier of the current ASR session

asr-finalresultresponse payload

Sample payload: Example 1

"data": {
  "response": {
    "result": {
      "resultType": 0,
      "absStartMs": 360,
      "absEndMs": 9820,
      "utteranceInfo": {
        "durationMs": 9460,
        "clippingDurationMs": 0,
        "droppedSpeechPackets": 0,
        "dsp": {
          "snrEstimate": 27,
          "level": 16897,
          "stereo": false,
          "initialSilenceMs": 380,
          "initialEnergy": -70.5528,
          "finalEnergy": -63.327,
          "meanEnergy": 151.079
        }
      },
      "hypotheses": [{
          "rejected": false,
          "formattedText": "My credit card number is 3141592653589793",
          "minimallyFormattedText": "My credit card number is three one four one five nine two six five three five eight nine seven nine three",
          "averageConfidence": 489,
          "words": [{
              "text": "My",
              "confidence": 500,
              "startMs": 380,
              "endMs": 500,
              "silenceAfterWordMs": 0
            }, {
              "text": "credit",
              "confidence": 500,
              "startMs": 500,
              "endMs": 960,
              "silenceAfterWordMs": 0
            }, {
              "text": "card",
              "confidence": 500,
              "startMs": 960,
              "endMs": 1200,
              "silenceAfterWordMs": 0
            }, {
              "text": "number",
              "confidence": 500,
              "startMs": 1200,
              "endMs": 1520,
              "silenceAfterWordMs": 0
            }, {
              "text": "is",
              "confidence": 500,
              "startMs": 1520,
              "endMs": 1740,
              "silenceAfterWordMs": 600
            }, {
              "text": "3141592653589793",
              "confidence": 367,
              "startMs": 2340,
              "endMs": 8820,
              "silenceAfterWordMs": 0
            }
          ]
        }
      ]
    }
  },
  "asrSessionid": "31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c",
  "dataContentType": "application/x-nuance-asr-finalresultresponse.v2+json",
  "requestid": "3136437b-f95b-4a37-8c18-5c6ee836e212"
}

Sample payload: Example 2-Call recording suppressed

"data":
{
  "response":
  {
    "result":
    {
      "resultType": 0,
      "absStartMs": 980,
      "absEndMs": 8760,
      "utteranceInfo":
      {
        "durationMs": 7780,
        "clippingDurationMs": 0,
        "droppedSpeechPackets": 0,
        "dsp":
        {
          "snrEstimateDb": 20,
          "level": 20481,
          "numChannels": 1,
          "initialSilenceMs": 220,
          "initialEnergy": -60.0424,
          "finalEnergy": -51.0569,
          "meanEnergy": 118.718
        }
      },
      "dataPack":
      {
        "language": "eng-GBR",
        "topic": "GEN",
        "version": "4.1.0",
        "id": "GMT20200227064218"
      },
      "hypotheses": [],
      "redactedReason": "suppressCallRecording"
    }
  },
  "asrSessionid": "a3134271-8d79-450d-9b2e-9bd53e196056",
  "dataContentType": "application/x-nuance-asr-finalresultresponse.v2+json",
  "requestid": "3136437b-f95b-4a37-8c18-5c6ee836e212"
}

Current version: v2

The application/x-nuance-asr-finalresultresponse payloads provide the final recognition result.

In addition to the standard fields described in Data field, messages with the application/x-nuance-asr-finalresultresponse dataContentType include the following fields:

Field Description
response Contains the Result payload for the final result
redactedReason If content was redacted, provides the reason. For example, the "suppressCallRecording" value indicates that the suppress_call_recording RecognitionFlags field was set to true. See ASR values possibly masked for details.
asrSessionid Identifier of the current ASR session

asr-finalstatusresponse payload

Sample payload

"data": {
  "response": {
    "status": {
      "code": 200,
      "message": "Success"
    }
  },
  "processingTime": {
    "startTime": "2020-12-05T13:15:56.176Z",
    "durationMs": 10101
  },
  "asrSessionid": "31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c",
  "audioDurationMs": 9940,
  "dataContentType": "application/x-nuance-asr-finalstatusresponse.v2+json",
  "requestid": "3136437b-f95b-4a37-8c18-5c6ee836e212"
}

Current version: v2

The application/x-nuance-asr-finalstatusresponse payloads provide the content of the 200 Success status message that is sent back to the client to indicate that audio was processed, recognition completed, and a result returned with at least one hypothesis.

This event is usually sent after the finalresultresponse event is received.

In addition to the standard fields described in Data field, messages with the application/x-nuance-asr-finalstatusresponse dataContentType include the following fields:

Field Description
response Contains the 200 Success status code. See Status messages and codes for details.
asrSessionid Identifier of the current ASR session
audioDurationMs Audio duration in milliseconds

asr-callsummary payload

Sample payload

"data": {
  "dataContentType": "application/x-nuance-asr-callsummary.v2+json",
  "callsummary": {
    "sessionId": "31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c",
    "clientData": {
      "applicationName": "appID:test_user_nuance_com_test3:geo:us:clientName:default"
    },
    "serverData": {
      "hostname": "ntefast-eng-usa-410-krypton-bdd85d4ff-6nnm8",
      "platform": "linux",
      "arch": "x64",
      "versions": {
        "Krypton": "4.6.0-202011042154",
        "S3": "12.24.100.04509",
        "MRec": "1.42.100.19403",
        "TextProc": "15.07.100.02109",
        "S3Config": "4.12.0"
      }
    },
    "absStartTime": 1607174156199,
    "dataPack": {
      "language": "eng-USA",
      "topic": "NTEFAST",
      "version": "4.1.0",
      "id": "ndp40-ntefast-eng-USA-r1"
    },
    "recognitionObjects": [],
    "Recognize": [{
        "absStartTime": 1607174156203,
        "recognitionParameters": {
          "autoPunctuation": false,
          "enableNBestOnRejection": true,
          "enableSpeakerProfileUpdate": true,
          "enableUtteranceDetection": true,
          "audioFormat": "audio/l16;rate=16000",
          "autoEnd": true,
          "enablePartialResults": false,
          "immutablePartialResults": false,
          "startRecognitionTimers": true,
          "includeTokenization": false,
          "enableProfanityFilter": false,
          "suppressCallRecording": false,
          "confidenceFloatRepresentation": false,
          "suppressInitialCapitalization": false,
          "allowZeroBaseLmWeight": false
        },
        "audioPacketStats": {
          "packetsReceived": 497,
          "bytesReceived": 318080,
          "minimumSize": 640,
          "maximumSize": 640,
          "firstPacketTime": "2020-12-05T13:15:56.279Z",
          "lastPacketTime": "2020-12-05T13:16:06.208Z",
          "audioDurationMs": 9940
        },
        "recognitionObjects": [],
        "utterances": [{
            "nBest": [{
                "formattedText": "My credit card number is 3141592653589793",
                "minimallyFormattedText": "My credit card number is three one four one five nine two six five three five eight nine seven nine three",
                "IP!tokens": "yZS...bA=",
                "IP!prepostprocessing_tokens": "29uzyWVLBE4VrANT7g....HAw=="
              }
            ],
            "info": {
              "duration": 9460,
              "stereo": false,
              "level": 16897,
              "clippingDuration": 0,
              "snrEstimate": 27,
              "droppedSpeechPackets": 0,
              "droppedNonSpeechPackets": 0,
              "initialSilence": 380,
              "initialEnergy": -70.5528,
              "finalEnergy": -63.327,
              "meanEnergy": 151.079
            },
            "relStartTime": 360,
            "relEndTime": 9820,
            "completionCause": "SUCCESS"
          }
        ],
        "IP!S3Configuration": "QPh...OgSbjw==",
        "metrics": {
          "firstAudioToStartOfSpeech": 743,
          "firstAudioToResult": 9931
        },
        "absEndTime": 1607174166228,
        "IP!S3Events": "L4sFrmz..so+bzPlqw==",
        "audioDump": "http://minio-prod.fabric-prod.svc.cluster.local:9000/afss/da268859842ef28cd22f7b277e9c755538e9d70e45f2b8d785128afecd26affa/asraas/31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c",
        "audioUrn": "urn:nuance-mix:log:service:audio/asraas/31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c"
      }
    ],
    "IPKey": "tk.....KHLNp0hFA==",
    "absEndTime": 1607174166275,
    "IP!S3Events": "L4s...jZs="
  },
  "processingTime": {
    "startTime": "2020-12-05T13:15:56.176Z",
    "durationMs": 10101
  },
  "asrSessionid": "31bd3cb7-aaa0-4b22-a8ee-6a5993dbd50c",
  "requestid": "3136437b-f95b-4a37-8c18-5c6ee836e212"
}

Current version: v2

The application/x-nuance-asr-callsummary payloads provide summary information about the ASR interaction (full recognition turn), including request parameters, results, statistics, and internal events.

In addition to the standard fields described in Data field, messages with the application/x-nuance-asr-callsummary dataContentType include the following fields:

Field Description
serverData Details about the Krypton server. See serverData for details.
absStartTime Audio stream start time
dataPack Information about the current data pack. See DataPack in the ASRaaS documentation for details.
recognitionObjects List of recognition objects (wordsets and domain LMs) used during the session. See recognitionObjects for details.
Recognize Information about the Recognize command. See Recognize for details.
IPKey RSA-encrypted key used to encrypt IP data (S3Configuration, S3Events, and tokens)
absEndTime Audio stream end time
IP!S3Events S3 events received during the session. Encrypted.

serverData

Details about the Krypton server.

Field Description
hostname Name of the machine running the Krypton server
platform Software platform of the Krypton server
arch Software architecture of the Krypton server
versions Versions of the different components:
Krypton Krypton software version
S3 S3 library version
MRec MRec library version
TextProc TextProc library version
S3Config S3Config library version

RecognitionObjects

List of recognition objects (wordsets and domain LMs) used during the session.

Field Description
id ID of the recognition object, as provided by the client
S3Id ID correlating the related S3 event
type Type of recognition object (DLM/wordset)
weight Weight value provided by the client application
url URL of the object

Recognize

Information about the Recognize command.

Field Description
absStartTime Audio stream start time
recognitionParameters Recognition parameters and values used for the session. Some of these values can be configured in the ASRaaS RecognitionInitMessage message. The other values are server defaults.
autoPunctuation Whether to enable auto punctuation
enableSpeakerProfileUpdate If speaker profiles are used in the session, whether updated speaker adaptation data is stored after the session
enableNBestOnRejection Whether N-best results are returned (true) or discarded (false) when recognition ends with a rejection (also known as “no-match”)
enableUtteranceDetection Whether to activate the internal utterance detection mechanism
audioFormat Incoming audio mime type
autoEnd Whether the recognition turn ends when Krypton detects the end of an utterance
enablePartialResults Whether to enable partial results
immutablePartialResults When partial results are enabled (see enablePartialResults above), whether to return stable partial results
startRecognitionTimers Whether timers are activated when a recognition turn starts
includeTokenization Whether to include tokenized results
enableProfanityFilter Whether to filter profanities and other unacceptable language from transcriptions
suppressCallRecording Whether call recording is disabled for the recognition turn
confidenceFloatRepresentation Nuance internal
utteranceEndSilence Minimum amount of silence (time designation) that causes an utterance to end
AudioPacketStats Provides statistics about the audio packets received
packetsReceived Number of audio packet received by ASRaaS
bytesReceived Total audio bytes received
minimumSize Size (in bytes) of the smallest audio packet received
maximumSize Size (in bytes) of the largest audio packet received
firstPacketTime Date and time the first audio packet was received
lastPacketTime Date and time the last audio packet was received
audioDurationMs The duration of the audio received minus begin and end silence periods as detected by ASRaaS
RecognitionObjects List of recognition objects (wordsets and domain LMs) used for recognition. See RecognitionObjects for a description of the fields.
nBest List of n-best results
info Information about the recognized sentence in the transcription result. See UtteranceInfo for a description of each field.
relStartTime Start time, relative to Recognize.absStartTime, computed on audio
relEndTime End time, relative to relative to Recognize.absStartTime, computed on audio
completionCause In response to Recognize, the reason for the response:
  • SUCCESS: The result was correctly produced.
  • ERROR: An error prevented the result from being computed.
  • NO-INPUT-TIMEOUT: The noInputTimeout interval has elapsed: no result is delivered.
  • RECOGNITION-TIMEOUT: The recognitionTimeout interval elapsed: the result may or may not be delivered.
  • END-OF-STREAM: An EndOfInput command terminated the Recognize turn: the result may or may not be delivered.
  • NO-MATCH: The result has been produced, but is marked as rejected. This completion cause is returned for extremely noisy and/or unintelligible audio, or when the audio is not compatible with the declared audio format.
IP!S3Configuration S3 parameters provided by krypton to S3. Encrypted with an IPKey.
metrics Metrics for the call; available if enabled in configuration
absEndTime Audio stream end time
IP!S3Events S3 events received during the call. Encrypted.
audioDump URL of the audio recording
audioUrn URN of the audio file for the utterance. This field can be used to download the audio file with the AFSS API. See AFSS API for details.

nBest

List of n-best results.

Field Description
confidence The confidence score for the entire transcription, 0 to 1
PCI!formattedText Formatted text of the result, for example, $500. Encrypted with a PCIKey.
PCI!minimallyFormattedText Slightly formatted text of the result, for example, Five hundred dollars. Encrypted with a PCIKey.
IP!tokens Nuance-internal representation of the recognition result. Encrypted with an IPKey.

metrics

Metrics for the call; available if enabled in configuration.

Field Description
firstAudioToStartOfSpeech The number of milliseconds from the arrival of the first audio message/binary packet after Recognize to transmission of the StartOfSpeech event
firstAudioToResult The number of milliseconds from the arrival of the first audio message/binary packet after Recognize to the current time

ASR values possibly masked

When the suppress_call_recording RecognitionFlags field is set to true, PCI information is suppressed from the event logs and the corresponding audio is removed. The hypothesis field is empty and the redactedReason field is provided, as shown in the second example in asr-partialresultresponse payload.

The minimally formatted text is still available in the final result returned to the application, but this information is not logged.

Correlating event logs

A single interaction can create multiple event logs. Event logs can be correlated in different ways:

Correlate logs for a specific API interaction

Example: Event logs in dialog start request interaction with requestid "50f04cf5-debd-9a6a-b1f0-75e408ce21d1"


// NII session-start

{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key":
  {
    "service": "DLGaaS",
    "id": "ca6e4281-47c1-4cc8-a3e2-48d0bcda29de"
  },
  "value":
  {
    ...
    "data":
    {
      "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
      "traceid": "428c127f3c1adc1e39c90efee1c78331",
      "requestid": "50f04cf5-debd-9a6a-b1f0-75e408ce21d1",
      "sessionid": "b15302ac-be35-4d9c-b593-3789b091e83f",
      "locale": "en-US",
      "clientData": {},
      "seqid": "1",
      "events": [
        {
          "name": "session-start",
          "value":
          {
            ...
          }
        }
      ]
    }
  },
  "partition": 0,
  "offset": 1057
}

// NII selector
{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key":
  {
    "service": "DLGaaS",
    "id": "2d5afb58-928b-4b22-91eb-bd9cec61681e"
  },
  "value":
  {
    ...
    "data":
    {
      "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
      "traceid": "428c127f3c1adc1e39c90efee1c78331",
      "requestid": "50f04cf5-debd-9a6a-b1f0-75e408ce21d1",
      "sessionid": "b15302ac-be35-4d9c-b593-3789b091e83f",
      "locale": "en-US",
      "clientData": {},
      "seqid": "2",
      "events": [
        {
          "name": "selector",
          "value":
          {
            "channel": "default",
            "language": "en-US"
          }
        }
      ]
    }
  },
  "partition": 0,
  "offset": 1058
}

//DLGaaS-Start-Start
{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key":
  {
    "service": "DLGaaS",
    "id": "892e1679-065d-4bdd-8112-ff12f64e70cf"
  },
  "value":
  {
    ...
    "data":
    {
      "dataContentType": "application/x-nuance-dlg-interaction-summary.v1+json",
      "traceid": "428c127f3c1adc1e39c90efee1c78331",
      "requestid": "50f04cf5-debd-9a6a-b1f0-75e408ce21d1",
      "sessionid": "",
      "sessionTimeoutSec": 900,
      "locale": "en-US",
      "events": [
        {
          "event": "DLGaaS-Start-Start",
          "time": "2021-03-15T14:39:00.393Z"
        }
      ]
    }
  },
  "partition": 0,
  "offset": 1066
}

// DLGaaS-Start-End
{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key":
  {
    "service": "DLGaaS",
    "id": "f3ceed51-8a5b-4496-8fc1-6da596816164"
  },
  "value":
  {
    ...
    "data":
    {
      "dataContentType": "application/x-nuance-dlg-interaction-summary.v1+json",
      "traceid": "428c127f3c1adc1e39c90efee1c78331",
      "requestid": "50f04cf5-debd-9a6a-b1f0-75e408ce21d1",
      "sessionid": "b15302ac-be35-4d9c-b593-3789b091e83f",
      "channel": "default",
      "sessionTimeoutSec": 900,
      "locale": "en-US",
      "processingTime":
      {
        "startTime": "2021-03-15T14:39:00.393Z",
        "durationMs": 128
      },
      "events": [
        {
          "event": "DLGaaS-Start-End",
          "time": "2021-03-15T14:39:00.521Z"
        }
      ],
      "request":
      {
        ...
      },
      "response":
      {
        ...
      }
    }
  },
  "partition": 0,
  "offset": 1067
}


A single API request/response interaction will create multiple event logs across the different services and may also include dialog NII logs.

For example, consider a dialog Start request/response interaction; it might generate the following event logs:

To correlate all the event log records that apply to a specific request/response interaction, use the requestid field. All the logs for a specific customer interaction share the same requestid value.

The requestid field is particularly useful for interactions that span multiple services. For example, consider a dialog Execute request that requires NLU interpretation. In addition to the related NII logs, such an interaction might also include the following event logs:

Correlate ASR, NLU, and TTS logs for a specific dialog session

Example: ASRaaS event log for dialog session "b15302ac-be35-4d9c-b593-3789b091e83f"


{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key":
  {
    "service": "ASRaaS",
    "id": "6c4945f4-bc62-495d-9602-75015692bbbd"
  },
  "value":
  {
    "specversion": "1.0",
    "service": "ASRaaS",
    "source": "nuance.asr.v1.Recognizer/Recognize",
    "type": "Recognize",
    "id": "a77f74fc-0274-44e6-9e33-f2269784b6df",
    "timestamp": "2021-03-15T14:39:07.782Z",
    "datacontenttype": "application/json",
    "data":
    {
      "request":
      {
        "recognitionInitMessage":
        {
          ...
          "clientData":
          {
            "x-nuance-dialog-session-id": "b15302ac-be35-4d9c-b593-3789b091e83f",
            "x-nuance-dialog-seqid": "11"
          }
        }
      },
      "asrSessionid": "6c4945f4-bc62-495d-9602-75015692bbbd",
      "dataContentType": "application/x-nuance-asr-recognitioninitmessage.v2+json",
      "requestid": "516ef36d-5a12-9939-a232-ac6256110d0f",
      "audioUrn": "urn:nuance-mix:log:service:audio/asraas/6c4945f4-bc62-495d-9602-75015692bbbd"
    },
    "appid": "DEMO-OMNICHANNEL-APP-DEV"
  },
  "partition": 0,
  "offset": 1078
}

When DLGaaS calls ASRaaS, NLUaaS, or TTSaaS, it includes the following fields in the logs of the service, as part of the clientData of the request:

These fields can be used to identify all the ASR, NLU, and TTS event logs that relate to a dialog session.

Correlate NII logs for a specific dialog session

Example: First message in NII sequence for dialog session "c475ff51-d60c-4ad0-b1df-7fb4aae43b13"


{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key":
  {
    "service": "DLGaaS",
    "id": "03997a3c-c18d-419a-83a9-6ccc9e5cfd71"
  },
  "value":
  {
    ...
    "data":
    {
      "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
      "traceid": "1f16b4bd07b2f1f9c548acaedc487c56",
      "requestid": "01b73d0f-bfd6-9921-a160-ad037d2e6036",
      "sessionid": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13",
      "locale": "en-US",
      "clientData": {},
      "seqid": "1",
      "events": [
        {
          "name": "session-start",
          "value":
          {
            ...
          }
        }
      ]
    }
  },
  "partition": 0,
  "offset": 1091
}

Example: Second message in NII sequence for dialog session "c475ff51-d60c-4ad0-b1df-7fb4aae43b13"


{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key":
  {
    "service": "DLGaaS",
    "id": "0e491cdd-37c3-4895-8657-5da5701c931b"
  },
  "value":
  {
    ...
    "data":
    {
      "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
      "traceid": "1f16b4bd07b2f1f9c548acaedc487c56",
      "requestid": "01b73d0f-bfd6-9921-a160-ad037d2e6036",
      "sessionid": "c475ff51-d60c-4ad0-b1df-7fb4aae43b13",
      "locale": "en-US",
      "clientData": {},
      "seqid": "2",
      "events": [
        {
          "name": "selector",
          "value":
          {
            "channel": "default",
            "language": "en-US"
          }
        }
      ]
    }
  },
  "partition": 0,
  "offset": 1092
}

To correlate the NII logs that correspond to a specific dialog session, use the following data fields:

The seqid ensures that you can determine the sequence of the NII logs in a dialog session, since these logs may not be received in the order in which they are created and sent.

For a description of the NII log events, see dlg-nii-logs payload.

Inject content into the payloads

Your client application can inject the following content into the payloads:

Client Data

Sample client data field, in NII and DLGaaS logs

// DLGaaS log
{
    "topic": "DEMO-OMNICHANNEL-APP-DEV",
    "key": {
        "service": "DLGaaS"
    },
    "value": {
        ...
        },
        "data": {
            ...
            "request": {
                ...
                "clientData": {
                    "field1": "default",
                    "field2": "optional"
                }
            },
            "response": {
            ...
            }
        }
    },
    "partition": 0,
    "offset": 890
}

// NII log
{
    "topic": "DEMO-OMNICHANNEL-APP-DEV",
    "key": {
        "service": "DLGaaS"
    },
    "value": {
        ...
        },
        "data": {
            ...
            "clientData": {
                "field1": "default",
                "field2": "optional"
            },
            "seqid": "1",
            "events": [{
                    "name": "session-start",
                    "value": {
                        ...
                    }
                }
            ]
        }
    },
    "partition": 0,
    "offset": 893
}

You can inject a map of key-value pairs from the XaaS APIs, through the client_data field, as follows:

Service API Message
ASRaaS RecognitionInitMessage
NLUaaS InterpretRequest
DLGaaS StartRequest
TTSaaS SynthesisRequest

This data is stored in the clientData field of the request.

If the client_data field is provided in the XaaS request, it will be included in all the event logs related to this request.

User ID

Sample user ID field

{
    "topic": "DEMO-OMNICHANNEL-APP-DEV",
    "key": {
        "service": "DLGaaS"
    },
    "value": {
        ...
        },
        "data": {
            "dataContentType": "application/x-nuance-dlg-interaction-summary.v1+json",
            "traceid": "139d166b4b898ccb",
            "sessionid": "",
            "sessionTimeoutSec": 0,
            "locale": "en-US",
            "userid": "533663ad61ab5b49214f55129d6dd7fa44ebd13f5f4e6c42212ee286048b6c85",
            "events": [{
                    "event": "DLGaaS-Start-Start",
                    "time": "2020-11-23T22:05:49.385Z"
                }
            ]
        }
    },
    "partition": 0,
    "offset": 889
}

You can inject a user ID from the XaaS APIs, through the user_id field, as follows:

Service API Message
ASRaaS RecognitionInitMessage
NLUaaS InterpretRequest
DLGaaS StartRequest, ExecuteRequest, and StopRequest
TTSaaS SynthesisRequest

The user ID is hashed using SHA256 and is provided as a 64-character hexadecimal string in the event logs, in the userid field under data.

If the user_id field is provided in the XaaS request, it will be included in all the event logs related to this request.

Client-provided request ID

Sample metadata header in Python client for execute request

    metadata = [('x-client-request-id', 'sample_request_id')]
    execute_response, call = stub.Execute.with_call(execute_request, metadata=metadata)

Event logs for execute request in dlg-interaction-summary log

{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key": {
    "service": "DLGaaS",
    "id": "dd1e5bd9-516a-4b0e-8d77-d7ffe3503792"
  },
  "value": {
    "specversion": "1.0",
    "service": "DLGaaS",
    "source": "nuance.dlg.v1.DialogService/Execute",
    "type": "Execute",
    ...
    "data": {
      "dataContentType": "application/x-nuance-dlg-interaction-summary.v1+json",
      "traceid": "8a99dc457aec7d9793aa549f2a2749f6",
      "requestid": "6a6f6188-7a19-9dc6-b7c8-b6f8caf7b9e4",
      "clientRequestid": "sample_request_id",
      ...

Event logs for execute request in dlg-nii-logs log

{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "key": {
    "service": "DLGaaS",
    "id": "b45634f6-fc1a-4e1e-885c-a2d9630ebab1"
  },
  "value": {
    "specversion": "1.0",
    "service": "DLGaaS",
    "source": "NIIEventLogger",
    "type": "NIIEventLog",
    ...
    "data": {
      "dataContentType": "application/x-nuance-dlg-nii-logs.v1+json",
      "traceid": "8a99dc457aec7d9793aa549f2a2749f6",
      "requestid": "6a6f6188-7a19-9dc6-b7c8-b6f8caf7b9e4",
      "clientRequestid": "test1234",
      ...

You can inject your own request ID in the event logs by setting the x-client-request-id header in the gRPC metadata for a request in the client application. This field can be used for tracing purposes.

The header value will get logged in the clientRequestid field of the Data field for all the event logs generated for that request.

For example, the sample code shows how the x-client-request-id header is specified in the metadata for the Execute request in the client application. This header value then gets included in the clientRequestid field of all the event logs related to this Execute request.

Kafka consumer API

Version: v1

This document covers v1 of the Mix Kafka REST Consumer API.

URL

https://log.api.nuance.com

Partitions

Get Partitions

Example responses

200 Response

{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "partitions": [
    0
  ]
}

GET /partitions

Retrieve the list of available partitions for a topic.

Responses

Status Meaning Description Schema
200 OK List of available partitions topicPartitions
404 Not Found Topic does not exist errorResponse

Get Partition Offsets

Example responses

200 Response

{
  "beginning_offset": 441,
  "end_offset": 641
}

GET /partitions/{partition}/offsets

Retrieve the first and last available offsets for a partition.

Parameters

Name In Type Description
partition path number Partition number

Responses

Status Meaning Description Schema
200 OK Partition offset details partitionsOffsets

Consumers and Groups

Create Consumer

Example responses

400 Response

{
  "error_code": 401,
  "message": "The request could not be authorized"
}

POST /consumers

Create a new consumer instance in a consumer group.

The consumer group name must follow the pattern appID-{appId/topic}-clientName-{clientName}-{two-digit number}, for example, appID-DEMO-OMNICHANNEL-APP-DEV-clientName-default-01

The consumer name must follow the pattern consumer-{UUIDv4}, for example, consumer-9872ec64-50d9-4dcd-9f4f-e8b5099a7f4f

Body parameter

{
  "auto.offset.reset": "latest",
  "consumer.request.timeout.ms": "1000",
  "fetch.min.bytes": "-1",
  "auto.commit.enable": "false"
}

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}
body body createConsumers Object that provides the parameters used when creating a consumer

Responses

Status Meaning Description Schema
204 No Content Consumer instance created successfully None
400 Bad Request Bad Request, topic does not exist errorResponse

Delete Consumer

DELETE /consumers

Delete the specified consumer instance from a consumer group.

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}

Responses

Status Meaning Description Schema
204 No Content Consumer instance deleted successfully None

Subscriptions

Subscribe

POST /consumers/subscription

Subscribe a consumer to a topic to get dynamically assigned partitions. If a subscription already exists, it will be replaced.

Note: Use either the Subscribe endpoint (to assign partitions automatically) or Assign Partitions endpoint (to manually assign partitions).

Body parameter

{
  "topics": [
    "DEMO-OMNICHANNEL-APP-DEV"
  ]
}

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}
body body subscription Object that provides the topic to subscribe to

Responses

Status Meaning Description Schema
204 No Content Consumer successfully subscribed to the topic specified None

Unsubscribe

DELETE /consumers/subscription

Remove the existing topic subscription for a consumer.

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}

Responses

Status Meaning Description Schema
204 No Content Consumer successfully unsubscribed from the topic specified None

Get Subscription

Example responses

204 Response

{
  "topics": [
    "DEMO-OMNICHANNEL-APP-DEV"
  ]
}

GET /consumers/subscription

Retrieve the topic to which the consumer is currently subscribed.

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}

Responses

Status Meaning Description Schema
204 No Content Current subscribed topic subscription

Records

Get Records

Example responses

200 Response

[
  {
    "topic": "DEMO-OMNICHANNEL-APP-DEV",
    "key": {
      "service": "DLGaaS",
      "id": "3a4b511c-c06b-497f-ba0a-24c1f95a7e60"
    },
    "value": {},
    "partition": 0,
    "offset": 210
  }
]

GET /consumers/records

Fetch data for the topic to which the specified consumer is subscribed. This request requires the accept: application/json header.

Notes:

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
Accept header string Required accept: application/json header
consumer-name header string Consumer name consumer-{UUIDv4}

Responses

Status Meaning Description Schema
200 OK Records fetched by the consumer instance, in JSON format. jsonConsumerRecords

Assignments

Assign Partitions

POST /consumers/assignments

Manually assign a list of partitions to this consumer. This request requires both the content-type: application/json header and the accept: application/json.

Note: Use either the Subscribe endpoint (to assign partitions automatically) or Assign Partitions endpoint (to manually assign partitions).

Body parameter

{
  "partitions": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0
    }
  ]
}

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
Accept header string Required accept: application/json header
Content-Type header string Required content-type: application/json header
consumer-name header string Consumer name consumer-{UUIDv4}
body body partitionsAssignment Object that provides the partitions to assign to a consumer

Responses

Status Meaning Description Schema
204 No Content Partition successfully assigned to the consumer None

Get Assigned Partitions

Example responses

200 Response

{
  "partitions": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0
    }
  ]
}

GET /consumers/assignments

Retrieve the list of partitions currently manually assigned to this consumer. This request requires the accept: application/json header.

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
Accept header string Required accept: application/json header
consumer-name header string Consumer name consumer-{UUIDv4}

Responses

Status Meaning Description Schema
200 OK List of partitions assigned to the specified consumer partitionsAssignment

Offsets

Get Committed Offsets

Example responses

200 Response

{
  "offsets": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0,
      "offset": 412,
      "metadata": "string"
    }
  ]
}

POST /consumers/committed/offsets

Retrieve the last committed offsets for the partitions specified.

Body parameter

{
  "partitions": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0
    }
  ]
}

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}
body body partitions Object that provides the partitions for which to retrieve offsets

Responses

Status Meaning Description Schema
200 OK Last committed offsets for the partitions specified committedOffsets

Commit Offsets

POST /consumers/offsets

Commit a list of offsets for the consumer. When the POST body is empty, this endpoint commits all the records that have been fetched by the consumer instance.

Body parameter

{
  "offsets": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0,
      "offset": 412
    }
  ]
}

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}
body body offsets Object that provides the offsets to commit

Responses

Status Meaning Description Schema
204 No Content Committed offsets for the partitions specified None

Positions

Override Offsets

POST /consumers/positions

Overrides the offsets that the consumer will use when fetching the next set of records.

Body parameter

{
  "offsets": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0,
      "offset": 412
    }
  ]
}

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}
body body offsets Object that provides the offsets to use when fetching the next set of records

Responses

Status Meaning Description Schema
204 No Content Committed offsets for the partitions specified. None

Seek First Offset

POST /consumers/positions/beginning

Seek to the first offset for each of the partitions specified.

Body parameter

{
  "partitions": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0
    }
  ]
}

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}
body body partitions Object that specifies the partitions for which to seek the first offsets

Responses

Status Meaning Description Schema
204 No Content The current offset is set to the first available offset None

Seek Last Offset

POST /consumers/positions/end

Seek to the last offset for each of the partitions specified.

Body parameter

{
  "partitions": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0
    }
  ]
}

Parameters

Name In Type Description
consumer-group header string Consumer group name appID-{appId/topic}-clientName-{clientName}-{two-digit number}
consumer-name header string Consumer name consumer-{UUIDv4}
body body partitions Object that specifies the partitions for which to see the last offsets

Responses

Status Meaning Description Schema
204 No Content The current offset is set to the last available offset None

Schemas

createConsumers

{
  "auto.offset.reset": "latest",
  "consumer.request.timeout.ms": "1000",
  "fetch.min.bytes": "-1",
  "auto.commit.enable": "false"
}

Object that provides the parameters used when creating a consumer

Properties

Name Type Description
auto.offset.reset string Offset reset policy for the consumer group. Determines the initial position to use when the group is created. Valid values are earliest (earliest available offset), latest (latest available offset), and none (current offset if the consumer group is present). If there is no current offset associated with the consumer group, the option none will throw an error.
This property applies only if no offset is committed. When an offset is committed, auto.offset.reset is not considered.
consumer.request.timeout.ms string The maximum amount of time to wait for messages for a request if the maximum number of messages (as per fetch.min.bytes) has not yet been reached.
fetch.min.bytes string Minimum number of bytes in message keys and values that must be returned by a single request before the timeout specified in consumer.request.timeout.ms expires.
A value of -1 disables this functionality.
auto.commit.enable string If set to true, the committed offset will be automatically updated when the record is fetched using the Get Records endpoint.

Enumerated Values

Property Value
auto.offset.reset earliest
auto.offset.reset latest
auto.offset.reset none
auto.commit.enable true
auto.commit.enable false

committedOffsets

{
  "offsets": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0,
      "offset": 412,
      "metadata": "string"
    }
  ]
}

Last committed offsets for the partitions specified

Properties

Name Type Description
offsets [any] Array of committed offsets
» topic string Name of the topic. In Mix, this is the AppID.
» partition number Number of the partition
» offset number Committed offset
» metadata string Currently not used

offsets

{
  "offsets": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0,
      "offset": 412
    }
  ]
}

Object that defines offsets

Properties

Name Type Description
offsets [any] Array of offsets
» topic string Name of the topic. In Mix, this is the AppID.
» partition number Number of the partition
» offset number Offset number

partitions

{
  "partitions": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0
    }
  ]
}

Object that defines an array of partitions

Properties

Name Type Description
partitions [any] partitions
» topic string Name of the topic. In Mix, this is the AppID.
» partition number Number of the partition

topicPartitions

{
  "topic": "DEMO-OMNICHANNEL-APP-DEV",
  "partitions": [
    0
  ]
}

Object that defines a topic and its available partitions

Properties

Name Type Description
topic string Name of the topic. In Mix, this is the AppID.
partitions [number] Array of partition numbers

errorResponse

{
  "error_code": 401,
  "message": "The request could not be authorized"
}

Object that defines an error response

Properties

Name Type Description
error_code string Error code
message string Error message

jsonConsumerRecords

[
  {
    "topic": "DEMO-OMNICHANNEL-APP-DEV",
    "key": {
      "service": "DLGaaS",
      "id": "3a4b511c-c06b-497f-ba0a-24c1f95a7e60"
    },
    "value": {},
    "partition": 0,
    "offset": 210
  }
]

Records fetched by the consumer

Properties

Name Type Description
Records fetched by the consumer [object] Records fetched by the consumer
topic string Name of the topic. In Mix, this is the AppID.
key object Key for the message
» service any Name of the runtime service that generated this message
» id any Randomly-generated UUID that uniquely identifies the message
value object Contents of the Kafka message
partition number Partition number for this topic
offset number Offset for this record

subscription

{
  "topics": [
    "DEMO-OMNICHANNEL-APP-DEV"
  ]
}

Object that describes a topic subscription

Properties

Name Type Description
topics [string] Topic name. In Mix, this is the AppID.

partitionsAssignment

{
  "partitions": [
    {
      "topic": "DEMO-OMNICHANNEL-APP-DEV",
      "partition": 0
    }
  ]
}

Object that describes the assigned partitions

Properties

Name Type Description
partitions [object] Array of assigned partitions
» topic string Topic name. In Mix, this is the AppID.
» partition number Partition number

partitionsOffsets

{
  "beginning_offset": 441,
  "end_offset": 641
}

Object that provides the first and last available offsets in a partition

Properties

Name Type Description
beginning_offset number First available offset in the partition specified
end_offset number Last available offset in the partition specified

AFSS API

The Audio File Storage Service (AFSS) API lets you download audio files of the utterances that were sent for recognition to ASRaaS.

The API requires the URN of the resource to download, in the following format:

urn:nuance-mix:log:service:audio/{service}/{audioFileName}

For ASRaaS, this is obtained from the audioUrn field in the asr-callsummary payload event logs.

For example, consider the following event log:


...
"absEndTime": 1616532498670,
"IP!S3Events": "L4sFrmz...9MMxM0EC2s=",
"audioDump": "http://minio-prod.fabric-prod.svc.cluster.local:9000/afss/fd228157ac40497a6f9a51f43f476b8c40dc270de6d3f218810745104242b098/asraas/44620fec-4c48-4d41-82d9-68fd8732f84f",
"audioUrn": "urn:nuance-mix:log:service:audio/asraas/44620fec-4c48-4d41-82d9-68fd8732f84f"
...

To get the audio file for this URN, the REST endpoint would look something like this:

GET https://log.api.nuance.com/api/v1/audio?urn=urn:nuance-mix:log:service:audio/asraas/44620fec-4c48-4d41-82d9-68fd8732f84f

Your client application must provide an access token to access the AFSS API. This access token can be shared between the AFSS and the Kafka APIs. See Authorization for details.

AFSS version: v1

This document covers v1 of the Nuance AFSS Rest API.

AFSS URL

https://log.api.nuance.com

Audio resources

Get audio resources

GET /api/v1/audio

Retrieve the audio file that corresponds to the URN specified.

Parameters

Name In Type Description
urn query string URN of the resource to download. For ASRaaS, this is obtained with the audioUrn field.

Responses

Status Meaning Description Schema
200 OK Success Inline

Change log

2021-06-16

Added retention period for audio files.