1. Forms composition and elements

2. Form population

3. Service configuration in Compose

4. Final comments

Compose form population API

An important part of the Compose system is the possibility to populate forms through external services. This can e.g. allow you to populate a form with a respondents contact information only by providing their social security number. With less information to fill out, form submissions are faster, which means it saves time and patience for the respondents. Used in the right way this can also make data trustworthier and less error-prone, thereby improving post-processing. This document describes how you can build and use your own form population services using virtually any programming language that supports building of JSON web services.

1. How forms work

Before we get into how the population works we need to talk about forms. Forms in Compose have several parts. First of all there is a separation between the collected data and the visual side. The data is represented in a data model and visualization is done through views. For form population you work with the data model, since that is what the answer data is based on.

1.1 Data model

The data model consists of a tree structure of questions and question groups. All questions and question groups have unique ids and can thereby not occur more than once in the structure. When it comes to form answers the question groups can have multiple answers, but the questions only single answers. In order to duplicate answers for a question you thereby have to put it inside a question group and make that duplicatable. The root of the data model can also be viewed as a question group, but without possibility of duplication.

1.2 Questions and their properties

There are multiple types of questions in the system, but they share that they have a validation (data type). Validations dictate the rules for valid answers. For instance, a value input question may have the validation integer, which means answer values must match what an integer looks like. Now even if that is what is expected, respondents, when they fill out a form, may provide some other value, like e.g. a decimal number, or some random string. In that case the question will still have an answer, but it is not valid. Since form population happens before validation it is important to note that both valid and invalid answers may be present. The respondent must however still correct anything that is not valid before they leave the form.
All questions in the data model have labels with the question text. These labels are localized to whatever languages the form supports. Important to note here is that designers have an option of overriding the labels in the views, which means what you see in the data model may not always be what you see as a respondent.

2. Form population

Form population works by sending a description of the form and what answer values it currently has to an external service. The response from the service contains which answer values should be changed and the answer data is then updated accordingly and showed to the respondent. The external service has three different ways to locate the questions to populate:

• Via question id – basically meaning you hard-code for a specific form version.

• Via question label – risky if you make changes to the labels later, since it may not be obvious that it can affect population. You also may have to take into account which language is used.

• Via question alias – normally best way, but must match exactly.

Question alias is a unique additional identifier that may be set on a question. It is non-mandatory and there are rules as to which names it can take. It can't e.g. contain white-space characters and must have a length of 1 to 40 characters. Aliases are, further more, also available on question groups.

Message exchange-wise communication is very simple. Form data is sent as a JSON object via a HTTP POST request and a JSON object with the response is returned. The response is either a success response or a failure response. Implementing a service thereby involves writing code to read the requests and code to generate the responses. Everything is normal HTTP calls with UTF-8 encoding and content type application/json. Note however that HTTPS is strongly advised for security reasons given that authentication otherwise takes place in clear text.

2.1 Population request

At the root of the population request object the following can be found:

• messageFormatVersion: Integer

• operation: Object

• authentication: Object

• formVersion: Object

• formAnswers: Object

Whoever reads the data can also expect the data to be sent in this order.

2.1.1 messageFormatVersion

The messageFormatVersion is a simple integer value that currently has the value 1. When changes are made to the message format, that can cause issues, this will be incremented. Note that changes may happen where new attributes are introduced without this value being incremented. It is therefore good to build services in such a way that they can tolerate it. Changes like this will however never happen at root level. Updated documentation to customers should also be provided whenever changes are done.

2.1.2 operation

The operation currently just contains a hard-coded type value like this:

You would see a different value here if the operation was doing something else. There may then also be other parameters provided. It is therefore good practice to check that this value is correct when writing a form population service.

2.1.3 authentication

Currently only one type of authentication is supported, but there may be more added in the future. For the current type a user name and a password is sent. These are expected to be checked by the external service. If they do not match what is expected then an ACCESS_DENIED response should be returned (more on failure responses later). The full object looks like this (myUser and myPassword should be replaced with whatever is expected):

2.1.4 formVersion

The form version object contains at its root the following:

• formId: String

• formVersion: String

• formVersionUuid: String

• languageScope: Object

• displayName: Object

• formDataModel: Object

It can also be expected that the variables are sent in this order.

2.1.4.1 formId, formVersion and formVersionUuid

Form id, form version and form version UUID are all identifiers for the form. Form id and form version is a locally unique identifiers for the Compose repository they come from. The form version UUID is however globally unique, given the design of UUIDs . When creating form population services these value may be used to verify that the form is the expected one. Depending on the situation, this can simplify responses, since questions may be hard-coded.

2.1.4.2 languageScope and localized values

languageScope lists which languages are available in the form. One of the available languages is always the primary language. This means that for this language there will always be a label to use. For the other languages it may be so that they use the label of the primary language. Language codes are three letter ISO 639-3 codes, but in the case where the language is unspecifed null is used. Null can only be used if there is only one language. Here is what the JSON object may look like for language scope using null:

The same as above, but with multiple languages and English as the primary language:

For testing it can be worth mentioning that the server always returns the list of languages sorted. The same applies to localized values that we get to next. In the form data model, which we get to later, there are two different types of localized objects using the languages defined in the language scope. They both have the same structure, but the content is in one case normal text and in the other text markup. They are called LocalizedString and LocalizedFormattedText respectively and the main difference is the value of the class name attribute. The represented data is basically a look-up table with language codes as keys and texts as the values. Since null is used to represent unspecified languages the structure is based on two arrays instead of just one associative array. Here is what it looks like for LocalizedText:

For LocalizedFormattedText it looks like this (showing with different set of languages):

The markup used in the values of the formatted text is out of scope for this document, but, in short, it is an HTML-like set of tags that are available. Many times the texts also come without markup.
It should be noted that the default language currently always matches the primary language. Idea is also same – if a language is not present then the default language text is used.

2.1.4.3 displayName

The display name is a localized string value for the title of the form. For form population this may not be so interesting to use for anything else than logging. It is provided mainly for completeness.

2.1.4.4 formDataModel

The form data model is, as was described earlier, a tree structure, but in order to simplify integration it has been flattened and consists instead, in the population request, of two look-up tables – one for questions and one for question groups. The keys are paths to the respective elements. Since you don't have to use recursion when reading the structure it is thought of as simpler. The paths consists of question and question group ids separated by underscores. A path may be e.g. QG1_QG13_Q12. Question ids start with Q and ends with an integer. Question group ids start with QG and end with an integer. Underscore has been chosen as separator since it is a valid character that may be used in JavaScript/JSON variable names.

2.1.4.4.1 Question groups

Question group objects have the following content:

• id: String

• alias: String

• isRepeatable: Boolean

• label: Object

The id is as it says the unique identifier of the question group, e.g. Q15. Alias is, as was mentioned earlier, a way to give an additional unique identifier. If it is not set then it will have value null. The isRepeatable flag indicates if a question group may have more than one instance. The label is the title of the question group. Here a LocalizedFormattedText object is used.

2.1.4.4.2 Questions

Questions come in different types, but they all have the following content:

• id: String

• alias: String

• label: Object

• className: String

The id is, of course, the unique id of the question, e.g. Q42. Alias is again the optional unique identifier that may be given. Label is the question text and it is, as with question groups, of type LocalizedFormattedText.
The className indicates the type of question. Currently only value input questions are supported and for these the class name value is ValueInputQuestion. The following additional attributes are also provided for value input questions:

• valueType: String

• characterLimit: Integer

The type of answer value is indicated by valueType. This is the base data type of the question. It has been given a different name in order not to confuse it with the actual validation, since you can create your own custom validations in Compose. Value type currently take the following values: STRING, INTEGER, DECIMAL and DATE. The maximum number of characters that an answer value can take is indicated by characterLimit and exceeding this will lead to cropping.

2.1.5 formAnswers

Form answers are conceptually a tree structure, but also here the structure has been flattened. Since only question answers are interesting there is no separation between question groups and questions meaning only one look-up table is needed. The keys are answer paths that differ from the data model by the fact that question groups have an index. A question path can be e.g. QG1i0_QG2i0_Q1. Reason that an i-character is used as a separator and not something else has to do with, again, that these keys need to be valid JavaScript/JSON variable names.

2.1.5.1 Question answers

Question answers are represented by question answer objects. All of these have the following property:

• className: String

The class name signals which type of answer it is. Currently only one type of answer is supported: ValueInputAnswer. This type of answer is what is expected for value input questions. It has the following additional property:

• answerValue: String

This is the value of the answer as a string value. For this to be valid, which as mentioned earlier it does not have to be, it has to match the value type of the question. This is what the different value types expect:

• STRING – any string value.

• INTEGER – Integer value that consists of numbers 0-9 and does not start with zero if it has more than one digit. The value may be negated by having a starting minus. Note however that -0 is not a valid value.

• DECIMAL – Decimal value with optional dot as separator and similar constraints as the integer value before the dot. What follows after the dot does not have any constrains other than that it needs to be numbers 0-9. Also here minus zero is an invalid value.

• DATE – Date value in format YYYY-MM-DD. Formatting is done by the system.

Note that it is likely that future changes make new types of answer values available. It is therefore good to build services in such a way that they ignore types that they can't handle.

2.2 Population response

The population response is a JSON object that always contains this:

• isSuccess: Boolean

If the response was a success then form data follows and otherwise failure information.

2.2.1 Success

For the success response the following, additionally to isSuccess, is expected:

• formAnswers: Object

This an object with the same structure as the formAnswers object in the initial request, with a small difference: you can also return that answers are null. This means that any existing answer for that question will be cleared. If you do the same for a question group then all values of that question group will be cleared, meaning the instance is removed. Important to note here is that only values that are changed need to be sent back. All other values will be kept as they were before.

2.2.2 Failure

For the failure response the following, additionally to isSuccess, is expected:

• failureMessage: String

• failureType: String

The failure message is a technical error message intended for logs. It should normally be in English. The failure type can have the following values:

• ACCESS_DENIED – Signals that wrong access credentials where provided. It can also be used for if the server e.g. connects from the wrong IP. It can thereby depend on how the service is created.

• REQUEST_ERROR – Signals that the Compose has sent something unexpected, indicating that most likely something is misconfigured.

• SERVER_ERROR – Signals that the population service had an internal error and thereby could not fulfill the request.

• OBJECT_NOT_FOUND – Signals that the population service failed to find whatever it was supposed to use for population. This can e.g. be that the vehicle registration number provided in the form is not in the database. Idea is that by using this failure type a more informative error can be given to the respondent. If something invalid was entered by mistake it can possibly be corrected.

Important to note is that for the OBJECT_NOT_FOUND failure type also the following must be included in the response:

• objectName: String

This is simply a non-null string value with a description of which object was not found. It can be set to, more or less, anything as long as it is descriptive for the respondent.

3. Service configuration in Compose

The external services, that are used for form population, need to be configured in Compose before they can be used. This is done in the Service repository item. Conceptually a service has a mount point used to access it and exposes a set of operations. You also configure URL and authentication. The operations have some restrictions in terms of what you may name them and they are normally accessed by appending the operation name to the service URL. In cases where this is too restrictive you can also override what is appended. This can also be useful if you want to e.g. pass URL parameters. In short this means that as long as you can access all form population service operations at unique URLs everything should be possible.
Let us go through the steps necessary to configure a Service with the current version of Compose. Note that future versions of Compose may change in style and ease of use, but the steps should match even if this document is not kept up to date.

3.1 Service creation

Service creation is done like other repository items via the organizer of Compose, or directly in the service builder. In other words, click a button and provide a name after selecting where to store the new repository item.

3.2 Service configuration

Once created and opened the service builder currently looks like this (here with some example data already filled out). The following may be configured on the service tab:

Open

• Remote service URL

• Mount point

• User name

• Password

The remote service URL is the base URL where the service can be accessed. The URL to each operation normally starts with this and then has a slash followed by the operation name.
The mount point is used to internally locate the service within Compose. There can be several services running at the same time and using the same mount point, but the operations must have unique signatures. If operations collide, then you will be prevented from starting the conflicting service. The idea behind this setup is that you don't have to hard-code the service URLs in the workflows. This means you can easily switch out services or move them. The user name and password is what is used for authentication in the requests. As mentioned earlier the external service is expected to verify that these match.

The operations are currently configured on a separate tab and it looks like this. Operations are added by clicking an add button. The properties for each operation are the following:

Open

• operation name

• operation type

• override remote operation name

The operation name is a unique name for the operation. It must be unique to the service in terms of type and arguments. Currently however only one type of operation is supported and it does not have any arguments. This means that in practice right now only the name must be unique. The operation type may currently just have the following value: Generic incomplete form answer population. This basically means that what is sent for population does not have to be valid. The override remote operation name means that what is appended to the base URL may be changed to something different. This can e.g. be handy if URL parameters are needed – the operation names exposed to the rest of the system have a naming policy preventing this, as it is planned that the operation names will be used directly in scripting for certain functionality. A design like this also helps with externalizing configurations from the workflows.
Once everything is configured correctly you only need to save and start the service.

3.3 Service use in workflows

To configure form population for a form you need to build a workflow and select the form activity where you want this to happen. On the form activity you have an option called populate via service that you can check. You can then either browse for running services or enter their mount point name and the operation you want to use. Once filled out correctly it can look something like this (here a very simplified workflow):

There is an additional box that may be checked on form population called block progress if population fails. This means that only if there were no errors will you be given access to the form. Depending on situation you then have the options of retrying or going back. It should be noted that for most cases of form population you also want to set some values in the form. This can be e.g. the registration number of the vehicle you are looking for, that has been entered in an earlier form, or maybe the social security number provided through some authentication method. In order to do achieve this you click the pre-populate Question answers button and select what you want to set.
After everything is set up properly you need to save and publish the workflow and then start it. Provided everything is done correctly you can then test your form population service.

Open

4 Final words

We started off by talking about forms and how they work. We have then gone through how population services are created. Finally we have talked about how you configure and use them in Compose. In the future there will be more services for Compose, but already this offers many new possibilities.

Appendix

Example request:

Example success response:

Example failure response:

Example object not found failure response: