LiveChat Experiments API: Let's Test!

LiveChat Experiments API

A few months ago LiveChat released a new feature (the onboarding checklist) that should have simplified tasks that had not been satisfactorily handled before. To our astonishment, the conversion rate dropped. We didn’t know whether it was the result of the new feature or some other coinciding factors. We didn’t set any control group as we were sure that the new feature would only make the things better.

This was when we realized that we should incorporate **A/B testing **into the deployment process for almost every change in the LiveChat product. We decided to create our own tool for managing A/B testing and this is how **LiveChat Experiments API** was born.

Measured to fit and tailored to measure

Why did we decide to come up with a custom solution, when there are more advanced third-party tools on the market? The reason was simple. These tools were primarily dedicated to frontend applications and we wanted something more integrated with LiveChat’s backend.

This way we could integrate a third-party tool with our solution and track our customers’ progress from the first visit on our website, through signing up for a trial, to the conversion to the paid version of LiveChat. Furthermore, creating the tool in-house allowed us to tailor it to our specific needs. We wanted, however, to make it generic and universal, so that we could use it in other projects.

So, how does LiveChat Experiments API work, then?

Thanks to LiveChat Experiments API we can set up an A/B test for any number of LiveChat licenses and with numerous variations and conditions.

We used it, for example, to test a new set of default LiveChat greetings (chat invitations).

LiveChat Experiments API - LiveChat Greetings LiveChat greetings appear when a user has performed a certain action on the website or spent there a specific amount of time.We wanted to measure if the time when a greeting appears affects user behavior. Let’s take a look at the details to see what Experiments API is all about.

LiveChat Experiments API is built around six basic concepts: Experiment, Subject, Population, Dependency, Variation, and Action. In our greetings test they break down like this:

1. Experiment

An Experiment is a single A/B test. In this case, it was a default greetings test. We presented various sets of differently timed greetings to our test groups to measure the conversion rate. Set Four and Five made it to the final round.

Set Four greetings:

Displayed after 10 seconds on the website
Displayed after 7 seconds on the website to a returning visitor
Displayed after a user has visited 3 subpages

Set Five greetings:

Displayed after a user has returned immediately after leaving the website
Displayed after 15 seconds on the website
Displayed after a user has spent 10 seconds on the website and visited 3 subpages
Displayed after a user has spent 5 seconds on the website and visited 5 subpages
Displayed after 1 minute on the website

Code

curl -X POST \
  http://*****.livechatinc.com/*****/defaultGreetingsSet5 \
  -H 'authorization: *******' \
  -H 'content-type: application/json' \
  -d '{
    "description": "Default greetings set no. 5",
    "population": 100,
    "start_timestamp":1508371200,
    "apply_to_new_subject":1,
    // ...

2. Subject

A subject is an entity that participates in an Experiment. It must be an entity with a unique identifier: a license, an account, a user, or a visitor. You can assign a Subject to the Experiment either manually or automatically.

In our Experiment, we examined LiveChat licenses.

3. Population

A Population is the part of all Subjects that qualified for an Experiment (see Dependency below) and will actually participate in it.

83,825 LiveChat licenses in total took part in the greetings Experiment.

4. Dependency

A Dependency is a condition that must be met to qualify a Subject for an Experiment. This can be, for example, Subject type, a parameter, etc.

Only a defined fraction of Subjects that fulfills the Dependencies conditions qualifies to participate in the Experiment. After the qualification process, the Subjects are randomly assigned to Variations (see below) in the specified proportions (even or custom).

We selected new trial LiveChat licenses to take part in the Experiment.

Code

    "dependencies": [
        {
            "type":"param_value",
            "params":[
                {
                    "name":"lc_version",
                    "value":"2"
                }
            ]
        },
        {
            "type":"subject_type",
            "params":[
                {
                    "name":"subject_type",
                    "value":"license"
                }
            ]
        }
    ],

5. Variation

A Variation is a part of the Population of Subjects that corresponds to one of the cases of the Experiment that we test.

In our Experiment, we presented new and old greetings to the Subjects in equal proportions (50/50).

Code

    "variations": [
        {
            "variation": 1,
            "percentage": 50
        },
        {
            "variation": 2,
            "percentage": 50
        }
    ],

6. Action

An action is an optional operation performed directly after a Subject has been assigned to an Experiment or unassigned from it.

In our case, setExperimentLicenceProp added a custom property to a Subject so that we know that it took part in the Experiment.

Code

"actions": [
        {
            "name":"setExperimentLicenceProp",
            "variation":0,
            "type":"setExperimentLicenceProp",
            "for_assignment":1
        },
        {
            "name":"removeExperimentLicenceProp",
            "variation":0,
            "type":"removeExperimentLicenceProp",
            "for_assignment":0
        },
        {
            "name":"setGreetingsSet4",
            "variation":1,
            "type":"setDefaultGreetingsSet",
            "for_assignment":1,
            "params": [
                {
                    "name":"greetings_set",
                    "value":"4"
                }
            ]
        },
        {
            "name":"setGreetingsSet5",
            "variation":2,
            "type":"setDefaultGreetingsSet",
            "for_assignment":1,
            "params": [
                {
                    "name":"greetings_set",
                    "value":"5"
                }
            ]
        }
    ]

See full code

curl -X POST \
  http://*****.livechatinc.com/*****/defaultGreetingsSet5 \
  -H 'authorization: *******' \
  -H 'content-type: application/json' \
  -d '{
    "description": "Default greetings set no. 5",
    "population": 100,
    "start_timestamp":1508371200,
    "apply_to_new_subject":1,
    "variations": [
        {
            "variation": 1,
            "percentage": 50
        },
        {
            "variation": 2,
            "percentage": 50
        }
    ],
    "dependencies": [
        {
            "type":"param_value",
            "params":[
                {
                    "name":"lc_version",
                    "value":"2"
                }
            ]
        },
        {
            "type":"subject_type",
            "params":[
                {
                    "name":"subject_type",
                    "value":"license"
                }
            ]
        }
    ],
    "actions": [
        {
            "name":"setExperimentLicenceProp",
            "variation":0,
            "type":"setExperimentLicenceProp",
            "for_assignment":1
        },
        {
            "name":"removeExperimentLicenceProp",
            "variation":0,
            "type":"removeExperimentLicenceProp",
            "for_assignment":0
        },
        {
            "name":"setGreetingsSet4",
            "variation":1,
            "type":"setDefaultGreetingsSet",
            "for_assignment":1,
            "params": [
                {
                    "name":"greetings_set",
                    "value":"4"
                }
            ]
        },
        {
            "name":"setGreetingsSet5",
            "variation":2,
            "type":"setDefaultGreetingsSet",
            "for_assignment":1,
            "params": [
                {
                    "name":"greetings_set",
                    "value":"5"
                }
            ]
        }
    ]
}'

The results of our experiments showed in hard numbers that the new greetings convert better than the old ones. Therefore, we had reliable data to base on when we updated the default greetings in our product.

Further notes

As mentioned above, Subjects can be assigned to an Experiment automatically or manually. The manual assignment gives you full control over which Subjects should participate in the Experiment with the specific Variations. This is the case when, if you need, you can use the data from a third-party A/B testing tool that already operates on your website. In turn, the automatic assignment is when LiveChat Experiments API makes all the work for you, according to the rules set up in advance.

After a Subject is assigned (manually or automatically), the Actions (if specified) are performed. An Action can add any property to a Subject and enable or disable its features. This is when close integration with the LiveChat environment pays off big time. From now on you can just sit back and wait for the Subjects to accumulate in Experiment’s Population and analyze the data.

Test, improve, then test again

Since the first iteration of LiveChat Experiments API, we have released a new version of default greetings, the onboarding chat tutorial, and a new layout of the subscription page. All these had a positive impact on conversion rate and the number of chats because we were able to make informed decisions thanks to A/B testing. Although in many cases you may think that you know what is best for your customers just by following your gut, it is always better to support your claims with hard numbers.