Now we’ll train the model. You can choose between training texts from your uploaded file or upload annotated data as a new file.

Option 1: Annotate selected instances from the uploaded file

Select instances to annotate from the uploaded text

Select MODELS from the side navigation
Select your model from the drop-down
Go to the UPDATE tab
Select Mode: REQUEST and select how many instances you want to annotate.
Select sampler:
1. Random: Returns random text instances from your dataset
2. Margin RECOMMENDED : A metric to find uncertain instances for the underlying model
Then click the button Request

Now you want to annotate the requested instances. To do that, proceed as follows:

Select Mode: LABEL
Select label default
1. SKIP - Allows you to annotate the post by yourself
2. Model prediction - Allows you to see the model’s prediction.
For each given post, select the most suitable label or the SKIP option to skip a post annotation.
Click Update to retrain the model. This process takes a little while. You may see a message "Model not ready. Come back later".

You can repeat this procedure multiple times.

Upload a file that has a column with post annotations
Select the text column
Select the label column (post annotations)
Click Update to retrain the model. This process takes a little while. You may see a message "Model not ready. Come back later".

After the model has been trained at least one time, the user will be able to monitor the evolution of the model performance.

The platform provides a way to monitor model convergence. That is, it answers the question if annotating more will further improve the model.

This is done by means of a regression model that estimates a normalized F1-score on unseen test data.

With your model selected, go to the INSPECT tab.
After each iteration, a new point will be drawn on the line plot. You can deduce from it that, when the curve starts to flatten, the model quality is converging, and probably it's not going to learn much more. If the curve is not stable, you have to continue training the model as it is still learning.

Using a different dataset from the one the model was trained with, you can evaluate the model’s performance.

Prepare your data file with the following columns:
1. Text: Your raw text instances
2. Label: Label annotations for each text
With your model selected, go to the EVALUATE tab.
Upload your file
Select the text column and label column
As a result, you’ll see a table with different performance metrics and a list of text instances (Errors) where the predicted label and the annotated label mismatch below it.

In case the results are not sufficient, there might be multiple reasons:

you might have to revise your labels and hypotheses and use Promptranker to create another model in order to achieve better results
your evaluation set is not good enough - the distribution in the examples in it is not equal (e.g. in the example above in the evaluation set we had only 30 examples for special occasions but 148 for social gatherings, and respectively the latter got better results)