Now we’ll retrain train the model. You can choose between retraining training texts from your uploaded file or upload annotated data as a new file.

...

Panel

Select Action:
Status
title ModelModelS
from the side navigation
Select your model Select Action: from the drop-down
Go to the
Status
title Update
tab
Select Mode:
Status
colour Yellow
title request
and select how many instances you want to annotate.
Select sampler:
1. Random: Returns random text instances from your dataset
2. Margin
  Status
  colour Green
  title recommended
  : A metric to find uncertain instances for the underlying model
Then click the button “Request”. Request
Image Removed

...

Annotate selected instances

Now you want to annotate the requested instances. To do that, proceed as followingfollows:

Select Mode:
Status
title Label
Select label default
1. SKIP - Allows you to annotate the post by yourself
2. Model prediction - Allows you to see the model’s prediction.
For each given post, select the most suitable label or the SKIP option to skip a post annotation.
Click “Update” Update to retrain the model. This process takes a little while. You may see a message “Model "Model not ready. Come back later”later".

You can repeat this procedure multiple times.

...

Upload a file that has a column with post annotations
Select the text column
Select the label column (post annotations)
Click “Update” Update to retrain the model. This process takes a little while. You may see a message “Model "Model not ready. Come back later”later".
Image Removed

...

Model convergence estimation

After the model has been trained at least one time, the user will be able to monitor the evolution of the quality model performance.

The platform provides a way to monitor model convergence. That is, it answers the question if annotating more will further improve the model.

...

With your model selected, select action go to the
Status
title inspectInspect
tab.
After each iteration, a new point will be drawn on the "estimated quality performance" line plot. You can deduce from it that, when the curve starts to flatten, the model quality is converging, and probably it's not going to learn much more.
Image Removed

...

Using a different dataset than from the one the model was created trained with, you can evaluate the model’s performance.

Prepare your data file with the following columns:
1. Text: Your raw text instances
2. Label: Label annotations for each text
Select Action: With your model selected, go to the
Status
title Evaluateevaluate
tab.
Upload your file
Select the text column and label column
As a result, you’ll see a table with different performance metrics and a list of text instances (Errors) where the predicted label and the annotated label mismatch below it.
Image Removed

Image Added

Info

In case the results are not sufficient, there might be multiple reasons:

you might have to revise your labels and hypotheses and use Promptranker to create another model in order to achieve better results
your evaluation set is not good enough - the distribution in the examples in it is not equal (e.g. in the example above in the evaluation set we had only 30 examples for special occasions but 148 for social gatherings, and respectively the latter got better results)

...