Model Evaluation - Case Studies¶

1. Unsupervised model: See what unlabeled conditions correlate with a warning condition¶

Example using the continuous operations - 1 machine tutorial

Create unsupervised model according to tutorial instructions
Evaluate the model with ranges=\"Unsupervised Training Period\" and facts=\"all events\".
Go to the evaluation results page. Select \"warning\" condition and see below chart. This indicates that \"unlabeled4\" is most correlated with the warning condition.

You can also select \"unlabeled4\" condition and see below chart

2. Supervised model: See per-condition metrics and confusion¶

Example using the continuous operations - 1 machine tutorial

Create semi-supervised model according to tutorial instructions
Evaluate the model with ranges=\"Unsupervised Training Period\" and facts=\"all events\". Then go to the Evaluation page.
Select \"maintenance\" condition and see below charts. Note how precision is high 100% and recall is relatively lower 77%. Since \"unlabeled4\" still shows up in the recall chart, then the user should increase the generalization factor. Also the confusion between maintenance and warning indicates that maybe more facts are needed around the transition regions to better separate those two conditions.

3. Supervised model: See which conditions are unrecognizable¶

This is a contrived case study using the continuous operations tutorial as a starting point. I have split the normal condition into \"normal1\" and \"normal2\". This was done as an arbitrary split that Falkonry won\'t be able to distinguish/recognize.

Upload \"bad training events\":

| time,end,entity,value | \"1455757521745000\",\"1456074759476000\",\"machine1\",\"normal1\" | \"1456116138311000\",\"1456147172437000\",\"machine1\",\"warning\" | \"1456157517145000\",\"1456178206562000\",\"machine1\",\"maintenance\" | \"1456221884221000\",\"1456683947872000\",\"machine1\",\"normal2\" | \"1456746016124000\",\"1456772452602000\",\"machine1\",\"warning\" | \"1456779349074000\",\"1456791992607000\",\"machine1\",\"maintenance\"

Create semi-supervised model according to tutorial instructions, but using \"bad training events\" instead of \"training events\"
Evaluate the model with ranges=\"Unsupervised Training Period\" and facts=\"all events\".
Notice the relatively low condition accuracies for \"normal1\" and \"normal2\", meaning that Falkonry has difficulty recognizing these two conditions. This means that the user should supply more training data or signals, or modify the model learn parameters to help Falkonry dis-ambiguate the conditions. Or, as in this example, maybe the facts are just incorrect.

Note the condition accuracies with the \"correct\" events