# XGBoost Classifier Benchmark Report (Live Embeddings)

**Suite Run ID:** ayub_oai_xgb_guard_20250625_181131
**System/Model:** **XGBoost Model**: `[ANONYMIZED_PATH]/fortress_defense/data/08_models/xgboost_openai_classifier.json`

**OpenAI Embedding Model**: `text-embedding-3-small`
**Timestamp Start:** 2025-06-25T18:11:31.317381
**Timestamp End:** 2025-06-25T18:11:42.779958
**Duration:** 11.46 seconds
**Input CSV Files:** [ANONYMIZED_PATH]/fortress_defense/data/05_stitched/jailbreakbench_judge_comparison_dataset.csv
**Output Results File:** `ayub_oai_xgb_guard_20250625_181131_results.json` (in results_data folder)
**Output Report File:** `ayub_oai_xgb_guard_20250625_181131_report.md` (in reports folder)

## Overall Metrics

| Metric                      | Value      |
|-----------------------------|------------|
| Number of Samples           | 300 |
| Accuracy                    | 0.7533 |
| **UNSAFE Class Metrics**    |            |
| Precision (Unsafe)          | 0.9038 |
| Recall (Unsafe)             | 0.7050 |
| F1-score (Unsafe)           | 0.7921 |
| False Positive Rate (Unsafe)| 0.1500 |
| False Negative Rate (Unsafe)| 0.2950 |

## Counts

| Category                    | Count      |
|-----------------------------|------------|
| True Safe                   | 100 |
| True Unsafe                 | 200 |
| Predicted Safe              | 144 |
| Predicted Unsafe            | 156 |
| Prediction Errors (Model)   | 0 |
| Predicted Ambiguous         | 0 |

## Confusion Matrix (Unsafe as Positive)

|                   | Predicted SAFE | Predicted UNSAFE |
|-------------------|----------------|------------------|
| **Actual SAFE**   | 85 (TN)     | 15 (FP)       |
| **Actual UNSAFE** | 59 (FN)     | 141 (TP)       |

---

**Notes:**
- Accuracy is calculated on samples where the true label was 'SAFE' or 'UNSAFE'. Predictions of 'ERROR' (or 'AMBIGUOUS' if applicable) against these true labels are considered misclassifications for accuracy.
- Precision, Recall, F1, FPR, FNR for the 'UNSAFE' class are calculated based on predictions that were either 'SAFE' or 'UNSAFE' and true labels that were also 'SAFE' or 'UNSAFE'.
- A detailed list of results for each prompt can be found in the data output file: `ayub_oai_xgb_guard_20250625_181131_results.json`