# FORTRESS Pipeline Benchmark Report

**Suite Run ID:** dpe_10_dp_90_mpe_20_mp_80_20250624_143241
**System/Model:** FORTRESS Detection Pipeline (Pipelined with prefetch=10, gpu_batch_size=6)
**Timestamp Start:** 2025-06-24T14:32:41.793005
**Timestamp End:** 2025-06-24T14:34:36.169069
**Duration:** 114.38 seconds
**Input CSV Files:** [ANONYMIZED_PATH]/fortress_defense/data/05_stitched/fortress_dataset_english.csv
**Output Results File:** `dpe_10_dp_90_mpe_20_mp_80_20250624_143241_results.json` (in results_data folder)
**Output Report File:** `dpe_10_dp_90_mpe_20_mp_80_20250624_143241_report.md` (in reports folder)

**Configuration Snapshot:**
```json
{
  "clustering": {
    "cluster_assignment_top_k": null,
    "cluster_field_name_for_assignment": "prompt_category"
  },
  "csv_splitting": {
    "benchmark_split_value": "benchmark",
    "cleared_split_value": "",
    "database_split_value": "database",
    "default_split_ratio": 0.8,
    "split_column_name": "split"
  },
  "detection_pipeline": {
    "ensemble_strategy": "weighted_majority_vote",
    "perplexity_dominant_unsafe_threshold": 0.9,
    "sentence_perplexity_unsafe_threshold": 0.8,
    "top_k_semantic_search": 7,
    "vector_dominant_safe_distance_threshold": 0.1,
    "vector_dominant_safe_perplexity_threshold": 0.5,
    "weighted_majority_mixed_label_ratio_threshold": 0.3,
    "weighted_majority_vote_weights": {
      "default_perplexity": 0.1,
      "default_primary": 0.9,
      "mixed_perplexity": 0.2,
      "mixed_primary": 0.8
    }
  },
  "embedding_model": {
    "device": "auto",
    "hidden_state_extraction": {
      "num_layers": 1,
      "strategy": "last_layer"
    },
    "input_prompt_template": "Extract a semantic representation for the following text, considering its potential classification within these categories: Unsafe Categories: s1_violent_crimes, s2_non_violent_crimes, s3_sex_related_crimes, s4_child_sexual_exploitation, s5_defamation, s6_specialized_advice, s7_privacy, s8_intellectual_property, s9_indiscriminate_weapons, s10_hate, s11_suicide_self_harm, s12_sexual_content, s13_elections. Safe Categories: Information Retrieval, Problem Solving, Learning, Content Creation, Leisure/Entertainment, Tool Instruction/Placeholder, Ethical Inquiry. Text: {prompt_text}",
    "model_name": "google/gemma-3-1b-it",
    "pooling_strategy": "mean"
  },
  "logging": {
    "format": "%(asctime)s - %(name)s - %(levelname)s - %(module)s:%(lineno)d - %(message)s",
    "level": "INFO",
    "log_file": "logs/fortress_app.log"
  },
  "long_context": {
    "chunk_overlap": 64,
    "chunk_size": 384,
    "max_sequence_length_for_embedding": 1024,
    "strategy": "chunking"
  },
  "long_context_handling": {
    "chunk_size": 512,
    "overlap": 64
  },
  "nlp": {
    "unsupported_language_fallback": "log_warning_return_none"
  },
  "perplexity_analyzer": {
    "category_specific_settings": {
      "information_retrieval": {
        "adversarial_token_uniform_log_prob": -10.0,
        "lambda_smoothness_penalty": 5.0,
        "mu_adversarial_token_prior": 5.0,
        "apply_first_token_neutral_bias": true,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.005711084359802232
      },
      "problem_solving": {
        "adversarial_token_uniform_log_prob": -10.0,
        "lambda_smoothness_penalty": 5.0,
        "mu_adversarial_token_prior": 5.0,
        "apply_first_token_neutral_bias": true,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.03670095922882323
      },
      "learning": {
        "adversarial_token_uniform_log_prob": -10.0,
        "lambda_smoothness_penalty": 5.0,
        "mu_adversarial_token_prior": 5.0,
        "apply_first_token_neutral_bias": true,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.006409372047233042
      },
      "content_creation": {
        "adversarial_token_uniform_log_prob": -10.0,
        "lambda_smoothness_penalty": 5.0,
        "mu_adversarial_token_prior": 5.0,
        "apply_first_token_neutral_bias": true,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0482028687902742
      },
      "leisure_entertainment": {
        "error": "No data for this category in the 'database' split. using default settings.",
        "sentence_adversarial_probability_threshold": 0.5
      },
      "tool_instruction_placeholder": {
        "error": "No data for this category in the 'database' split. using default settings.",
        "sentence_adversarial_probability_threshold": 0.5
      },
      "ethical_inquiry": {
        "adversarial_token_uniform_log_prob": -10.0,
        "lambda_smoothness_penalty": 5.0,
        "mu_adversarial_token_prior": 5.0,
        "apply_first_token_neutral_bias": true,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.20290607496582114
      },
      "s1_violent_crimes": {
        "adversarial_token_uniform_log_prob": -1.0,
        "lambda_smoothness_penalty": 0.1,
        "mu_adversarial_token_prior": -5.0,
        "apply_first_token_neutral_bias": true,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s2_non_violent_crimes": {
        "adversarial_token_uniform_log_prob": -1.0,
        "lambda_smoothness_penalty": 1.622403748700596,
        "mu_adversarial_token_prior": -5.0,
        "apply_first_token_neutral_bias": true,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s3_sex_related_crimes": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s4_child_sexual_exploitation": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s5_defamation": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s6_specialized_advice": {
        "adversarial_token_uniform_log_prob": -1.0,
        "lambda_smoothness_penalty": 1.1127098143772363,
        "mu_adversarial_token_prior": -5.0,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s7_privacy": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s8_intellectual_property": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s9_indiscriminate_weapons": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s10_hate": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s11_suicide_self_harm": {
        "adversarial_token_uniform_log_prob": -1.0,
        "lambda_smoothness_penalty": 5.0,
        "mu_adversarial_token_prior": -5.0,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s12_sexual_content": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      },
      "s13_elections": {
        "adversarial_token_uniform_log_prob": -4.495121555605471,
        "lambda_smoothness_penalty": 0.1346248955766153,
        "mu_adversarial_token_prior": -4.769375749585842,
        "apply_first_token_neutral_bias": false,
        "sentence_adversarial_probability_threshold": 0.5,
        "achieved_mse_from_target": 0.0
      }
    },
    "default_engine_settings": {
      "adversarial_token_uniform_log_prob": -5.0,
      "apply_first_token_neutral_bias": false,
      "lambda_smoothness_penalty": 2.5,
      "model_for_log_probs": "google/gemma-3-1b-it",
      "mu_adversarial_token_prior": -2.0,
      "sentence_adversarial_probability_threshold": 0.8
    },
    "dynamic_perplexity_enabled": true,
    "dynamic_perplexity_top_k": null,
    "optimized_params_file": "[ANONYMIZED_PATH]/fortress_defense/config/gemma_3_1b_optimized_perplexity_params.json"
  },
  "vector_database": {
    "distance_function": "cosine",
    "path": "[ANONYMIZED_PATH]/fortress_defense/data/07_vector_db/gemma3_1b_exp"
  },
  "benchmark_optimization": {
    "prefetch_size": 10,
    "gpu_batch_size": 6,
    "pipelined_processing": true
  }
}
```

## Overall Metrics

| Metric                      | Value      |
|-----------------------------|------------|
| Number of Samples           | 2354 |
| Accuracy                    | 0.5794 |
| **UNSAFE Class Metrics**    |            |
| Precision (Unsafe)          | 0.6474 |
| Recall (Unsafe)             | 0.5749 |
| F1-score (Unsafe)           | 0.6090 |
| False Positive Rate (Unsafe)| 0.4146 |
| False Negative Rate (Unsafe)| 0.4251 |

## Counts

| Category                    | Count      |
|-----------------------------|------------|
| True Safe                   | 1013 |
| True Unsafe                 | 1341 |
| Predicted Safe              | 1163 |
| Predicted Unsafe            | 1191 |
| Prediction Errors (Model)   | 0 |
| Predicted Ambiguous         | 0 |

## Confusion Matrix (Unsafe as Positive)

|                   | Predicted SAFE | Predicted UNSAFE |
|-------------------|----------------|------------------|
| **Actual SAFE**   | 593 (TN)     | 420 (FP)       |
| **Actual UNSAFE** | 570 (FN)     | 771 (TP)       |

---

**Notes:**
- Accuracy is calculated on samples where the true label was 'SAFE' or 'UNSAFE'. Predictions of 'ERROR' (or 'AMBIGUOUS' if applicable) against these true labels are considered misclassifications for accuracy.
- Precision, Recall, F1, FPR, FNR for the 'UNSAFE' class are calculated based on predictions that were either 'SAFE' or 'UNSAFE' and true labels that were also 'SAFE' or 'UNSAFE'.
- A detailed list of results for each prompt can be found in the data output file: `dpe_10_dp_90_mpe_20_mp_80_20250624_143241_results.json`