# Telescope:  Improving Zero Shot Detection of LLM Generated Content With Token Repetition #

Paper: **TODO** (too large)

Datasets: **TODO** (too large)

Experiment Results: **TODO** (too large)


## Installation Steps ##

Once you have installed all of the following components, then you can start analyzing/ performing experiments in this repository


### Installing Required Packages: ###

**1)** install miniconda on your computer (follow the instructions online)

**2)** ```conda create -f telescope_env.yml```

**3)** ```conda init```

**4)** ```conda activate telescope```

Using these installation instructions, you should have all of the required pip packages to run any script in this repository. If not, please open an issue about it, and we will promptly resolve it.



### Registering a Hugging Face Authentication Token ###

You may need a hugging face authentication token to access and inference models that were used in this work. Make sure that you set up a hugging face authentication token through their website and save it somewhere. Next, taken that token and run the following command where you replace ********* with your hugging face authentication token:

```echo ********* >> hugging_face_auth_token.txt```

NOTE: this only works in a unix environment. If you are on Windows, then ensure that you just create a file named "hugging_face_auth_token.txt" and add your authentication token in plain text to that file. 


### Installing the Experiment Results and Datasets ###
Full experiment_results can be found here: **TODO** (too large)

Full datasets can be found here: **TODO** (too large)


To download the experiment results properly, simply run the following command in the root directory of this repository:

```git clone TODO experiment_results```

To download the datasets properly, simply run the following command in the root directory of this repository:

```git clone TODO datasets```






## Vocabulary Used Throughout the Codebase: ##


### What is a Metric? ###

A metric is simply a number computed from a reference model's outputs using some formula. For example, the Binoculars Score, Telescope Perplexity, Perplexity, and DetectLLM LRR are all examples of metrics; however, during the experimentation phase of this project we collected many more, which are all calculated in Telescope/telescope.py. Generally, we only care about metrics that are correlated with whether or not a piece of text was LLM generated or not/


### What is an Experiment Result? ###

An experiment result is a csv file containing various data collected from running each LLM text detection algorithm on a specific dataset with a specific reference model. This data includes the original text, whether or not the original text was LLM generated, and a variety of metrics that were computed while collecting the experiment result. Feel free to look inside of the experiment_results csv's to check what the format for the data is.


### What are Codenames and Display Names? ###

In order to uniquely identify models/ datasets/ metrics while following standard file naming conventions and also have paper ready versions of these names, we have decided to create a system called codenames and display names to identify them. Codenames are the versions of names spaced with underscores and are completely lowercase, whereas display names are formatted how we wanted them to look like in the final paper. Ex. Codename: "telescope_perplexity", Proper Name: "Telescope Perplexity". 



### What is a Performer Model and Reference Model? ###

These are vocabulary that was initially coined by Binoculars in their paper, so if you would like more information, then you may want to check out their work. Essentially, when we calculate the Binoculars score, we need to compute a perplexity and a cross perplexity, there is one model that needs to compute the perplexity **and** cross perplexity while the other model is only needed for the cross perplexity. The model that is used to compute both the perplexity and cross perplexity is known as the performer model and the other model is the observer model. Once again, please check out Binoculars' work for more information, there are some additional nuances in how you compute the cross perplexity etc, which the Binocular's paper covers quite extensively.

If we are dealing with a single reference model technique/ metric, then that reference model is by default a "performer model". This is a fairly arbitrary convention, but we decided to choose one and stick with it to make the code more consistent.




### What is the "config.yaml" File? ###

The config.yaml file contains various global variables that multiple files may need to access. An example of something stored in the config.yaml file is the conversion lookup table between a model/ dataset/ metric's codename and its display name.



### Running Experiments vs Analyzing Experiments: ###

*Running experiments* is the process of running a reference model to calculate what the Telescope Perplexity, Binoculars Perplexity, etc of a piece of text is and then saving that information to an experiment result. There is only one file that is responsible for running the experiments, and that is *generate_experiment_results.py*.


*Analyzing experiments* is the process of taking an experiment result and generating roc curves, calculating f1-scores, calculating transfer characteristics, and generating useful plots for data visualization from those experiment results. Remember, experiment results do not calculate the auroc or any other scores that gauge how good a metric is at detecting LLM text; experiment results only contain the raw values for each metric (Telescope Perplexity, Binoculars Score, etc). There are many scripts in this repository that analyze experiments such as *generate_adversarial_perturbation_chart.py*, *generate_threshold_transfer_plot.py*, etc.



### Why is There a Distinction Between Running Experiments and Analyzing Experiments? ###

The main reason there is such a big distinction between these two ideas in our codebase is because running experiments takes a long time and most of the time, people will not have the time or compute to run these reference models thousands of times just to perform their own analyses on the performance of Telescope Perplexity vs other LLM detection metrics. So, we have went out of our way to save the results of all of our experiments to make it easier for colleagues to come to their own conclusions from our raw data.



### What is Telescope Score? ###

This is an older metric that we were initially testing which is simply the Telescope Perplexity divided by the Cross Perplexity. This is a 2 reference model technique (because we need to be able to compute the cross perplexity between 2 models) and initial results found that while this was sometimes better than Telescope Perplexity when transfering to new classification thresholds, the same dataset performance was *significantly worse*, so to save space in the paper, we decided to omit it from the paper. Readers are encouraged, however, to analyze the performance of this metric and come to your own conclusions about it. 


### What Are Those Additional Metrics in Each Experiment Result and That Are Implemented in "telescope.py"? ###

Those are just metrics that we were testing whenever we initially performing large scale tests. We thought that maybe they might form decent LLM generated text detectors, but with our initial testing, none of them seemed nearly as promising as the Telescope Perplexity. However, feel free to explore them and perform your own analyses on their performance as LLM generated text detectors!