# WebAgent
# Instructions for Applying Code Changes in WebArena and BrowserGym

## Overview
This guide outlines the changes that need to be applied to specific files in the WebArena and BrowserGym packages. The modifications improve functionality, fix bugs, and add new features, such as fuzzy matching, user feedback, and PDF content extraction. Follow the steps below carefully to implement these changes.

### Files to Modify:
1. **webarena/evaluation_harness/evaluators.py**
2. **browsergym/webarena/task.py**
3. **browsergym/core/env.py**

---

## 1. Modify `evaluators.py`

**Location**: `/opt/anaconda3/envs/workarena/lib/python3.11/site-packages/webarena/evaluation_harness/evaluators.py`

### Add the following lines:
```python
for approach, value in configs["eval"]["reference_answers"].items():
    match approach:
        case "exact_match":
            score *= self.exact_match(ref=value, pred=pred)

            # Added by Ido
            # Initialize the scores to 0
            ua_match_score = 1.0
            fuzzy_match_score = 0.0
            if score != 1:
                # Overwrite and always return fuzzy match score
                ua_match_score = 1.0 * self.ua_match(
                    intent=configs["intent"],
                    ref=configs["eval"]["reference_answer_raw_annotation"],
                    pred=pred,
                )
                # Overwrite and always return fuzzy match score
                # if value is a list
                if isinstance(value, list):
                    for reference in value:
                        fuzzy_match_score *= self.fuzzy_match(
                            ref=reference, pred=pred, intent=configs["intent"]
                        )
                else:
                    fuzzy_match_score = 1.0 * self.fuzzy_match(
                        ref=value, pred=pred, intent=configs["intent"]
                    )
                score = max(ua_match_score, fuzzy_match_score, score)
```

---

## 2. Modify `task.py`

**Location**: `/opt/anaconda3/envs/workarena/lib/python3.11/site-packages/browsergym/webarena/task.py`

### Change the following block:
```python
with open(self.config_file, "r") as f:
    configs = json.load(f)
# llm_fuzzy_match() bugfix (assert "correct" in response)
except AssertionError as e:
    logger.info(
        "llm_fuzzy_match() bugfix applied: AssertionError in evaluator, using score = 0.0"
    )
    score = 0.0

if score > 0 or last_action["action_type"] == ActionTypes.STOP:
    return score, True, "", {'task': configs}
else:
    return score, False, "", {'task': configs}
```

---

## 3. Modify `env.py`

**Location**: `/opt/anaconda3/envs/workarena/lib/python3.11/site-packages/browsergym/core/env.py`

### Add imports:
```python
from PyPDF2 import PdfReader
from io import BytesIO
```

### In the `step()` and `reset()` functions, add:
```python
# extract obs and info from environment
obs = self._get_obs()
self.obs = obs  # Added to the original code
```

### In the `__init__` method of the `BrowserEnv` class, add:
```python
self.action_mapping_predefined = action_mapping_predefined
self.feedback_collecting = True if feedback_collecting else False
self.feedback = []
self.obs = None
```
#### Add to self.observation_space = gym.spaces.Dict():
```python
"read_page": gym.spaces.Sequence(
                    Unicode(min_length=0, max_length=TEXT_MAX_LENGTH)
                ),
```

### Add a new function for sending feedback:
```python
def send_message_to_user(text: str):
    if self.feedback_collecting:
        self.feedback.append(text)
    if 'action: ' in text:
        text = text.replace('action: ', '')
        self.chat.add_message(role="assistant", msg=text)
```

### Update the action execution code:
```python
# try to execute the action
logger.debug(f"Executing action")
try:  # Added to the original code
    if self.action_mapping_predefined:
        code = self.action_mapping_predefined + '\n\n' + action
    elif self.action_mapping:
        code = self.action_mapping(action)
    else:
        code = action
    execute_python_code(
        code,
        self.page,
        send_message_to_user=send_message_to_user,
        report_infeasible_instructions=report_infeasible_instructions,
    )
```

### Add a new function to read webpage content:
```python
def read_webpage_content(self) -> str:
    start_time = time.time()
    url = self.page.url

    try:
        self.page.wait_for_load_state('networkidle')

        if 'pdf' in url.lower():
            pdf_content = BytesIO(self.page.content())
            reader = PdfReader(pdf_content)
            content = reader.pages[0].extract_text()
        else:
            self.page.wait_for_selector("body")
            content = self.page.evaluate('() => document.body.innerText')

        if not content.strip():
            content = self.page.inner_text('body') or self.page.evaluate('() => document.body.textContent') or ""

        return content

    except Exception as e:
        print(f"Error reading page: {str(e)}")
        return ""

    finally:
        print(f"Execution time: {time.time() - start_time:.2f} seconds")
```

### Modify `_get_obs` to include `read_page`:
```python
obs = {
    "chat_messages": copy.deepcopy(self.chat.messages),
    "goal": goal_msg,  # TODO: redundant with chat messages, to be removed?
    "goal_image_urls": goal_image_urls,  # TODO: redundant with chat messages, to be removed?
    "open_pages_urls": [page.url for page in self.context.pages],
    "active_page_index": np.asarray([self.context.pages.index(self.page)]),
    "url": self.page.url,
    "screenshot": extract_screenshot(self.page),
    "dom_object": dom,
    "axtree_object": axtree,
    "extra_element_properties": extra_properties,
    "focused_element_bid": focused_element_bid,
    "last_action": self.last_action,
    "last_action_error": self.last_action_error,
    "elapsed_time": np.asarray([time.time() - self.start_time]),
    "read_page": self.read_webpage_content(),
}
```

---

## Conclusion

Once all changes have been implemented, test the system by running it through a few tasks to ensure that fuzzy matching, PDF extraction, and feedback collection function as expected.

Feel free to reach out if you encounter any issues or need further assistance!
```