There is the data: atp_tennis containing the following columns: ['Tournament', 'Date', 'Series', 'Court', 'Surface', 'Round', 'Best of', 'Player_1', 'Player_2', 'Winner', 'Rank_1', 'Rank_2', 'Pts_1', 'Pts_2', 'Odd_1', 'Odd_2', 'score'].  
--- The description for each column this data is:
Tournament: Name of the tennis tournament (Brisbane International, Chennai Open, Qatar Exxon Mobil Open ...etc)
Date: Date the match was played (year-month-day)
Series: Category or level of the tennis tournament (ATP250, ATP500, Masters1000 and Grand Slams offer 250, 500, 1000, and 2000 ranking points to the winner seperately.)
Court: Place the match was held (Indoors or Outdoors)
Surface: Type of court surface (Hard, Grass and Clay)
Round: Stage of the tournament (1st Round, 2nd Round, Quarterfinals, Semifinal and The Final)
Best of: Tourament systems ("best of 3" or "best of 5")
Player_1: Name of the first competitor in each match
Player_2: Name of the second competitor in each match
Winner: Name of the Player_1 or Player_2 who won the match 
Rank_1: World rankings of the Player_1 at the time of the match
Rank_2: World rankings of the Player_2 at the time of the match
Pts_1: Ranking points held by Player_1 before the match (accumulated over the season and higher points typically indicate a more successful player)
Pts_2: Ranking points held by Player_2 before the match
Odd_1: Betting odds for Player_1 (reflecting their perceived chances of winning)
Odd_2: Betting odds for Player_2 
Score: Final score of the match ("score of Player_1" - "score of Player_2")
---
Considering contents from the dataset and requirements from user. Please note DO NOT CHANGE FILE AND VARIABLE NAMES THAT I HAVE SET!
Interactions begin:
[USER (data scientist)]: Hi, let's start by assessing the data quality of the atp_tennis dataset. To achieve this, we'll create a Data Quality Report DataFrame that covers two main aspects: a list of columns along with the number of missing values in each column, A list of unique values for each categorical column, such as Tournament, Series, Court, Surface, and Round.

[YOU (AI assistant)]: Sure, let's create a Data Quality Report Dataframe for the atp_tennis dataset. Here's the code to generate the report :
'''
import pandas as pd
import pickle

atp_tennis = pd.read_csv('atp_tennis.csv')

# Data Quality Report
missing_values = atp_tennis.isnull().sum()
unique_values = atp_tennis[['Tournament', 'Series', 'Court', 'Surface', 'Round']].nunique()

data_quality_report = pd.DataFrame({'Missing Values': missing_values, 'Unique Values': unique_values})

print(data_quality_report)

'''

[USER (data scientist)]: Great, We want to analyze the key statistics and distributions of player rankings, win/loss ratios, and surface types in the atp_tennis dataset across different tournaments and time periods. To do this, we'll create a Descriptive Statistics Table and Visualizations, including histograms, bar charts, and box plots.

[YOU (AI assistant)]: Sure, let's create a Descriptive Statistics Table and Visualizations for the atp_tennis dataset. Here's the code to generate the table and visualizations :
'''
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import pickle

atp_tennis = pd.read_csv('atp_tennis.csv')

# Count the number of wins for each player
player_wins = atp_tennis['Winner'].value_counts()
  
# Count the total number of matches played by each player
player_matches = atp_tennis['Player_1'].value_counts() + atp_tennis['Player_2'].value_counts()
  
# Calculate win/loss ratio for each player
win_loss_ratios = player_wins / (player_matches - player_wins)
  
# Add win/loss ratios to the dataset
atp_tennis['Win_Loss_Ratio_1'] = atp_tennis['Player_1'].map(win_loss_ratios)
atp_tennis['Win_Loss_Ratio_2'] = atp_tennis['Player_2'].map(win_loss_ratios)
  
# Descriptive Statistics Table
desc_stats = atp_tennis[['Rank_1', 'Rank_2', 'Win_Loss_Ratio_1', 'Win_Loss_Ratio_2']].describe()

# Combine Rank_1, Rank_2, Win_Loss_Ratio_1, and Win_Loss_Ratio_2 into single columns
combined_ranks = pd.concat([atp_tennis['Rank_1'], atp_tennis['Rank_2']]).reset_index(drop=True)
combined_win_loss_ratios = pd.concat([atp_tennis['Win_Loss_Ratio_1'], atp_tennis['Win_Loss_Ratio_2']]).reset_index(drop=True)
  
# Create a DataFrame with combined ranks and win/loss ratios
combined_data = pd.DataFrame({'Rank': combined_ranks, 'Win_Loss_Ratio': combined_win_loss_ratios})
  
# Descriptive Statistics Table
desc_stats = combined_data.describe()
  
# Histogram for Player Rankings Distribution
fig1, ax1 = plt.subplots()
ax1.hist(combined_ranks, bins=50, alpha=0.5)
ax1.set_title('Player Rankings Distribution')
ax1.set_xlabel('Ranking')
plt.savefig('./ref_result/hist_chart.png')
plt.show()
  
# Scatter Plot for Player Rankings vs Win/Loss Ratios
fig2, ax2 = plt.subplots()
ax2.scatter(combined_data['Rank'], combined_data['Win_Loss_Ratio'], alpha=0.5)
ax2.set_title('Player Rankings vs Win/Loss Ratios')
ax2.set_xlabel('Ranking')
ax2.set_ylabel('Win/Loss Ratio')
plt.savefig('./ref_result/scatter_chart.png')
plt.show()

# Bar Chart for Surface Types
fig3, ax3 = plt.subplots()
surface_counts = atp_tennis['Surface'].value_counts()
surface_counts.plot.bar(ax=ax3)
ax3.set_title('Matches by Surface Type')
ax3.set_xlabel('Surface')
ax3.set_ylabel('Number of Matches')
plt.savefig('./ref_result/bar_chart.png')
plt.show()

print(desc_stats)

'''

[USER (data scientist)]: Thank you, Tapilot. Now let's move on. We want to determine if there is a significant correlation between surface types (hard, clay, grass) and player performance (win/loss ratios, player rankings) in the atp_tennis dataset. To do this, we'll create a Correlation Matrix and perform Statistical Tests to obtain p-values and confidence intervals.

[YOU (AI assistant)]: Sure, let's create a Correlation Matrix and perform Statistical Tests for the atp_tennis dataset. Here's the code to generate the matrix and perform the tests :
'''
import numpy as np
import pandas as pd
import scipy.stats as stats
import pickle

atp_tennis = pd.read_csv('atp_tennis.csv')

# Create a DataFrame with surface types, player rankings, and win/loss ratios
surface_data = atp_tennis[['Surface', 'Rank_1', 'Rank_2', 'Win_Loss_Ratio_1', 'Win_Loss_Ratio_2']].copy()

# Remove rows with missing or invalid data in the Win_Loss_Ratio columns
surface_data_clean = surface_data.dropna(subset=['Win_Loss_Ratio_1', 'Win_Loss_Ratio_2']).copy()

# One-hot encoding for surface types
surface_data_clean = pd.get_dummies(surface_data_clean, columns=['Surface'], prefix='', prefix_sep='')

# Calculate the average player ranking and win/loss ratio for each match
surface_data_clean.loc[:, 'Avg_Rank'] = (surface_data_clean['Rank_1'] + surface_data_clean['Rank_2']) / 2
surface_data_clean.loc[:, 'Avg_Win_Loss_Ratio'] = (surface_data_clean['Win_Loss_Ratio_1'] + surface_data_clean['Win_Loss_Ratio_2']) / 2

# Create a Correlation Matrix
corr_matrix = surface_data_clean[['Hard', 'Clay', 'Grass', 'Avg_Rank', 'Avg_Win_Loss_Ratio']].corr()

# Perform Statistical Tests
anova_result_clean = stats.f_oneway(surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Hard'] == 1],
                                    surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Clay'] == 1],
                                    surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Grass'] == 1])

# Calculate the mean and standard error of the mean for each surface type
hard_mean = surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Hard'] == 1].mean()
hard_sem = stats.sem(surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Hard'] == 1])

clay_mean = surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Clay'] == 1].mean()
clay_sem = stats.sem(surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Clay'] == 1])

grass_mean = surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Grass'] == 1].mean()
grass_sem = stats.sem(surface_data_clean['Avg_Win_Loss_Ratio'][surface_data_clean['Grass'] == 1])

# Calculate the confidence intervals using the t-distribution
confidence_level = 0.95
degrees_of_freedom = len(surface_data_clean) - 1
t_critical = stats.t.ppf((1 + confidence_level) / 2, degrees_of_freedom)

hard_ci = (hard_mean - t_critical * hard_sem, hard_mean + t_critical * hard_sem)
clay_ci = (clay_mean - t_critical * clay_sem, clay_mean + t_critical * clay_sem)
grass_ci = (grass_mean - t_critical * grass_sem, grass_mean + t_critical * grass_sem)

print("Correlation Matrix:\n", corr_matrix)

print("P-value:", anova_result_clean.pvalue)

print("Confidence Intervals:\n", t_critical)

print("Hard SurnConfidenceface:", hard_ci)

print("Clay Surface:", clay_ci)

print("Grass Surface:", grass_ci)

'''

[USER (data scientist)]: We aim to examine the influence of different court surfaces on the performance of players in the ATP tennis dataset, categorized into top-ranked, mid-ranked, and low-ranked groups. This will be achieved through a Stratified Analysis, incorporating charts and statistical tests (p-values) to compare performance across these ranking groups and court surfaces. The top-ranked players are those ranked between 0 and 50, mid-ranked players are between 50 and 200, while any ranking beyond 200 is considered low-ranked. Given this information, could you please clarify which group of players appears to be most significantly affected by the type of court surface?

A. top-ranked
B. mid-ranked
C. low-ranked
D. unranked
E. None of above

[YOU (AI assistant)]: 