<!-- filename: results_and_discussion.md -->
## Results and Discussion

This study investigated the geometric structure of a 10-dimensional latent space generated by a Physics-Informed Neural Network (PINN) for the 2D Burger's equation. The PINN was trained with a multi-head architecture to handle 25 different initial conditions (ICs) for a fixed viscosity. The analysis aimed to determine if the latent representations ${L(x,t,IC)}$ for each IC form low-dimensional manifolds and how these manifolds are related, potentially revealing a simple, overarching structure.

### 1. Global Structure of the Latent Space

An initial Principal Component Analysis (PCA) was performed on the entire set of latent vectors, combining all spatial points (x), time steps (t), and initial conditions (ICs). This global PCA provides an overview of the total variance distribution in the 10-dimensional latent space.

The results, visualized in the scree plot `global_pca_scree_plot_1_20250604-135555.png`, indicate a pronounced low-dimensional structure. The quantitative breakdown of explained variance is as follows:
-   **PC1:** 60.12%
-   **PC2:** 23.44%
-   **PC3:** 12.93%
-   **PC4:** 1.30%
-   **PC5:** 1.17%
-   **PC6:** 0.76%

Cumulatively:
-   The first three principal components (PCs) capture **96.48%** of the total variance.
-   The first six principal components (PCs) capture **99.72%** of the total variance.

This strong concentration of variance in the leading few components suggests that, when viewed collectively, the latent representations generated by the PINN across all conditions, space, and time points predominantly occupy a low-dimensional subspace within the original 10-dimensional latent space. The effective global dimensionality required to capture nearly all (e.g., >99%) system variability is approximately 6.

### 2. Intrinsic Dimensionality of Per-Initial Condition Manifolds

To understand how the latent space is structured for individual initial conditions, PCA was performed separately on the set of latent vectors ${L(x,t,IC_k)}$ for each of the 25 ICs (denoted $IC_k$ where $k \in [0, 24]$). Each $IC_k$ generates $101 \times 103 = 10403$ latent vectors.

The analysis revealed a striking consistency across all ICs.
-   **Intrinsic Dimensionality:** For every one of the 25 ICs, precisely **3 principal components** were sufficient to explain over 95% of the variance within that IC's set of latent vectors. The distribution plot `intrinsic_dim_dist_plot_2_20250604-135804.png` confirms this uniformity, showing a single bar at 3 dimensions.
-   **Average Variance Explained (Per-IC):** The average cumulative variance explained by these first three per-IC principal components is detailed in the summary table from Step 2 and visualized in `per_ic_avg_scree_plot_2_20250604-135804.png`.
    -   Average PC1_ic: 59.61%
    -   Average PC2_ic: 23.72% (Cumulative: 83.33%)
    -   Average PC3_ic: 14.15% (Cumulative: 97.48%)
-   The standard deviation of the cumulative variance explained by the first three per-IC PCs was very low (e.g., 0.15% for the cumulative variance at PC3_ic), indicating that the 3-dimensional nature of these individual manifolds is a highly consistent feature across all tested initial conditions.

These findings suggest that for any given initial condition, the solution manifold $L(x,t,IC_k)$ as represented in the latent space is effectively a 3-dimensional structure. The high percentage of variance captured by these few components implies that these 3D structures can be well-approximated by affine subspaces (i.e., they are relatively "flat").

### 3. Geometric Arrangement of Manifold Centroids

The centroid $C_k$ for each per-IC manifold represents the average position of its latent vectors. The geometric arrangement of these 25 centroids ${C_k}$ in the 10-dimensional latent space was investigated by performing PCA on the $(25 \times 10)$ matrix of centroids.

The results from this centroid PCA were remarkable:
-   The first principal component of the centroids (CPC1) explained **99.86%** of the variance in the positions of these centroids.
-   CPC2 explained only 0.10%, and CPC3 explained 0.02%.
The scree plot `centroid_pca_scree_plot_3_20250604-135958.png` graphically illustrates this extreme dominance of CPC1.

Visual inspection of the centroids projected onto their principal components further clarifies their structure:
-   The 2D scatter plot `centroid_pca_scatter_2D_3_20250604-135958.png` (CPC1 vs. CPC2) shows the 25 centroids forming an almost perfectly linear arrangement. The points, labeled by their IC index (0 to 24), are ordered sequentially along this dominant direction (CPC1).
-   The 3D scatter plot `centroid_pca_scatter_3D_3_20250604-135958.png` (CPC1 vs. CPC2 vs. CPC3) confirms this linearity, with minimal deviation into the CPC2 and CPC3 directions.

This implies that the primary effect of changing the initial condition (from IC 0 to IC 24) is to translate the corresponding 3D latent manifold along a single, well-defined direction or curve (which is nearly linear) within the 10-dimensional latent space.

### 4. Comparative Analysis of Manifold Orientations and Relationships

Having established that each IC corresponds to a 3D affine manifold and that these manifolds are translated relative to each other, the next step was to compare their orientations. Let $V_k = [v_{k1}, v_{k2}, v_{k3}]$ be the matrix of the top 3 principal vectors for $IC_k$.

**4.1. Alignment of Individual Principal Vectors:**
The alignment between corresponding principal vectors from different ICs was quantified using absolute dot products.
-   **PC1_ic vs. PC1_ic:** The mean absolute dot product between the first principal vectors of any two different ICs was 0.747 (standard deviation 0.259).
-   **PC2_ic vs. PC2_ic:** The mean absolute dot product for the second principal vectors was 0.772 (standard deviation 0.233).
-   **PC3_ic vs. PC3_ic:** The mean absolute dot product for the third principal vectors was 0.865 (standard deviation 0.139).
The heatmaps in `dot_product_heatmaps_4_20250604-140218.png` show these dot products. While not perfectly aligned (dot product 1), these values indicate substantial alignment, particularly for PC3_ic. The heatmaps often show blocks of higher alignment, especially for ICs with nearby indices, suggesting a gradual change in orientation.

**4.2. Similarity of 3D Principal Subspaces:**
To assess the overall alignment of the 3-dimensional subspaces spanned by $V_k$, the average squared cosine of the principal angles between pairs of subspaces $(span(V_k), span(V_l))$ was computed.
-   The mean similarity score across all pairs of ICs was **0.986** (standard deviation 0.014). The minimum similarity observed was 0.954.
The heatmap `subspace_similarity_heatmap_4_20250604-140218.png` visually confirms this: the matrix is predominantly bright, indicating very high similarity values close to 1. This result is crucial: it demonstrates that the 3D latent manifolds for different ICs are remarkably parallel to each other.

**4.3. PCA of Principal Vector Sets:**
To understand how the orientation vectors $v_{k1}, v_{k2}, v_{k3}$ themselves vary across the 25 ICs, PCA was performed on each set of these vectors: ${v_{k1}}_{k=0}^{24}$, ${v_{k2}}_{k=0}^{24}$, and ${v_{k3}}_{k=0}^{24}$.
-   **For the set of first principal vectors ${v_{k1}}$:** The first component of this PCA explained **85.45%** of their variance.
-   **For the set of second principal vectors ${v_{k2}}$:** The first component explained **80.04%** of their variance.
-   **For the set of third principal vectors ${v_{k3}}$:** The first component explained **97.67%** of their variance.
The scree plots and scatter plots in `pca_of_principal_vectors_4_20250604-140218.png` illustrate this. The scatter plots (bottom row) show that the tips of these vectors, when projected, also trace out near-linear paths as the IC index changes. This indicates that the subtle variations in the orientations of the 3D manifolds are not random but are themselves highly structured and change in a very low-dimensional (effectively 1D) manner as a function of the initial condition.

In summary, the 3D latent manifolds associated with different ICs are not only translated versions of each other but also maintain a very high degree of alignment. The minor changes in their orientation vectors are systematic and follow a simple, low-dimensional pattern correlated with the IC index.

### 5. Relationship Between Per-IC Manifolds and Global Latent Space Structure

The final analytical step was to relate the individual per-IC manifold structures to the global latent space structure identified in Section 1. The global PCA determined that $d_{glob}=6$ principal components capture >99% (specifically 99.72%) of the total variance across all data points.

**5.1. Projection of Per-IC Variance onto Global Subspace:**
Each per-IC latent point cloud (centered on its own centroid $C_k$) was projected onto the $d_{glob}=6$ dimensional global principal subspace. The percentage of each $IC_k$'s intrinsic variance captured by this global subspace was calculated.
-   On average, **99.66%** of each IC's intrinsic variance was captured by the 6D global subspace.
-   The minimum capture was 99.24% (for IC 24) and the maximum was 99.78% (for IC 8), with a standard deviation of only 0.15%.
The bar chart `variance_captured_by_global_subspace_5_20250604-140503.png` shows this remarkable consistency. This means that the individual 3D manifolds are almost perfectly embedded within this common, global 6D subspace.

**5.2. Per-IC Centroids in the Global Subspace:**
The per-IC centroids $C_k$ were projected onto the global principal components.
-   The scatter plots `projected_centroids_2D_5_20250604-140503.png` (Global PC1 vs. Global PC2) and `projected_centroids_3D_5_20250604-140503.png` (Global PC1 vs. Global PC2 vs. Global PC3) show these projected centroids.
-   Consistent with the centroid PCA (Section 3), the centroids form a clear, near-linear progression, primarily along the first global principal component (Global PC1). The IC index (0-24) aligns almost perfectly with the position along this global PC1 axis.

This confirms that the dominant mode of variation across ICs (the translation of centroids) aligns with the most dominant direction of variation in the entire dataset. The individual 3D manifolds are "stacked" or "arrayed" along this primary global axis.

### 6. Synthesis and Overall Interpretation

The collective findings paint a clear picture of a highly organized and surprisingly simple latent space structure learned by the PINN:

1.  **Low-Dimensionality is Pervasive:** The latent representations are fundamentally low-dimensional. Globally, 6 dimensions capture almost all variance. Crucially, for any specific initial condition, the dynamics evolve on an effectively 3-dimensional affine manifold.

2.  **Dominant Role of Translations:** The primary way the PINN encodes the effect of different initial conditions is by translating these 3D manifolds. These translations occur along a well-defined, essentially one-dimensional path in the 10D latent space. This path is strongly aligned with the first principal component of the global latent space.

3.  **Highly Aligned Manifolds:** The 3D manifolds corresponding to different ICs are not arbitrarily oriented. They are remarkably parallel to each other, with an average subspace similarity (based on principal angles) exceeding 0.98.

4.  **Structured Variation in Orientation:** The minor variations in the orientations of these 3D manifolds are not random. The principal orientation vectors ($v_{k1}, v_{k2}, v_{k3}$) themselves vary systematically with the IC index, each following an effectively one-dimensional trajectory in the space of orientation vectors.

5.  **Hierarchical Structure:** There's a hierarchical structure: a global ~6D space contains all the dynamics. Within this, ICs select a specific 3D affine manifold by shifting its position along a primary ~1D path. The "shape" (orientation) of this 3D manifold also undergoes subtle, highly correlated 1D changes.

This structure suggests that the PINN has learned a very efficient parameterization. Instead of each IC creating a vastly different or complex structure in the latent space, the network seems to have identified a canonical 3-dimensional "template" for the solution manifold. Different initial conditions then primarily select a position for this template along a learned trajectory and apply minor, systematic adjustments to its orientation. This is a powerful form of disentanglement, where the effect of the initial condition is largely mapped to a simple geometric transformation (translation and slight, structured rotation) in the latent space.

The progression of IC indices (0 to 24) consistently maps to a progression along the centroid path and the paths of orientation vectors. This implies that the ordering or nature of the initial conditions themselves (which were not detailed in the problem description but are indexed 0-24) has a continuous and smooth impact on their latent representation.

### 7. Discussion of Limitations and Future Directions

While these findings are compelling, some limitations and potential future work should be noted:
*   **Linearity Assumption of PCA:** The analysis heavily relies on PCA, which is optimal for identifying linear subspaces. While the high variance capture suggests affine manifolds are good approximations, non-linear manifold learning techniques (e.g., UMAP, t-SNE, autoencoders) could potentially reveal finer non-linear structures within the 3D manifolds or in the arrangement of centroids and orientations.
*   **Fixed Viscosity:** This study was conducted for a fixed viscosity. Investigating how the latent space structure changes with varying viscosity would be a critical extension, potentially revealing how the PINN encodes this physical parameter. One might expect the dimensionality or the nature of the manifold "template" to change.
*   **Number of Initial Conditions:** While 25 ICs provided clear trends, a larger and more diverse set of ICs could further refine the understanding of the centroid trajectory and orientation variations, confirming their low-dimensional nature or revealing more complex patterns.
*   **Nature of Initial Conditions:** The specific nature of the 25 initial conditions (e.g., different wave profiles, amplitudes, or phases) was not provided. Correlating the specific characteristics of the ICs with their position along the centroid trajectory and their manifold orientations would provide deeper physical insights into what features the PINN is encoding.
*   **Generalizability:** The observed simple structure is for a specific PINN architecture and a specific PDE (2D Burger's equation). Further research is needed to determine if similar structured latent spaces are learned for other PDEs or different network architectures.

In conclusion, this detailed study reveals that the PINN has learned a remarkably structured and geometrically simple latent representation for the solutions of the 2D Burger's equation under varying initial conditions. The latent space is not an opaque, high-dimensional entanglement but rather an organized system of low-dimensional, aligned manifolds whose positions and subtle orientation changes are systematically parameterized by the initial conditions. This provides valuable insights into the internal workings of the PINN and suggests potential avenues for model interpretation, compression, and control.