Implicit Neural Representation (INR) has revolutionized scene representation using neural networks, offering high-quality rendering and memory efficiency. Leveraging a neural network as a continuous function, INR learns to render images through reconstruction loss, including RGB images for 3D scenes using multi-view images. In this project, we tackle the problem of style transfer for implicit neural representations. Existing methods for INR style transfer rely on test-time operations using fixed network parameters, leading to limited diversity in stylized images. To overcome these limitations, we propose an enhanced style transfer approach for INR. Our framework introduces generalizable INRs for style transfer, incorporating a ViT-based hypernetwork to predict instance-specific features for an MLP. By leveraging Adaptive Instance Normalization (AdaIN), we fuse content and style features, providing the MLP with the necessary inputs. We evaluate the effectiveness of our generalizable style transfer INR framework on the ImageNette dataset, demonstrating its ability to produce high-quality stylized images.

Generalizable Style Transfer for Implicit Neural Represenation

Graph generation poses challenges in handling large graphs due to their complexity. This paper presents a framework that effectively bridges existing graph compression methods with powerful generative models. Our key contribution lies in leveraging compression techniques and integrating them with the denoising diffusion model to address these challenges. By employing graph compression, we enable efficient diffusion processes and mitigate permutation problems. The framework facilitates the diffusion process by compressing the graph in terms of node count, allowing for the generation of large graphs with reduced computational complexity. Importantly, our approach ensures lossless decompression to preserve information during reconstruction. Experimental evaluation demonstrates the superior performance of our method compared to the base model, specifically on atom-level graphs. Our framework holds promise in advancing graph generation techniques and enabling their application across diverse domains, striking a balance between performance and efficiency while leveraging the power of the diffusion model.

Efficient Graph Generation: Bridging Compression and Diffusion Models for Large-scale Graphs

In recent years, text-to-image generation has received significant attention as researchers aim to automatically generate realistic images from textual descriptions. Despite the promising results of diffusion models in producing high-quality images, they often struggle to capture the richness and diversity of natural language expressions, making it difficult to generate images that align with user intentions. To tackle this challenge, personalization has been proposed as a potential solution. Personalization involves fine-tuning pre-trained text-to-image generation models using user-provided images that contain specific concepts or subjects, enabling the models to generate images with the desired subject or style. However, existing personalization techniques typically require direct fine-tuning of the text encoder, diffusion model, or both, which can result in high computational costs for incorporating user-provided concepts and potentially compromise the model's knowledge. This paper introduces a novel approach to personalizing a text-to-image model by leveraging a BLIP-2 encoder. We provide the image that contains objects we wish to generate using the Stable Diffusion model as inputs to the BLIP-2 encoder. Then, we use the output queries of the BLIP-2 Q-former as visual prompts to guide the Stable Diffusion model to generate images that capture the visual representations of the input image.

Personalizing text-to-image generation with visual prompts using BLIP-2

Spatiotemporal data mining plays a crucial role in real-world scenarios such as air quality monitoring and intelligent traffic management. However, real-world spatiotemporal data collected in such scenarios is often incomplete due to sensor failures or transmission loss. Spatiotemporal imputation aims to fill in the missing values based on the observed values and their underlying spatiotemporal dependence. Previous dominant imputation models relied on autoregressive methods, which suffered from error accumulation. To overcome this limitation, emerging generative models like diffusion probabilistic models (DPM) can be employed for imputing missing values. These models are conditioned on observations to avoid relying solely on inaccurate historical imputation methods. However, applying diffusion models to spatiotemporal imputation presents challenges, particularly in extracting and utilizing conditional information from observed data. In this paper, we propose a novel framework for utilizing diffusion models for spatiotemporal imputation, by formulating imputation as inpainting problem. We first train an unconditional diffusion model for predicting the whole spatiotemporal data. To condition the generative process, we follow the scheme proposed by RePaint \citep{lugmayr2022repaint}, only alter reverse diffusion iterations by sampling the unobserved regions using the observed data information. To model spatial dependencies, we utilize a GNN-based backbone for DPM. We compare our model with state-of-the-art baselines in various missing patterns of two real-world spatiotemporal benchmark datasets. 

Imputation as Inpainting: Diffusion models for SpatioTemporal Data Imputation

Recent advancements in generative AI models, particularly the emergence of ChatGPT, have significantly increased interest in generative AI. However, the reliance on text prompts as the primary medium of interaction poses various challenges, especially for users with limited AI knowledge. Prompt engineering has emerged as a technique to address this issue, but it is still time-consuming and may not always yield satisfactory results. In this research, we aim to test the feasibility of an AI system that provides customized answers tailored to individual users, focusing on engineering model responses rather than solely relying on prompt engineering. We developed a system that enables users to evaluate their preferences and provide feedback on several model responses for the same prompt. And through this system, we collected user preference data and applied instruction tuning to guide model responses in a manner preferred by a certain user based on the certain user's preference data. Our study serves as a proof of concept to explore the potential of more personalized and user-centric AI systems in the future.

Towards Personalized AI: Engineering Model Responses for Customized User Interactions in Generative AI Systems

The success of deep learning-based molecular generation has been well-documented in various fields, but generating 3D molecular structures has been challenging due to their complex geometric symmetries. Recent developments in geometric deep learning have opened up new possibilities for 3D molecular representation and diffusion-based models for molecular generation. This thesis focuses on extending the Equivariant Diffusion for Molecule Generation in 3D (EDMs) model, which generates equivariant molecular structures that retain their properties irrespective of their position or orientation in 3D space. We propose using an equivariate graph neural network (GNN) for categorical data with bit diffusion and evaluate its performance on a larger three-dimensional small molecules dataset. Additionally, we explore the generalizability of our proposed models to other types of datasets such as dimers, dipeptides, solvated amino acids, and inorganic materials. Our results demonstrate the potential of deep learning-based molecular generation for various applications in materials science and drug discovery.

Enhancing the Applicability of the Equivariant Diffusion Model to Diverse Molecule Datasets

The success of Transformer-based systems in vision and language-related tasks led to the development of state-of-the-art approaches in time series. However, there are several fundamental problems with existing methods and their benchmarks. Although the real-world data is non-stationary, existing datasets are mostly presented with stationary data. This entails a focus shift of the models towards learning simple patterns. As a result, models struggle with predicting noisy data, such as stock data and Brownian Motion. In this work we present TimePatch Diffusion, which is a novel architecture, to deal with complex time-series forecasting tasks. It is composed of a DDPM encoder and a TPatch decoder. The former learns time-series noise, while the latter extract encoded features from captured representations.