From-scratch PyTorch implementation of Detection Transformer (DETR), including sinusoidal positional encoding, multi-head attention, bipartite matching loss, and a training pipeline tailored for efficient convergence within 100 epochs.

From-scratch PyTorch implementation of YOLOv3 with a ResNeXt backbone, featuring multi-scale anchor-based detection and distributed mixed-precision training with Hugging Face Accelerate.

From-scratch PyTorch implementation of ResNeXt for image classification using grouped convolutions and residual blocks.

Won 3rd place in the Multi-class Bi-Atrial Segmentation Challenge at MICCAI 2024 (STACOM workshop) by developing a two-stage cascaded 3D CNN for segmenting atrial structures in late gadolinium enhancement (LGE) MRI volumes, achieving Dice scores up to 0.931, minimizing Hausdorff distance, and earning inclusion in the official benchmarking study paper; the task involved separate segmentation of atrial cavities and thin-walled atrial tissue—a particularly difficult subtask due to anatomical ambiguity and voxel-level sensitivity.

Conducted a comprehensive ablation study exploring over a dozen model, training, and inference configurations to optimize 3D atrial segmentation from LGE-MRI, including systematic variations in architecture (nnU-Net ResEnc vs. MedNeXt), resolution, patch sampling, receptive field size, loss functions, and data augmentation; validated the two-stage cascade design, refined post-processing, and implemented a 50% faster inference strategy to meet competition runtime constraints.

Developed a 3D CNN with deep supervision to classify glioblastoma MGMT promoter methylation status, enabling non-invasive prediction of a key chemotherapy response biomarker, by training on 4-channel multimodal MRI data (T1, T1Gd, T2, FLAIR) and optimizing with patch-based sampling, tumor masking, and heavy data augmentation to mitigate overfitting on a small, heterogeneous dataset (~500 patients).

Improved model interpretability and training efficiency on high-dimensional 3D medical data, by building custom tools to visualize MRI slices, tumor segmentations, and prediction metadata, and by implementing efficient 3D data loading and augmentation pipelines using MONAI, TorchIO, and PyTorch Lightning under memory and resolution constraints.

Led development of deep learning object detectors for Sentry Tower and RoadRunner, delivering production-grade models for ground surveillance and aerial threat interception in counter-UAS applications, with continual upgrades driven by systematic architectural and data-driven experimentation.

Improved model robustness and reduced false positives by identifying failure modes in field environments, coordinating test site validation events, and applying targeted data augmentations and pipeline changes.

Developed a deep learning system that learns compact visual embeddings for real-world fashion images using weakly labeled data from social media. The model is trained with a triplet loss to capture style similarity based on shared attributes like garments and colors, without requiring explicit style labels. The resulting embedding generalizes well to downstream tasks such as fashion style classification.

Investigated alternative sampling strategies for experience replay in Deep Q-Learning, implementing stratified, recency, reward-based, and low-discrepancy methods in an Atari 2600 game-playing agent. Experiments showed that stratified sampling consistently improved training stability and policy performance, achieving higher average rewards and Q-values than uniform sampling in games like Breakout and Kangaroo, likely due to its enhanced temporal diversity in sampled experiences.

Designed and built scalable infrastructure to preprocess and annotate over 200 hours of aerial video via Amazon Mechanical Turk, enabling the creation of a large-scale dataset for studying human navigation in crowded outdoor environments. This supported research on trajectory forecasting, including a new model that predicts motion by accounting for interactions between different types of agents (e.g., pedestrians, cyclists) and their varying sensitivity to social cues.
[Dataset Webpage]

Implemented the GrabCut algorithm for foreground-background segmentation and conducted three experiments to improve segmentation quality: varying the number of Gaussian Mixture Model (GMM) components, reinitializing GMM parameters mid-optimization, and restricting background GMMs to pixels within the bounding box. Results showed that increasing GMM components improved segmentation on certain images, while reinitialization stabilized convergence. The best performance—96.94% accuracy and 87.71% Jaccard similarity—was achieved by constraining the background model spatially, enabling better capture of local color distributions near the object. This project was completed as part of CS231B: The Cutting Edge of Computer Vision at Stanford University.

Implemented the R-CNN object detection pipeline, combining selective search for region proposals, CNN-based feature extraction, SVM classification, and bounding box refinement via L2-regularized linear regression. This project was completed as part of CS231B: The Cutting Edge of Computer Vision at Stanford University.

Explored video object detection by extending YOLO-style single-stage detectors to process multiple consecutive frames, integrate auxiliary inputs like optical flow, and incorporate a custom transformer module with attention across spatiotemporal feature maps. Conducted extensive ablation studies showing improved recall in detecting small, fast-moving aerial targets such as drones in cluttered environments.

Demonstrated feasibility of real-time optical flow-assisted detection by adapting FastFlowNet for self-supervised training (using pixel-wise reconstruction loss instead of ground truth flow) and deploying it on NVIDIA Jetson AGX Xavier hardware via a custom CUDA plugin integrated into the C++ tracking engine; tested live on cUAS towers at test ranges.

Experimented with deep learning-based multi-object tracking models, including SiamMOT and CenterTrack, to track aerial targets such as drones for counter-UAS applications, training on internal datasets and benchmarking against single-frame object detection baselines like Faster R-CNN.

Improved real-time multi-object tracking accuracy for ground surveillance and counter-drone (UAS) operations on platforms such as Sentry Tower and Dust by replacing fragmented, bespoke integration code with a modular, reusable, testable C++ computer vision library that combined a TensorRT-compiled deep learning detector with asynchronous Kalman filtering and sparse optical flow tracking, validated in live military field exercises and deployed in production.

Developed core face tracking technology that contributed to Meta Quest Pro’s Face Tracking API (released after my tenure at Meta) enabling naturalistic avatar expressions in social VR, by implementing keypoint tracking algorithms that map facial movements to FACS-based blendshape activations in real time.
[Face Tracking SDK]

Proposed a two-stage algorithm for activity recognition in temporally untrimmed videos by first localizing potential activity segments and then classifying them with a two-stream convolutional neural network. Video localization was performed using a shot detection heuristic to segment the video into shots, followed by a tubelet-based heuristic to extract a clip likely containing an activity. Activity classification was then performed using a two-stream CNN, with one stream processing RGB frames and the other processing optical flow computed across 30-frame intervals. Experiments on the UCF-101 dataset showed that the RGB frame-based model outperformed the optical flow stream, likely because static visual cues in individual frames were often sufficient for activity recognition, whereas the optical flow model received only a single frame of motion information, limiting its temporal context. Additionally, uncorrected camera motion may have degraded the quality of the optical flow. This project was completed as part of CS231N: Convolutional Neural Networks for Visual Recognition at Stanford University.

Implemented the TLD (Tracking-Learning-Detection) algorithm for robust online object tracking and evaluated several extensions to improve accuracy and adaptability. The TLD framework combines optical flow tracking, a patch-based detector, and an online learning component that updates the object model to handle appearance changes across frames. The detector uses an ensemble of fern classifiers, each computing binary hashes over image patches and using nearest neighbor matching to determine whether a patch corresponds to the tracked object. Experiments included varying the number and size of ferns, replacing the fern ensemble with a linear SVM, and using Histogram of Oriented Gradients (HOG) features for patch representation. Results showed that HOG features and SVM-based detection improved robustness under occlusion and motion. This project was completed as part of CS231B: The Cutting Edge of Computer Vision at Stanford University.

Developed an activity recognition algorithm for videos using hand-crafted feature representations and a one-vs-rest linear SVM for final classification on the UCF-101 dataset. Improved Dense Trajectory Features (IDTF) were extracted by sampling feature points, tracking them across frames using optical flow, and computing descriptors—Trajectory, HOG (Histogram of Oriented Gradients), HOF (Histogram of Optical Flow), and MBH (Motion Boundary Histogram)—along the resulting trajectories to capture motion and appearance cues. The hundreds of IDTF descriptors per video were aggregated into a compact representation using Fisher Vectors, computed with respect to a Gaussian Mixture Model (GMM) fitted to the descriptor distribution across the dataset. Dimensionality of the Fisher Vectors was reduced using Principal Component Analysis (PCA) before SVM classification. This approach achieved up to 79.34% mean average precision, significantly outperforming static-frame baselines. This project was completed as part of CS221: Artificial Intelligence at Stanford University.

Implemented and refactored deep learning object detection models for efficient C++ inference using TensorRT, including adapting models for ONNX export, developing custom TensorRT plugins and CUDA kernels, and benchmarking performance on embedded devices like the NVIDIA Jetson AGX Xavier.

An ongoing from-scratch exploration of text-to-image generation with latent diffusion, focused on understanding core components like VAEs, U-Nets, Diffusion Tranformers, and diffusion schedulers. So far, I’ve trained a baseline Stable Diffusion model using a custom trainer, with custom model and scheduler implementations planned next.

Co-invented a neural-network-based facial video compression technique, reducing bandwidth needs while maintaining visual fidelity, by reconstructing facial video frames from transmitted keypoints and partial image data using a generative model.

Trained a CycleGAN to generate photorealistic facial video by transferring lighting and texture from real headset footage onto rendered 2D facial rig views, creating realistic synthetic training data for neural networks that predict 3D facial blendshape coefficients from Oculus VR face tracking cameras.

Implemented Flash Attention 2 in Triton and performed extensive benchmarking, achieving a 7× speedup in end-to-end runtime (forward + backward) of the attention operation compared to the naive PyTorch implementation on long sequences using bfloat16 on an RTX 5090 GPU. The speedup comes from GPU-efficient techniques such as operator fusion, tiling to shared memory to minimize global memory latency, and low-precision computation, which together reduce memory overhead and improve parallel execution efficiency.

From-scratch PyTorch implementation of BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding using Byte Pair Encoding (BPE) tokenization, whole-word masking, gradient accumulation, and 1M-step masked language model training on BookCorpus and Wikipedia.

Contributed to Oculus LipSync’s audio-driven facial animation system, widely adopted by VR developers to power real-time, expressive avatar movement in Unity and Unreal Engine, by implementing Temporal Convolutional Networks (TCNs) to predict viseme shapes from LogMel audio features, and co-inventing patented methods for full-face animation—including lips, eyebrows, and eyelids—based on speech and pitch-driven input.

Designed and implemented the real-time laughter detection neural network shipped with Oculus LipSync, achieving robust performance across diverse vocal profiles, by training a Temporal Convolutional Network (TCN) on a manually labeled and augmented dataset, and optimizing the model for real-time inference on CPU and DSP.

For technical details and discussion, see the blog posts for Oculus LipSync release v1.28.0 and release v1.30.0, and the [Oculus Lipsync Documentation].

Detailed overview of Oculus LipSync is provided at the PyTorch Developer Conference 2018 [video - my talk begins at 31:45].

Contributed to a team project to develop an AI-powered meeting assistant integrated with the Amazon Echo, designed to provide real-time voice-driven support during meetings. The system transcribed audio to text, extracted action items, and surfaced insights such as sentiment and energy scores, which were organized into a web dashboard for post-meeting analysis. Built in collaboration with VMWare as part of Stanford's CS210 Software Project Experience course.

Developed a dependency-tree-based LSTM model for answering Quiz Bowl questions that outperforms previous recursive neural network approaches by over 3% in accuracy, leveraging word2vec embeddings and tree-structured memory cells to better capture syntactic and semantic information from the questions. The Tree-LSTM extends standard linear-chain LSTMs by propagating information along the syntactic dependency parse tree of each sentence, allowing the model to selectively retain or forget information based on grammatical structure.