Jonathan Morton's Resume

WORK AND PROJECTS

Applied Research Scientist @ Meta (2016-present)

Controller Tracking Developed a model to predict position of quest headset controllers when they are out of FoV of any of the cameras by combining visual (e.g. visible arms) and non-visual (e.g. controller IMU) cues. The model combines adaptive multi-view fusion, temporal recurrence, sensor fusion, and uncertainty estimation to make reliable predictions in difficult scenarios. Shipped on quest 3.
Video Matting Lead team of 5 to achieve the "holy grail" of background replacement — real-time video matting — and replace Meta's existing frame-level segmentation models. Our models outperformed SOTA (RVM) on like metrics and significantly improved internal capability on an equal compute basis. Improvements include a stronger backbone, improved foreground loss, tweaked output activation, higher quality data and knowledge distillation
Smart Camera Lead research, development, and deployment of real-time person detection model for Smart Camera, the hero feature on Facebook's smart home device Portal. This industry-first feature later copied by Apple and others hinged upon making huge improvements to the detection accuracy and compute cost of the research models. We achived this through extensive model architecture experimentation, a large data collection and cleaning effort, improved losses, and many small tweaks. One of the first to our knowledge to get batch normalization working in an R-CNN model.
Style Transfer Implemented internal prototype of new style transfer technique, jump-started a project around it internally, and lead the development of neural style transfer models for filters in the Facebook Camera. One of the first to show an image-to-image model running in real-time on a phone through aggressive model and inference optimizations. Our models were used by millions people in the facebook app.

Open Source

Detectron2 (github.com/facebookresearch/detectron2) - wrote & maintained keypoints pipeline, added giou loss support, several QoL improvements
tart (github.com/jonmorton/tart) - auto-regressive transformer model training and experimentation project. utilized to conduct research on tokenizer-free byte-level LLM design
NNT (github.com/jonmorton/nnt) - personal project containing common pytorch nn training tools

Publications

Feature-align network with knowledge distillation for efficient denoising (WACV 2022)
Lucas D Young, Fitsum A Reda, Rakesh Ranjan, Jon Morton, Jun Hu, Yazhu Ling, Xiaoyu Xiang, David Liu, Vikas Chandra
HIME: Efficient Headshot Image Super-Resolution with Multiple Exemplars (WACV 2023)
Xiaoyu Xiang, Jon Morton, Fitsum A Reda, Lucas D Young, Federico Perazzi, Rakesh Ranjan, Amit Kumar, Andrea Colaco, Jan P Allebach

EXPERIENCE

Machine Learning

Building training pipelines in pytorch since before pytorch's 1.0 release (and with luatorch before that).
Taken multiple projects through entire ML lifecycle of research, optimization for product, deployment, and feedback-driven iteration
Special interest in model architecture optimization, training efficiency, and ML frameworks/ML productivity

Software Engineering

Previously have: built and published an ad supported web game, written software for robotics competitions, built and run a web hosting service, freelanced, and implemented WebGL in Web Workers for Firefox during a Mozila internship
Extensive traditional software engineering experience across several different stacks, including: Web backends in django, web frontends in React, scrapers and NLP pre-processing in rust, interactive experiences in unity, relational DBMS, and linux server administration.
Proficient with Python, C, C++, Rust, Java, Lua, Javascript, PHP.