Self-Supervised Learning from Images with JEPA

Summary

[Submitted on 19 Jan 2023 (v1), last revised 13 Apr 2023 (this version, v3)] Title:Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture View a PDF of the paper titled Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, by Mahmoud Assran and 7 other authors View PDF Abstract:This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. We introduce the Image-based Joint-Embedding Predictive Architecture (I-JEPA), a non-generative approach for self-supervised learning from images. The idea behind I-JEPA is simple: from a single context block, predict the representations of various target blocks in the same image. A core design choice to guide I-JEPA towards producing semantic representations is the masking strategy; specifically, it is crucial to (a) sample target blocks with sufficiently large scale (semantic), and to (b) use a sufficiently informative (spatially distributed) context block. Empirically, when combined with Vision Transformers, we find I-JEPA to be highly scalable. For instance, we train a ViT-Huge/14 on ImageNet using 16 A100 GPUs in under 72 hours to achieve strong downstream performance across a wide range of tasks, from linear classification to object counting and depth prediction. Submission history From: Mahmoud Assran [view email] [v1] Thu, 19 Jan 2023 18:59:01 UTC (3,080 KB) [v2] Thu, 30 Mar 2023 18:28:46 UTC (3,077 KB) [v3] Thu, 13 Apr 2023 17:59:37 UTC (6,252 KB)