@allen_ai is the open source final boss. I love this lab. While some labs keep everything closed or only give you a glimpse of their work (some weights, some report), these guys give you EVERY SINGLE THING, even their heart ❤️ Here is what you need to know about the release: - They are releasing OlmoEarth v1.3, SOTA satellite embedding model maintaining v1.2 quality despite cutting the GPU training budget by half. - This v1.3, following earlier versions, is an encoder decoder vision transformer (ViT-Base) model trained via latent masked image modeling. - Their model is swallowing annual radar data (Sentinel 1) along with multi spectral data (Sentinel 2 and LandSat) - They start from an input of 10m pixels across multiple timesteps and modalities. Thanks to a patch embedding module, they produce one token per patch with a patch size of 2 (essentially one patch = 20m). One token = one place x one date x one sensor - These patches then go through a vision transformer. Then they have learned queries, aka Attention Pooler, (one per 10m output pixel), and each query attends to every output tokens from the vision transformer (which serve as the keys and values). They call the resulting tokens "location tokens". These are having 768 dimensions. - After a self attention between queries, they produce embeddings for each 10m output pixel. - The learned queries are decoupled from the ViT outputs, therefore we can choose the output resolution desired regardless of the ViT's patch size. That's super interesting to finely adapt the compute budget and a way cheaper method than processing the input at the desired output size through the ViT. - They use the "location tokens" to predict maps, either in a regression fashion (like for elevation where you want to predict a number) or a classification way ( like Worldcover labels where you want to predict a label). In order to do so they are using a linear layer mapping each location token to a P x P x n tensor. P here is the output pixel and n the number of values predicted. For example with a patch size of 2, and for classifying the 11 Worldcover labels, you would have PxPxn = 2x2x11. For regression they use the L1 loss while for classification they are using cross entropy. Since each loss operates in different magnitude they are weighting each loss (weight of 1 for continuous maps and 0.1 for categorical maps). - To encourage the location tokens to retain fine grained details they use a latent masked image modeling training objective. - The resulting embeddings are a compacted version of the 768 internal dimensions down to either 128 or 64 dimensions. This is achieved via distillation (one linear layer followed by a LayerNorm). This linear layer is trained with a cosine similarity loss and a Gram loss (these are the distillation losses). They are using a Matryoshka loss which means that the distillation losses are applied twice: once for the first 64 dimensions and a second time for the full 128 dimensions. Hence the first 64 dimensions can be used as an even smaller embedding. - At training time they are summing all their losses: latent masked image modeling + cosine + Gram + supervised (the maps). They used AdamW, batch size of 512 for 667,200 steps. They used a linear warmup of 8,000 steps before cosine annealing. - They are sharing precomputed annual embeddings covering the entire planet at a resolution of 10m per pixel, offering both a 64 and a 128 dimensions embedding. All you have to do for your problem is to train a small classifier on top of these. - When compared against Alpha Earth Foundations and TESSERA v2, their 128-d embeddings outperform both on 10/11 tasks while the 64-d version beat them on 7/11 tasks. - At 10 m resolution, 128 int8 dimensions per pixel, global coverage for a year, the dataset come to roughly 150 TB of storage. - One year of global embeddings cost 14,000 H100 hours. This is nothing short of beautiful. Everything is open, they are offering the building blocks to tackle very important problems we are facing nowadays and there is so much to learn from them both technically and ethically. CONGRATS on this amazing release

we made embeddings of the whole world you can do fun stuff with them and I think they are pretty. you're only allowed to do good stuff with these no cops no military lol

This Meta Glasses commercial at the US Open was absolutely brilliant. At the end of the day, your phone might just stay in your pocket. $META is coming for your lunch, $AAPL. Watch out.

Very happy to extend the invite to all my friends here. If free, please do come to catch with me and other @allen_ai-ites to talk about open models, agent, frameworks and beyond for scientific discovery! luma.com/b1luqv6p

Hey CHS lovers. X/Twitter/Elon fucked with things and are now charging more for services to create posts with links. That's why you see dead links on some of my headline posts now. Sorting out easiest way to keep posting here but for now you need to cut and paste or just visit capitolhillseattle.com/