
HARPO (AKA Carlos A. Catania). A Nerdo with decent social skills? #DataScience #Algorithms. English, Spanish and Français. From Mendoza Republic. South America
Argentina
Joined July 2014
- Tweets654
- Following62
- Followers123
- Likes1.1K
Since Mixture of Expert (MoE) LLMs are all the rage as of this weekend, thanks to the Mixtral-8x-7B release, here's a quick explainer. The figure below shows the architecture behind the Switch Transformer (arxiv.org/abs/2101.03961), a great intro to MoEs.
The model depicted in this figure uses 1 expert per token with 4 experts in total. Mixtral-8x-7B, on the other hand, consists of 8 experts and uses 2 experts per token.
Why MoEs? Combined, the 8 experts in a 7B model like Mixtral are still ~56B parameters. (Actually, it's less than 56B, because the MoE approach is only applied to the MoE layers, not the self-attention weight matrices. So, it's likely closer to 40-50B parameters.)
However, since the router reroutes the tokens such that only 7B parameters (instead of all 56B) are used at a time for the forward pass, the training (and especially inference) will be much faster compared to the traditional non-MoE approach.
If you read my AI and Open Source in 2023 article (magazine.sebastianraschka.co…) approx. 2 months ago, I mentioned that "It will be interesting to see if MoE approaches can lift open-source models to new heights in 2024". It looks like Mixtral started this trend early, and I am sure that this is just the beginning :).
Today @verovaleros gave the talk “Four Key Problems in OSINT for Cyber Threat Intelligence” in @enisa_eu CTI conf. #cti
"Researchers Pre-trained LLM Agents Acting as Human Penetration Testers" Our work w/@mrigaki @ondrej_lukas @harpolabs and me, as part of the Aidojo w/@muni_cz was covered in @The_Cyber_News cybersecuritynews.com/intend… @StratosphereIPS #RL #agents #automaticpentest
Our work with @MurisSladic @verovaleros @harpolabs in LLM honeypots was highlighted in @The_Cyber_News "shelLM – A New AI-Based Honeypot to Engage Attackers as a Real System" @StratosphereIPS cybersecuritynews.com/shellm… via @The_Cyber_News
🚨New Paper Alert! In our latest research we used LLMs as reinforcement learning red team agents: "Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments" arxiv.org/abs/2308.12086 with @ondrej_lukas @harpolabs @eldraco #AI #LLM #ML #infosec
@[email protected] retweeted
Replying to @predict_addict
I'm talking on this at #PositConf2023 in Chicago (posit.co/conference/) in about 2 weeks!
It’s time to admit that we were all overly optimistic during COVID, thinking that organizations will finally understand the “remote work” culture. Not only is back to office, but also back to paper and physical meets. We thought we could make a change and we lost. #fail
We are happy to announce that our paper with @eldracote was accepted in ESORICS 2023: "The Power of MEME: Adversarial Malware Creation with Model-Based Reinforcement Learning" arxiv.org/abs/2308.16562 @StratosphereIPS @ctu_cs #ai #adversarial #MachineLearning #malware #security
Thanks for the kits and the workshop! Help us get into the community and have fun!
Thanks all who came along to my workshop. Totally sold out and more wanted to join. I hope they enjoy their new #lora satellite stations and the @tiny__GS community. Bonus thanks to my fellow @TOG_Dublin crew for the helping hand too #cccamp23
@[email protected] retweeted
Desde el nuevo Media Lab de la @UNCUYO acabamos de lanzar un podcast dedicado a la inteligencia artificial. Se llama "Inteligencia natural" y acá están los primeros 3 episodios. open.spotify.com/show/4oeyHh…
Desde el nuevo Media Lab de la @UNCUYO acabamos de lanzar un podcast dedicado a la inteligencia artificial. Se llama "Inteligencia natural" y acá están los primeros 3 episodios. open.spotify.com/show/4oeyHh…
Join us in Las Vegas for our @BlackHatEvents
hands-on Advanced Malware Traffic Analysis training with @verovaleros! Learn how to detect botnets, ransomware, lateral movement, and more! Learn to detect attacks w/machine learning tools! #BlackHat #BHUSA blackhat.com/us-23/training/…
Automated Code Reviews by ChatGPT through GH Actions? Sure! Learn how to do it at jcrodriguez1989.medium.com/s…
#chatgpt #githubactions #codereview #pullrequest #llm
Our paper w/@MasarahClouston "On the dynamics behind profit-driven cybercrime from contextual factors to perceived group structures, and the workforce at the periphery" was finally published! I'm so proud and happy for this collaboration! #cybercrime tandfonline.com/doi/full/10.…
Let's gooo! 🔥
We're happy to introduce two new official Space Templates in collaboration with @posit_pbc: Build and share your R and Python Shiny apps directly on the Hub! 🥳
We’re so excited to announce that you can deploy both #RStats and #Python Shiny apps on @huggingface Spaces using our Docker templates.
Shiny is a great front-end for AI models, and we can’t wait to see what people build with this integration.
shiny.rstudio.com/blog/shiny…
Mornings are made for reading, nights are made for coding.🤓
... Except when you have kids 🤷🏻♀️
1
2
123
I believe we need open-source alternatives to ChatGPT for more transparency, inclusivity, accountability and distribution of power.
Excited to introduce HuggingChat, an open-source early prototype interface, powered by OpenAssistant, a model that was released a few weeks ago. – at Normandy Shores, FL
¡El #SAHTI2023 de las #52JAIIO @jaiio_oficial se prepara!
@bmassare desde CABA, Karina Bianculli desde MdP y yo desde Cba, armando un Simposio de Historias, Tecnología e Informática que va a estar muy bueno.
¡Hasta el 1 de mayo tenés tiempo de presentar el resúmen!
Te esperamos.
Another collaboration with visual artist Lili Fiallo. Using #dreambooth we fine tuned #stablediffusion to incorporate Lili's toy art style. incredibly simple using diffusers and gradio libraries.
instagram.com/fiallolili/
3e2bb3f1180757bdee.gradio.li…
2
213































