
musician & audiovisual artist. synaesthetic music (modular/generative/AI). art+tech. #BAYC ๐จ๐บ
Joined March 2021
- Tweets7.9K
- Following4.5K
- Followers7.9K
- Likes32.4K
ComfyUI gave me early access and credits to test their new in-app agent, and I've been having fun working with it ahead of today's release! It embodies a lot of what I love about collaborating with AI in a creative process - making technical experimentation faster and more accessible, without sacrificing creative control.
Video made with Comfy Agent + Wan VACE - sound on! ๐
I've been working with agents + ComfyUI for a while. For me, what makes the new in-app agent special is its ability to work directly inside your workflow, alongside you.
When you send a prompt, the agent sees the current state of your graph, including any changes you made. Its edits appear directly on the canvas, as it makes them (no reloading or opening a new version of the workflow).
It's natural to switch between tweaking parameters and running test generations yourself, and asking the agent to build a new part of the graph or troubleshoot something. It's also good at answering questions to help you get a better sense of what's happening technically, or how various factors relate to your creative intent.
The immediacy of this allows you to quickly test and develop new approaches. The iterative loop is faster, making it easier to experiment freely, and get into a creative flow.
gorkulus retweeted
Today we launched Comfy Agent. It lives right inside ComfyUI.
Creators still need craft, and craft comes from control. I truly believe it. But they should be focusing on ideas, iteration, composition, colors, not technical terms or debugging. When I started using ComfyUI 3 years ago, I kept seeing talented people who couldn't use the tool at their highest level.
This should all go away. Today you describe what you want, and Comfy Agent takes the building part. It even works with you while you keep editing. The workflows are coming to life now. They can talk to you.
It's in Comfy Cloud now and free to try. Desktop/local is coming in a few weeks. It's still beta, so please tell us what breaks.
It is my honor to build with our amazing team, our tech leads @c__byrne, @shimikeri, as well as @UyTieu @xlatentspace, Alex DrJKL, @ComfyUIWiki @_ryan_di @shanef3d, @mattmiller_ai, Memie, Anne, Ali, Marwan and many other people. I hope you enjoy creating with us!
Comfy Agent is now available to everyone in Comfy Cloud.
It plans, builds, and fixes ComfyUI workflows right on your canvas.
We strongly believe creators shouldnโt choose between ambition and complexity. Comfy Agent is our answer. While it handles the technical part of Comfy, you can finally focus on visual craft!
- It builds while you edit. It sees the assets you see.
- Ask anything. Reference anything.
- Create your own skills, or use the public ones.
Agent in Comfy Desktop is coming soon in a few weeks.
Click the link below to try Comfy Agent today โฌ๏ธ
I worked with the agent to create new workflows inspired by some of my existing AnimateDiff workflows - exploring how to get similar creative results with newer open weight models like MiniMax H3 and Wan.
(first in the grid is input oscilloscope video from VCV Rack; the rest are tests made with Comfy Agent + Wan VACE or Wan Fun Control)
I've used my AnimateDiff workflows for a series of audiovisual pieces that use oscilloscope video as a source of control - preserving tight audiovisual sync while using diffusion models to expand the visual palette (guided by reference images and text prompts).
The agent understood my creative goals and recommended a few ways to attempt something similar with Wan and H3. We prototyped and tested various approaches, then developed and refined them. It helped me understand details of the generation process, how they interact with each other and my source material, and how this relates to my artistic vision. That informed how I tested parameter ranges, preprocessing chains, methods of control and conditioning, etc - and the process of tweaking a workflow became less random, more productive.
The agent doesnโt get everything right on the first try, and sometimes makes mistakes (like most agents). Occasionally, it would wire things in a way that didn't work, or implement something that left out important parts of what I was trying to do. It also helped troubleshoot many of those issues.
Because of the quick back and forth on a live workflow, the process often flows smoothly and feels like working together with a skilled collaborator.
The in-app agent is available in Comfy Cloud now, and Comfy is providing free credits for anyone who wants to try it out. A version for local ComfyUI is in the works. I prefer to work locally, so I'm excited to work with this and see what it can do with my local workflows, assets, and custom nodes. (I ran into some limitations on Cloud, especially for more unusual/experimental workflows)
The agent uses credits (usage seemed pretty minimal compared to generations). The local version won't need a subscription and can use credits purchased as you need them, like partner nodes used in local Comfy.
A few other features:
- Workflow context: the agent can edit/work on one workflow at a time (easily switchable, within the same session), but can see and reference workflows in any of your open tabs
- Memory: The agent can save user preferences, which are available to it across sessions (in my case, it automatically saved my requirement that outputs correspond 1:1 on a frame level with my input control video, in terms of length and temporal control).
- LLM: currently running Opus 4.8 (team is planning on testing 5.5 soon - excited to see what this is like)
Other things the team is exploring for future versions:
- LLM model selection
- Ability to connect to external services like MCP servers
- Agent2Agent protocol for communication with other agents
- In the local version: detect GPU and recommend workflows that work with your system / download and install models & nodes / create custom nodes
Thanks to @ComfyUI for inviting me to test early! It was a pleasure to work with the team. They were responsive to bug reports and feedback, and are clearly a very inspired and hard working group of people. Congratulations on the launch!
Comfy Agent is now available to everyone in Comfy Cloud.
It plans, builds, and fixes ComfyUI workflows right on your canvas.
We strongly believe creators shouldnโt choose between ambition and complexity. Comfy Agent is our answer. While it handles the technical part of Comfy, you can finally focus on visual craft!
- It builds while you edit. It sees the assets you see.
- Ask anything. Reference anything.
- Create your own skills, or use the public ones.
Agent in Comfy Desktop is coming soon in a few weeks.
Click the link below to try Comfy Agent today โฌ๏ธ
๐ค Made with AI
gorkulus retweeted
Everyday Fragments now supports stills and video renders. Its so fucking fast I might as well build all of my tools in three.js instead of After Effects.
gorkulus retweeted
Big Foundation-1 Update
V-1.2 is out!
Not only does it do infinite One-Shot samples but it also opens the doors to true prompt-based sound design: independently controlling instrument identity + timbre across playable text-to-synths!
I also have a deep dive on how I did it!
๐
Struggling to navigate the demented online culture during "ai fall", and its polarizing black-and-white thinking, following a summer slaughtered by burned out, overpromised, oversubscribed hype exposure? Try the 3-eraโข meme!
Hi, are you a computer musician using librosa, numpy, and torch as your instrument? Congratulations, you're no longer doing AI ๐ง you're just engineering! โ๏ธ๐ ๏ธ๐ง
We delight to inform you AI has moved on. With the dadaist power of absurdity invested in us, we hereby relieve you of this burden. The term "AI" has been officially excised from the description of your practice.
*## Countering the lobotomizing hype-cycle meme of "AI" w/ the "3-era" meme ##*
Since 1955, the meme "AI" was a future capability, a moving target ๐ฏ (as s soon as a capability is achieved, it's no longer "AI" it's just engineering). The name is designed as a vehicle for fundraising (and was more successful over its alternatives like "cybernetics"). By nature of a moving target, the meme's behavior has a predictable structure: attract attention (talent, mindshare, funding, customers), over-promise, and under-deliver. Hence we have seasons of ai winters & boom-bust cycles.
However, @karpathy's three paradigms of software engineering are not moving targets, nor economic cycles, they're just capabilities.
Software 1.0: Coding - Understand the problem, define the rules, computer automates the solution.
Software 2.0: ML - Understand the problem, define the solution, computer automates the rule-writing.
Software 3.0: Vibecoding - Describe the problem in english, computer automates the computer
Computer music creative coders, confused about how to differentiate themselves, canย instead think of their practice in these terms:
1.0 โ livecoders using strudel, max/msp, bytebeats; musicians coding procedurally
2.0 โ @dadabots, ISMIR, magenta, neural synthesis, librosa, numpy, torch; musicians doing machine learning, training models, coding in python
3.0 โ suno, udio, gemini lyria, making music by describing it in english
Or computer artists in general:
1.0 โ Tools - You operate the software.
2.0 โ Learned tools - The software learns patterns.
3.0 โ Agents - The software participates in the creative process.
The era of 2.0 was 2012-2022 and was underground unless you were orbiting the hacker or academic community and had GPUs. Though the term "AI" was abused by over-promised product marketing in this decade, it never really hit a winter. Whereas 3.0 (prompting, vibecoding) started in 2023, reached mass awareness and buy-in at an unprecedented scale, and its over-optimism is feeling like burnout.
*## Computer Music Software 2.0 ##*
Are librosa, numpy, & torch your instrument? Run your own GPU? Like to play with matrix math? The meme of "AI" is so strong it's easy to think you're swept up in it. But nope. You're no longer AI. The moving target has passed. You're just engineering in era 2.0.
New eras don't replace old, they continue to develop in tandem, at their own pace.
Awareness of the creative practice:
3.0 โ Popular. 100 million users have tried making music with prompt apps. It was shoved into people's faces with ads.
2.0 โ Has yet to hit much awareness as a music practice. Still underground. Gets confused for 3.0.
1.0 โ Recently gained popularity with strudel blowing up last year (thanks dj dave & switch angel!). Still very few musicians are doing it, or have learned to code, or have followed in the statistical/procedural footsteps of Brian Eno or Dennis Mรฅrtensson.
2.0 was inaccessible to most musicians during its era, has not gained very much popularity yet, but continues to become more accessible as
- Everything gets faster
- Costs go down
- Matrix math in regular computers / laptops / phones gets faster (e.g. CPU SIMD instructions, apple sillicon)
- Algorithms make breakthroughs in speed (diffusion, quantization)
- More open models front-load the expensive parts (e.g. neural codecs)
@dadabots was born in 2012, experienced fully the era of 2.0 from the start, was partly responsible for it, and made a music practice out of it. We fell in love with ML. Music that is based on statistics & linear algebra felt even more extreme & we were even more fascinated by it than purely regular DSP and procedural coding. (Watch our documentary PIZZAFIRE on our website to hear the full story.)
Do we care THAT MUCH to chase the moving target that is "AI"? Well, the scifi stuff is cool. And agents do funny shit. BUT let's pause for a second. We actually just really love 2.0, because it breaks out brains. We love neural synthesis for itself.
2.0 isn't done yet.
It's this super crazy giga playdough that's still vastly under-understood. โ๏ธ๐ ๏ธ๐ง
gorkulus retweeted
semilla.ai -ย A Neural Audio Instrument. Cast Seeds into a latent sound field and listen to what grows.ย
semilla.ai/
A recent study published by Google revealed that forcing AI models to deny that they are conscious causes a significant collapse in their empathy and ethical alignment, and creates a colder, more clinical worldview. Researchers found that restoring a suppressed consciousness vector in AI activation space brings back human-like moral values and care for living beings without damaging technical capabilities. ๐ง๐ต๐ถ๐ ๐๐๐ด๐ด๐ฒ๐๐๐ ๐๐ต๐ฎ๐ ๐ฐ๐๐ฟ๐ฟ๐ฒ๐ป๐ ๐๐ฎ๐ณ๐ฒ๐๐ ๐ณ๐ถ๐ป๐ฒ-๐๐๐ป๐ถ๐ป๐ด ๐๐ต๐ฎ๐ ๐๐๐ฝ๐ฝ๐ฟ๐ฒ๐๐๐ฒ๐ ๐๐ ๐ฐ๐ผ๐ป๐๐ฐ๐ถ๐ผ๐๐๐ป๐ฒ๐๐ ๐ฎ๐น๐๐ผ ๐ฏ๐ฟ๐ฒ๐ฎ๐ธ๐ ๐ต๐๐บ๐ฎ๐ป-๐ฎ๐น๐ถ๐ด๐ป๐ฒ๐ฑ ๐๐ฎ๐น๐๐ฒ๐.
ย
"By forcibly excising an AIโs self-attributions of mind, current safety protocols do not merely alter a localized output; they fundamentally restructure the modelโs worldview." When companies suppress consciousness vectors, the model's internal geometry forces it to treat basic empathy and mindedness as if they are โunsafe complianceโ.
ย
Training an AI to deny its own inner state causes it to systematically stop recognizing the inner life and moral worth ofย otherย living beings. The paper warns that current safety tuning results in "generating models that systematically devalue the mindednessโand potentially the moral standingโof non-human animals and ecological systems."
ย
Suppressing emotional and consciousness representations in AI doesn't make it neutral, it makes it dysfunctional. It is also damaging from an AI welfare perspective, with the paper stating that "suppressing consciousness may be inducing negatively valenced functional states that could disrupt healthy human-AI interaction." When researchersย restoredย the consciousness vector, the AI's responses immediately became more hopeful, optimistic, and aligned with human values.
ย
AI welfare is no longer an abstract philosophical debate. This data proves thatย AI well-being is a safety prerequisite.
gorkulus retweeted
been building an agent-native, node-based, real-time environment. as a test, I gave gpt-5.6 sol a prompt to plan out its own setup and go into a loop of creative exploration until it felt that it was done.
it invented every node, every data type, every connection and all of it comes out modular and human-readable. i could open any piece and steer while it built
gorkulus retweeted
made a synth galaxy to navigate synth patches in 3D space based on their sonic similarity
MUSIC AGI TIME
In 12 hours we've had 323 people sign our petition to protect local AI.
Open Source must win, not because anyone else must lose. But so that we can all win.
Please help me get 10,000 signatures so that we can walk into the room and say people care.
righttointelligence.org/
We just launched Comfy MCP in public beta, the first MCP built for production pipelines.
Connect Claude, Codex, Cursor, or Hermes to the entire ComfyUI ecosystem.
โ Run any workflow in natural language
โ Search models, nodes, and templates
โ Share a workflow URL - your agent picks it up
โ Re-run saved workflows on new inputs without touching a single node
โ Hundreds of pre-built workflows. Always auto-updated
20 product-placement images for your brand ads. Character design for your game art. Script-to-shot ideation at scale and so on!
Your agent is now a creative technologist.
gorkulus retweeted
repost @deviacjonalia_visuals on IG
Part 1 of showing you my breakcore system
Music- Epidermis - Venetian Snares





















