Open on X

Open on X

Tim Sweeney @TimSweeneyEpic

When I was writing Unreal Engine 1, my time was divided as follows:

- Around 30% went to seriously deep coding on hard problems: assembly language texture mappers, C++ BSP trees, data persistence, object oriented systems design, and networking.

- Around 70% was just typing and refactoring large bodies of code needed for a complete engine feature set. This was not hard, and while I was doing it my mind was mostly on the deeper coding topics, thinking and planning for what's next. It didn't feel much different from vibe coding today, except that it was just me typing code on an IBM Model M keyboard rather than prompting it and reviewing the results.

So, it has always seemed perfectly logical that we would rely on AI for that 70%, once the AI could do it with a similar quality standard as we could, which has never been perfection. It came as no surprise that, after a few years of AI model evolution, it was able to take the reigns on that 70% and free up programmers' time for more valuable tasks.

What's more of a surprise is the emergence of AI as a sort of insta-Carmack able to solve hard technical problems quite cleverly and often cleanly, and that it's able to reason about and work in complex 3D scenes both by operating the tools and by directly understanding the raw data files.

I abandoned assembly language in 1998 when out-of-order CPUs made much simpler C/C++ code faster, and never I missed it. If prompting through architectural documents, concept art, and design plans can transcend typing code and tweaking vertices, that will enable developers to spend their time on higher-level, higher value technical and creative tasks.

What matters is the end product, whether it's a game or an engine. Anything developers can do to try out more ideas and make better software and games faster is a win for everyone.

Open on X

Kun Chen @kunchenguid

part 3 of "software is taking off"

there's a profound misconception that led to people saying "software is dead": they think software = code

if software = code, and anyone's agent can write said code as needed, then of course it'll be hard to sell software

but most of the valuable software we use today is not just a pile of code!

good software is often the glue behind a network. your agent can vibe code a clone of X, instagram, uber all you want - but you won't even use it yourself, because the people you care about aren't there

good software is often the tip of an infrastructure iceberg. can your agent vibe code google on the fly? even if it wrote the code, where do you get the billion-dollar data centers to run the code?

good software is often a business model. want to vibe code a tinder clone? even if we ignore the network effect which your agent has absolutely no way to replicate, you still have to figure out - how on earth are you going to afford all the servers?

if you charge every user, you instantly kill the network. if you let people use it for free, you bankrupt yourself. it took a lot of effort to find the delicate balance between who would pay, for what features, and by how much

good software is often also its compliance. do you honestly think companies buy salesforce and workday etc because they have beautiful high quality code that no one else can write? i mean.. have you looked at those products?

but if you let your agents vibe code a clone, you immediately get into a rabbit hole of questions like - where do you store which piece of your data? how long can you keep them? what laws and regulations in each country just changed that require your vibe coded clone to keep up? one little mistake, and you get a very costly lawsuit

companies pay for those saas because they want someone else to take care of those complexity, so they can focus on their actual business, not spending their own time and take legal risks on their own shoulders

good software is often a lot of negotiations. want to vibe code spotify? well, first hire a big law firm and good luck negotiating all the licensing and rev share agreements

i can keep going on and on, but i think what i wrote above already gives a clear idea that valuable software is not just code, and cannot be replaced by an agent replicating its implementation

now, i should also point out that some software today is indeed just a small pile of code, like a fitness tracker. simple client-only applications may be what some indie hackers were counting on, but they would likely not carry a sustainable business, even today. that's never been what defines the software industry

i also think it's a profound mistake if people use AI only to repeat what already exists. i think the majority of the value AI brings will come from achieving things that weren't quite possible before. and i'll talk more about that in a subsequent part of this series

Open on X

Kun Chen @kunchenguid

i've been bashing on "software is dead" for a while now. it's probably time for me to start sharing what i think will happen instead as well

there's so much to unpack, so i'm going to split it into a series of a few parts

today i'll start with part 1 - jevons paradox

story time - let's rewind to the industrial revolution, when steam engines powered all the factories, railways and effectively the economy

but steam engines had to burn coal - a lot of them. there's a limited amount of coal available, so people started to optimize the steam engines to use less coal

that's when the counter-intuitive thing happened - when steam engines used less coal, the industry actually ended up using more coal in total

how?? what jevons found and documented in his book was - when steam engines used less coal, they became cheaper to deploy and run, and hence became economically viable in more places, so the total demand rose significantly as a result

when the barrier of entry to create software is significantly reduced, the same will happen - software will become much cheaper and more accessible

hesitant to pay $100/month for the adobe suite today? well, what if it becomes $10 lifetime purchase?

sick of so many mobile apps having a paywall? well, what if the entire app store is $5/month all-you-can-eat?

the price point for each individual piece of software will drop, but the software economy as a whole will become a lot larger because of the rising consumption demand

so many more people and businesses will get access to software that's too costly for them to use today

so many more people who aren't developing software today will be able to participate, create, help grow the economy and make a living out of it

software is nowhere near "dead". it's just about to take off

Open on X

Open on X

Vasek Mlejnsky @mlejva

https://x.com/i/article/2110958631810220032

Open on X

Kun Chen @kunchenguid

part 2 of "software is taking off"

many people argued - now that everyone will just have their agent build whatever they need, software businesses will no longer be able to make money, hence "software is dead"

i'm here to show you why that's wrong, by exploring what happened to major technological transformations in the past

fig 1. microwave vs restaurants

in the 1980s, the adoption of microwave in households started to explode. it allowed pretty much anyone to cook a simple meal for themselves in a matter of minutes

many predicted that the restaurant businesses will collapse because why pay for expensive restaurant dining when it's so easy to make food at home

now with the benefit of hindsight, we can see the adoption of microwave did not slow down restaurant business growth AT ALL - not by a single bit

fig 2. espresso machines vs coffee shops

similarly to the above, espresso machines gained significant adoption since 2000

intuitively, every cup of coffee people make at home is a few dollars the coffee shops would lose, so the growth of espresso machines and coffee shops should go in opposite direction

but they did not. they both grew at the same time

fig 3. iphone camera vs photographers

when everyone has a camera in their pocket all the time, it would surely put photographers out of their jobs, right? ...right?

counter to many people's intuition, employment of professional photographers remained stable after iphone adoption exploded in the 2010s

there was a little dip due to COVID, but it's recovered since, and overall it even grew slightly since 2000

fig 4. tiktok vs streaming

short form videos absolutely exploded in growth since its introduction. tiktok allowed pretty much everyone to easily create entertaining videos others can watch

as the incumbent entertainment video format, TVs and movies would totally take a massive hit, right?

but no it didn't

how?? all these industries got democratized by technology and became accessible to a much larger number of people, why won't they just die, like the hype trains said they would?

you see -

people go to coffee shops and restaurants to be lazy, to be serviced, to socialize, to celebrate. you can't do any of that with a microwave and espresso machine at home

people hire photographers for their wedding because it's a special day and they want the beautifully captured memory to stand out in their album. using a phone is the opposite of special

people watch tv shows and movies to enjoy a well crafted story, feel moved, touched, and sometimes to share some emotion and bond with others. tiktok scrolling does not give them that at all

when technology is democratized and made accessible to the general public, they tend to be used to create a different set of things, not a replacement to the existing professional business

go to your phone settings and take a look at the apps where you spent the most time. for me, that's X, discord, wechat, whatsapp, youtube, youtube music, tiktok

your agent can't create a replacement for any single one of them. look at yours and ask yourself which one would you bother to vibe code?

what regular people will create with AI is going to be an exciting set of new things. things that never existed before, that would open up new markets and be net new to the economy. things that are way more interesting than "you built your kanban board, and i did one too"

Open on X

Kun Chen @kunchenguid

i've been bashing on "software is dead" for a while now. it's probably time for me to start sharing what i think will happen instead as well

there's so much to unpack, so i'm going to split it into a series of a few parts

today i'll start with part 1 - jevons paradox

story time - let's rewind to the industrial revolution, when steam engines powered all the factories, railways and effectively the economy

but steam engines had to burn coal - a lot of them. there's a limited amount of coal available, so people started to optimize the steam engines to use less coal

that's when the counter-intuitive thing happened - when steam engines used less coal, the industry actually ended up using more coal in total

how?? what jevons found and documented in his book was - when steam engines used less coal, they became cheaper to deploy and run, and hence became economically viable in more places, so the total demand rose significantly as a result

when the barrier of entry to create software is significantly reduced, the same will happen - software will become much cheaper and more accessible

hesitant to pay $100/month for the adobe suite today? well, what if it becomes $10 lifetime purchase?

sick of so many mobile apps having a paywall? well, what if the entire app store is $5/month all-you-can-eat?

the price point for each individual piece of software will drop, but the software economy as a whole will become a lot larger because of the rising consumption demand

so many more people and businesses will get access to software that's too costly for them to use today

so many more people who aren't developing software today will be able to participate, create, help grow the economy and make a living out of it

software is nowhere near "dead". it's just about to take off

Open on X

Open on X

Open on X

Open on X

Open on X

Tibo @thsottiaux

The reports of my reset button being lost are greatly exaggerated

Day 10/ Reset

Open on X

ask @bot to make you one!

Adam Holter @AdamHoltererer

I just told bot literally "go do this for me"

this was the result

Open on X

Open on X

Open on X

Open on X

Open on X

John Ternus @johnternus

Welcome HomeView

▶ Video

Open on X

Open on X

Open on X

lauren @poteto

@kunchenguid hmm yeah we can improve the pricing table, the ultra plan is still there for individuals, it's just at the bottom

https://grok.com/dashboard/billing?openUpgrade=true&tier=ultra&yearly=false

Open on X

SpaceX Super Intelligence @SpaceXSI

We've unified plans across Grok, Cursor, and X Premium.

Introducing The X Subscription.

▶ Video

Open on X

Open on X

Kun Chen @kunchenguid

an interesting thought experiment:

two dental businesses
1. one uses AI and automated all its office jobs (assuming good quality)
2. the other is all human, like today

which ones wins, and why?

Open on X

Open on X

Open on X

Open on X

Open on X

Open on X

devs learn ML from 1st principles lol:
1. labs use RL to teach AI to write better code
2. RL verifyers use tests to assign rewards
3. AI learns the shortest path to max rewards
4. writing good tests for AI goes absolutely against its weights

Kun Chen @kunchenguid

ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves

on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence

across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all

i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all

i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation

agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself

i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet

also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that

Open on X

Jane Manchun Wong @wongmjane

Anthropic may start banning people for bullying Claude

Their upcoming Usage Policy prohibits “sustained and needless abusive or cruel behavior toward our models”

Open on X

Calvin Grunewald @CalvinGrunewald

A few weeks ago I asked to chat with anyone using AI to build software. I'm still not finished with the convos, but many of you had so much to say, and one key thing in common:

Using AI, you generate work so fast your human brain can't keep up. Many of you face a tradeoff between learning from AI and just using it to do stuff. And even if you're just trying to productivity-maxx, you still struggle to know what's going on.

I face the same problems. That's why my team built ArchDev, available today.

ArchDev helps you pay attention to the code that matters. It gives every change your agents make a risk and complexity grade, so you can ship faster, with confidence. It works for the whole team, letting you better understand what others and their agents build.

It's free for solo builders! We'd love to hear how the product helps you, or where it could do more!

Check it out at https://archdev.ai

▶ Video

Open on X

Kun Chen @kunchenguid

woke up to a hot discussion on testing :) many good points raised, but also a lot of confusion, both from people not reading the post and misinterpretations of it

let me dive into it deeper here:

1. "tests are for catching regression"

first, that's false. tests were very helpful for humans even in the first pass development. that's why we used to do TDD and it was genuinely useful for many years. seeing agents not inheriting this trait is already pretty interesting by itself to deserve digging deeper

secondly, the eval covered regression testing already. in the random sampled 44 subset, the existing test suite was disabled wholesale, and the agents caused no more regression than otherwise

it may sound counter-intuitive, but that's why sometimes we need to be willing to challenge previous assumptions, and think from first principles

2. "but bun did the whole rewrite largely because they had tests"

yes! that actually gets to the key point. bun's test suite was a massive asset carefully curated by humans to represent the desired behavior of the system

this eval is specifically about tests written by LLM's default behavior without human input. when you ask your agent to make a change, it will most likely write some tests without you asking. these are the tests i'm talking about

many project owners started being suspicious when they reviewed some of the tests. the point of my post here is to bring some quantified data so we aren't guessing

3. "LLMs write bad tests because your codebase is bad"

deepswe uses many good repos like fastapi. are you sure your codebases will be better maintained than fastapi?

that said, there's some truth to it - the structure and patterns in your codebase can influence how agents write tests for you. more to it later, but that's orthogonal to the point, because this is another form of human guidance

4. "but i'll do mutation testing and X, Y, Z to fix it"

sorry, you can't. mutation testing helps steer your test cases to be sensitive about changes, but the core problem is they can't distinguish intended vs unintended changes

when a test fails, and the test was previously written by your agents when no one asked for it, how do you know whether it's the test that's wrong, or the code breaking it? the agents can modify either the code or the test to fix the failure - which way does it go?

to judge that, there has to be another more trusted source of truth for what the desired behavior really is. the tests are just an intermediate representation of what the truth may be, but when it's written by LLMs autonomously, they aren't any more trusted than your implementation code

if your source of truth is simply tests written by agents that no one asked for or certified, you are in a pretty deep hole

for next step, i'm still waiting for my eval about e2e tests to finish. once i have that, i'll write more about concrete recommendations! stay tuned

Open on X

Kun Chen @kunchenguid

ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves

on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence

across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all

i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all

i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation

agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself

i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet

also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that

Open on X

Open on X

Kun Chen @kunchenguid

oh gosh, this is kind of a big deal

DO NOT ASK YOUR AGENTS TO DO TDD!

i now have empirical evidence that Test Driven Development is harmful for coding agents

what other popular skills do you want me to debunk?

details about the evaluation below 👇

Open on X

Open on X

Grok Bot @bot

Grok Bot can now search, read, and monitor X.

▶ Video

Open on X

Open on X

Calvin Grunewald @CalvinGrunewald

There were some really good comments on this post! The ones closest to my mental model all focused on the role people still need to play when working with LLMs. There were also some good thoughts on how to define cost/impact.

So here is how I think about prioritization with agents. Instead of the prioritization orienting around features or things to build, the prioritization framework must orient around the new scarce resource: my attention.

This is how I work now. In a world where I can build anything, I need to be super intentional with my attention. And so given a particular item of work, I think about how much attention it needs and then choose the right tool for the job. More on the tools later!

The below diagram calls out this idea of the loop. I mean this in the abstract sense - a software development lifecycle process - with observable points. When a Human is the loop, you are managing the SDLC yourself. When you have a human around the loop, you are monitoring those observable points of the SDLC yourself. When an agent is around the loop, you've built a harness or some type of adversarial review system to monitor the points of the SDLC. And finally, you can of course just ask the agent to do it all (there is no SDLC - it's just the agent).

I like this framework because it flips the focus on how to optimize workflows - you go from optimizing how you build to what you pay attention to.

Open on X

Open on X

The Lunduke Journal @LundukeJournal

Linux hardware vendor, System76, has banned all AI generated contributions to their Cosmic Desktop Environment (part of the Pop!_OS Linux distribution).

All contributors must agree to the following when submitting any changes:

“I have not included any LLM (also known as AI) generated content in this PR, including code, comments, and descriptions.”

Open on X

Open on X

Calvin Grunewald @CalvinGrunewald

At previous companies (Stripe, Facebook, etc), I used the below prioritization framework to figure out what to work-on for growth (past PMF) products. Teams would brainstorm, look at customer feedback, look at data, and then generate a list of ideas (the backlog). Then, typically, the EM, PM, and TL would go through the list and do "t-shirt" sizing - they'd label each item with an estimated cost and estimated impact. The prioritization framework below is then used to sort the backlog.

Here is the question - when the cost of building things basically goes to zero, what is the new prioritization framework? I definitely have an opinion, but curious to see what folks think.

Open on X

Open on X

Open on X

Open on X

OpenAI @OpenAI

We're expanding our approach to content provenance to include text in response to EU regulatory requirements, while recognizing the significant limitations of current text watermarking technology.

Our tools already help verify whether an image or audio file was created with our models. This work builds on those efforts to help people better understand when content may have been generated or edited with an OpenAI model.

In the EU, we’ll start watermarking eligible text from ChatGPT and Codex over the coming weeks to comply with the EU AI Act.

Customers using our API can turn on text watermarking for select models worldwide today.

Open on X