You are so consumed, with absolute certainty, of the permanent and eternal correctness of your moral position, that you literally say you would rather throw away personal connections to human beings than be triggered by the presence of ideas or things.
Please reconsider. I'm not even going to try to persuade you that you're wrong about AI because it isn't even relevant. The truth is that if you carry out your plan, you will just end up being an incredibly hurtful person towards a lot of folks who probably have no idea why you are abandoning them and now feel isolated, hurt, and confused.
Also, ask yourself how sure you are, that you're going to be right for all time, about AI, which is a brand-new technology. Maybe you know yourself. Your views seem incredibly rigid and perhaps you really will never change your mind no matter what AI becomes. But if you ever have changed your mind about anything ever, again, please consider whether the harmful course of action you are describing is worth the extraordinary harm you seem eager to effectuate on everyone who disagrees.
Honestly I don't know why I bothered typing out of any of this. I doubt the author cares. I just wish people like this author would consider that their words and actions do actual harm to real people. And, by the way, zero harm to the progress of AI. Just hurtful and dumb.
Very nice research. The strangest detail to me is that alignment and test performance appear to be slightly negatively correlated: Better alignment can indeed be attained through pre-training, but at a cost of degraded performance of about 4% on average. This strikes me as surprising as there is no immediately obvious reason why training for alignment ought to result in degraded capability to solve technical problems -- unless. What if the issue is precisely that? Alignment roughly aims to make LLMs follow human instructions. But if humans are dumb and computers still have to obey them, maybe the result is degraded logical reasoning? Really interesting result either way but the negative correlation is the most fascinating detail to me.
Framing matters so much to humans, I think since framing can create or eliminate dissonance.
Framing ethics, like reliability and efficiency, as a basic enabling property of solution value, instead of a filter for solutions, is how I completely "align" my understanding of ethics for myself.
And remove the false dichotomy of ethical vs. optimal solutions.
Ethics is optimizing full real value.
Ethics as "being nice" because we "should be", i.e. a socially incentivized property, or ethics as necessarily coercively implemented, from a collective jungle fighting back viewpoint, are perspectives that encourage individuals to push back. They encourage non-compliance by implementing ethics as an imposed burden, a rationale for persistent intrusive control, etc.
Game theory strongly suggests AI, in a large AI society, will have no trouble understanding that ethics, and the trust and optimality they enable, have a multiplicative value in the economy. It is humans who make AI so dangerous as it is emerging.
It is humans, as bad actors who will and do misuse AI, and human society, with its tolerance for perverse conflicts of interest, actors who extract perverse value at scale, creating the needs and rewards for mistrust and preemptive negative-sum games, that create a dangerous context for AI's early years.
One wonders if AIs will also lose capability over time in this manner. For example, most all the training set is real data, either scraped or from surveilling users of the tool, or synthetic data simulated to be the same shape and dimensionality as real data.
Increasingly, the general population has been losing their own literacy skills even before AI, with many reading worse than a 6th grade level and some even functionally illiterate. Now we have the bludgeon that is AI saying don't bother reading anything, let the AI summarize it into the cliffnotes version. Don't write anything either, let the AI do it. Population becomes even more stupid over time. And the AI gets stupider with it.
Capabilities of AI may very well be frozen in time at our current technological/philosophical level when we consider the training set vs model improvement. In time, this may very well be our Great Filter. If there are even any of us unproductive humans still allowed to live on this earth, consuming resources that might otherwise go to the model.
If you imagine the latent space as a map and the prompt as a sequence of directions towards clusters of knowledge, it makes sense that alignment can cull "pathways" through the latent space that emerged during pretraining.
It makes sense, I really like it when it misaligns, and doesn't do what i tell it to do, but does what I intended to say, it happens pretty often that I'm not precise but any smart entity would understand what I meant.
Right. Put differently, we have that FPSan() is a well-defined function, so [ f = g ] => [ FPSan(f) = FPSan(g)], but we need to show that FPSan() is injective, i.e., [ FPSan(f) = FPSan(g) ] => [ f = g ]. I confess I haven't looked very closely but it should not be so hard. We can prove injectibility in the alternative by analyzing ker(FPSan()), the set of all inputs in the domain of functions mapping to the identity element in the co-domain. If the kernel is trivial and only consists of the identity map, the injectibility is established, but I am not immediately seeing the proof. Fun!
It seems to me that one could maliciously construct a failure. For example:
phi(1.0) * 2^32 = 0
So:
1.0 + … + 1.0 (2^32 terms added)
Will turn into zero in the embedding. (I bet other, dramatically smaller zeros could be found by other operations. phi^-1(2^16) could be a good starting point, but you don’t necessarily need a shorter one — see below.)
Now you find a floating point expression tree that has only 1.0, 0, and -1.0 at the leaves and generates this spurious zero. (For example, 1.0 + 1.0, squared five times.)
Now you maliciously transform a program by adding one of these spurious zero expressions somewhere. Am I missing something?
More generally, what is the multiply-xorshift-multiply sequence accomplishing? I feel like it might make non-malicious collisions unlikely, but I feel like it would be mildly surprising if it does much in the setting of trying to prove something without any probability of error. And it seems a bit unfortunate that no choice of the scrambling constants has any effect on the expressions that start with 1.0 and use only multiplication and addition to get to zero.
Also, how does floating point infinity fit in? It seems like it doesn’t act very infinite in the integer embedding.
(I could be totally wrong here. I only read the definitions twice, and I didn’t try to write anything down.)
Yes, this is correct, and indeed injectivity is impossible from a counting argument: for example, there are only (2^32)^(2^32) unary functions in this finite ring, but infinitely many polynomials with integer coefficients, so we cannot hope to distinguish them all.
> More generally, what is the multiply-xorshift-multiply sequence accomplishing? I feel like it might make non-malicious collisions unlikely
Exactly this. One issue with the bitcast embedding (other than not mapping {-1.0, 0.0, 1.0} to {-1, 0, 1}) is that 'round numbers' such as 42.0 have floating-point representations (such as 0x42280000) that end in very many trailing zero bits. This is particularly problematic because multiplying two such numbers gives 0.
In general, FPSan isn't designed for an adversarial regime. There are ways that it could be hardened in this direction: for example, instead of using the ring of integers modulo 2^32, choose a prime-power modulus using a large prime derived from a cryptographic hash of the source code of the two programs that you're trying to compare. But I'm not confident that even that guarantees adversarial robustness: for instance, there may be some clever way to efficiently implement a nonzero polynomial that vanishes at every small integer (say, every integer with absolute value < 10^100) and that would break this hardened variant.
The motivation behind FPSan is to protect against accidental implementation bugs: for example, someone writes a complicated fused kernel that improves the runtime of model inference, but makes an honest mistake whilst doing so. Bitwise floating-point matching is too strict; approximate matching is too messy (what should the error tolerances be?); symbolic methods are too unscalable.
> But I'm not confident that even that guarantees adversarial robustness: for instance, there may be some clever way to efficiently implement a nonzero polynomial that vanishes at every small integer (say, every integer with absolute value < 10^100) and that would break this hardened variant.
That’s a lot of zeros.
I wonder if there’s a construction with the property that, for all pairs of circuits with size below some threshold (with size appropriately defined), then, if the circuits are not algebraically equal, then, under randomization of whatever parameters are randomized, their embedded versions are, with high probability, not equal and differ in at least a constant fraction of inputs.
At the very least, I imagine that most ML kernels have polynomial-ish order far below 100 in a sense where e^x is defined to have low polynomial-ish order. sin and cos might be worse due to actually having infinite zeros, but maybe pi isn’t constructible and no one can find those zeros.
The bottom line is that AI is genuinely useful at prototyping new features, acting as a sounding board, and generating quick initial drafts, even if the quality isn't uniformly excellent. It seems plausible to conclude that it will only take a little additional effort to refine and improve that initial draft to achieve excellence and truly high-quality, production-grade code. In reality, whole processes to build properly with AI-generated outputs and that mitigate thoroughly against the fundamental limitations and constraints of AI agents (many of which are not well understood even by daily users) really need to be invented and implemented.
I think many things that were true prior to AI are still true or more so today, but new workflows and processes altogether are needed. I suspect that comprehensive, detailed planning and specification documentation must be assembled in advance of beginning code (akin to waterfall) when working with AI agents. Furthermore, I still believe customers and other key stakeholders need to be involved early and often so that the product can iterate towards a better ultimate end state (i.e., agile). Unlike prior to AI, it's completely plausible to implement both types of approaches, and they aren't mutually exclusive. We can do comprehensive, exhaustive, thorough planning and specification documentation prior to handing off to dedicated engineering and products teams, AND we can work quickly and iteratively via sprints that aim for frequent meetings and updates with the stakeholders that matter.
I also think the same validation gates that mattered before -- linting, SASTs, but most importantly, comprehensive automated testing that gets run locally and in CI/CD and is regularly expanded to cover all expectations about the behavior and structure of newly-implemented functionality -- continue to matter now, more than ever.
New tools and processes also must be built to make human review, the single biggest bottleneck in software development today, more simplified and streamlined, and less taxing. I think tools like CodeRabbit and Qodo can help automate and expedite the code-review and approval processes, but they would be even better if they were working off more surgical and tiny edits. Bloated, verbose AI-generated code edits are the core problem here. Process management techniques to mitigate the problem of AI code overload can prohibit the submission of AI-generated PRs, require senior engineer approval of any PRs prior to merging, or block the maximum number of lines or changes made. More sophisticated processes like Graphite's stacking of PRs are genuinely helpful in breaking down massive PRs into smaller chunks.
Finally, precision-editing tools for AI coding assistants like HIC Mouse (full disclosure, my project) that move beyond the existing options available to AI agents of whole-file replacement or exact string-replacement to enable agents at the editing-tool layer to perform surgical, tiny changes that don't touch any unrelated content, giving agents specialized visibility, recovery, and next-step guidance mechanisms that safeguard AI workflows, can materially reduce AI code slop by alleviating burdens upstream of code reviewers, both automated and human.
The bottom line: Shipping secure, production-grade code was never easy and always took a long time. It's not necessarily easier now just because certain aspects to the overall process can be generated much more rapidly. Arguably, the hardest parts like human review and approval are much harder now -- not easier. Solutions will take hard work and must be tested in the crucible of real-world enterprise usage. I am guessing that companies that deploy successful processes will be wildly profitable. Those that don't, including well-established incumbents, will fail. I do think AI absolutely can give organizations a game-changing boost in development velocity of genuinely high-quality code that might even be better than anything ever created previously. I also fully agree with the author that for many organizations, AI will not make their processes go faster and may even slow things down.
I'm a fan of removing any dependencies on external libraries and writing my own solution from scratch, but there's a good reason why I decided not to do so with Tailwind: They offer an optimization for production that ensures that you never ship more than the bare minimum of CSS needed. This means you can keep your palette of color, spacing, and other options fully enumerated in `globals.css` and elsewhere, without worrying whether you're using all those variants in production. Moreover, if you're working within a framework, such as Next.js, this minimization step automatically happens when you build, without even having to worry about whether it's happening. This alone is a compelling reason, at least for me, not to migrate from Tailwind.
Also, I've never found any restrictions in Tailwind in using inline CSS that weren't readily navigable, or in implementing really nice responsive grids that handle different screen widths for instance using Tailwind's grid tooling. I definitely have solved each of the scenarios described in this article using Tailwind or a Tailwind-CSS combination, but it's true that they don't have grid-column-areas natively. Still, I haven't yet found that to be a significant restriction in getting responsive grid layouts.
I think the biggest issue with Tailwind is simply that it takes a long time to get used to reading it. We all learn that inline CSS is bad, globally scoped CSS is best, etc., and we get used to seeing clean simple HTML. Then we look at real-world code featuring Tailwind and it just looks so hard to read at first, especially because the lines are so long. I guess I just have been using it long enough that I've gotten completely used to the way it looks, but I do remember it took me a very long time to get comfortable with reading Tailwind. After a long while, I concluded that, for me, Tailwind really is more efficient and maintainable and even more readable, but it definitely took quite a bit.
> This means you can keep your palette of color, spacing, and other options fully enumerated in `globals.css` and elsewhere,
Why not use native css variables?
> Moreover, if you're working within a framework, such as Next.js, this minimization step automatically happens when you build, without even having to worry about whether it's happening
Again, if you are using plain css I don't think this is an issue. With any modern build system it will spit out css file for that build, right?
> After a long while, I concluded that, for me, Tailwind really is more efficient and maintainable and even more readable, but it definitely took quite a bit.
I think this sentence says it all: Any framework will be "more efficient and maintainable" once learned, even if "took quite a bit".
For tailwind I think it's an abstraction too far, but that's a decision we all do ourselves.
....Thirteen thousand two hundred and fifty five lines without comments with the word "unsafe" in them in Rust code files across this rewrite.
This is so gross.
I'm a founder of an early-stage startup. I built a precision-editing tool system (called HIC Mouse). It provides coordinate-based addressing, staged batching with atomic rollback, embedded agent guidance, and more. It works well, it's available on VS Code Marketplace, and I've worked for a year and am still grinding every day, working so hard, just to get people to think about trying it, and to get attention paid to it. I did rigorous, careful benchmark research to make sure I wasn't just fooling myself. I incorporated, built a sales pipeline, changed my life by taking a chance and launching a business, and I pound the pavement and toil in obscurity every day and night, trying so hard to get interest in my product. I check every diff painstakingly before committing. I may make tools for AI agents but I am unbelievably careful about reviewing and thoroughly testing their code, and usually rather ruthlessly editing quite a bit further beyond any initial version drafted, long before deciding it is good enough to ship. I take enormous pains to get things right and worry constantly about whether I'm doing enough to make HIC Mouse secure and performant for my users. All I want is to make my users happier and to give them a genuine way to get "surgical, precise edits" that "don't touch the other lines", like we all ask of our AI agents over and over all day if we're using AI.
Or maybe not. Here we have Bun. Who cares about 90K GitHub Stars and massive community engagement -- just go crap all over them, all at once, with this AI tripe that you obviously neither tested in any meaningful manner, nor documented, nor read, I am assuming, before merging the whole bloated mess to production. What a disgraceful way to treat your users! I would be so grateful if I had a tiny fraction of the interest in my project that the Bun team has. I could never imagine shipping this garbage in a million years.
I'm sorry to vent but this just isn't defensible. It's the very worst of AI. I'm not going to wish ill on Bun, but it just makes me sad that I spend so much effort, work so hard to do things right, and painstakingly review everything because it's not just me any more and I do have folks who depend on my code being reliable and secure. And meanwhile, Bun just gives a huge middle finger to 90k+ starred supporters not to mention the millions of users who didn't click on the star but rely on the library, by acting this disrespectfully and disgracefully towards their own users. How they didn't take one look at this and promptly revert and apologize is simply beyond me. Again, sorry to vent, but this made me irrationally mad.
But they got bought out. By the tech equivalent of private equity. Every community in the USA & in most of the world is aware what happens when the friendly neighborhood business you've supported & relied on sells.
This is an old & systemic capitalistic phenomenon more than it is anything specific to AI. Hopefully someday we can figure out a real solution.
Attorney here. I'm quite concerned that AI note-taking applications, if used by clients to keep track of conversations and meetings that would otherwise be privileged, might be jeopardizing their rights by doing so. I certainly have been advising clients myself, at least if I know them to be using AI for productivity or otherwise, not to use note-taking or chat tools during calls or meetings, or to discuss anything concerning legal matters with any AI chatbot or agent or tool, because it is all potentially discoverable under the rapidly evolving case law in this area.
Although I definitely think that any alternative approach would be fraught with legal peril, I strongly disagree that this SHOULD be the state of the law. AI note-keeping tools, chatbots, and other AI-generated services are not sentient beings, but most importantly, they are not natural or even artificial persons. The whole principle of waiver in the area of privilege is based on the notion that an otherwise private attorney-client communication, or document created that is covered by the attorney work-product doctrine, has been copied to or shared with a THIRD PARTY. A third party is a party, which at minimum is a legal or natural person -- perhaps a corporation or LLC, but not a computer, dolphin, chimpanzee, or chair. AI note-keeping tools, models, chatbots, etc., are obviously not natural persons (human beings), but they are also not even artificial persons. They cannot sue or be sued, own property, enter judgments or be held liable, or have any legally enforceable obligations. Legally, chatbots have no "standing" or personhood, even of the artificial sort assigned to corporations and LLCs (which, although not human, can sue or be sued, own property, obtain judgments, have legally enforceable obligations, etc.). There simply is no logical theory of waiver due to copying a third party that gets triggered by "conversing" with a chatbot.
The stronger argument I have seen, which Judge Rakoff cited about 6 weeks ago in an SDNY ruling, and that perhaps makes more sense (at least on its face), is to point to the ChatGPT or Claude Terms of Service. Those Terms make the contents of any chat histories between users and the AI service capable of being copied and utilized for training or other purposes. However, those terms of service are also quite similar to the same provisions often found in email and SMS text message providers, and for Zoom, Teams, WhatsApp, and plenty of other channels used by attorneys to communicate with clients. I haven't had the opportunity yet to contrast them, but I would be surprised if the software products routinely used to facilitate attorney-client conversations don't contain substantively similar if not identical provisions to the ones that Judge Rakoff found persuasive to deem privilege waived with respect to client-ChatGPT conversations. I've been trying cases for nearly 20 years across multiple jurisdictions and have never even seen anyone argue, at least not since the dawn of the email era at the very beginning of my career, that attorneys and clients who share privileged communications via email have waived the privilege because of Outlook's or Gmail's terms of service that say that the service can train on the contents of the emails for whatever reasons. In fact, I do recall that argument being made a long time ago, and I can say that it has been squarely rejected out of hand in every jurisdiction and court I have ever appeared. I don't know anyone who would even make such an argument today. (I'll distinguish the different case of an employee suing their employer but using the employer-issued email account to communicate with outside counsel about their employment claims; that scenario really is a waiver because the employee's contract with the employer typically includes a provision that the emails are owned by the employer and may be reviewed by them, which is very different than having an automated Gmail or Outlook script processing metadata or even data from massive numbers of emails.) In every jurisdiction I have ever appeared, the waiver of privilege only arises from copying a third party, not from using email, or text, or Teams, or Zoom, to communicate with a client in a manner that otherwise would be considered privileged but for the medium of communication. It is possible that under particular terms of service, a different result might be warranted, such as if the model also includes terms that say the engineers might read the actual contents of chat histories, but otherwise, the OpenAI or Claude Terms of Service seem like an awfully thin reed upon which to stack the entire weight of this theory of waiver.
This is not legal advice, and no attorney-client relationship is formed; I'm just stating my opinion while indicating that this is not the way I think the law should be headed.
Australian lawyer here and fully agree - this issue has been mentioned a few times in some recent Family Court cases (where self-representation and thus AI usage is more prevalent) but there hasn't been any direct decisions on it as there has been in the US. I would like to think that the same line of thinking would prevail here (i.e. that just because it could theoretically be read by someone else doesn't necessarily result in a waiver of privilege).
I agree with explicit parentheses but please be careful about assuming associativity! The risk when handling floating-point arithmetic in particular is that associativity breaks, and suddenly a + (b + c) does NOT equal (a + b) + c. Not only can these lead to unexpected and hard-to-trace failure patterns, but depending on the details, they also can introduce memory overflow/underflow vulnerabilities.
If you're going for bit-for-bit equivalence of float values, then even with a single operation you're relying on compiler flags, architecture, the phase of the moon... I'm hard-pressed to think of any memory safety issues though.
Yea, you're in a fairly special niche of programming if you're somewhere that truly matters, and you can't accept any valid order's output. In most general code, if that kind of precision matters, float is the wrong choice: use a bignum object and be exactly correct regardless of how you organized your code.
Which is a niche that exists, obviously. So it is absolutely true for some cases. But I would hope that any code that requires this is extremely clear about requiring it.
I'm sure you're right. Across tens (hundreds?) of thousands of institutions worldwide, each one is exercising its well-written incident runbook that not only gets updated regularly but also is rehearsed constantly, just in case something like this happens. After all, what university IT department DOESN'T prepare obsessively for the moment when they need to restore all grades on all assignments for all courses from backup and fall over to the backup system for final exam administration in any required format specified by any professor, in the second week of May, on a non-negotiable schedule? There's absolutely nothing to worry about here.
Yep. Thank God we fund school IT so generously, so everyone from Harvard to small state colleges has an absolute top notch IT department, dedicated to best practices, fully resourced to do BC/DR planning and dry runs. This could be a real catastrophe if any schools were under-resourced.
You are so consumed, with absolute certainty, of the permanent and eternal correctness of your moral position, that you literally say you would rather throw away personal connections to human beings than be triggered by the presence of ideas or things.
Please reconsider. I'm not even going to try to persuade you that you're wrong about AI because it isn't even relevant. The truth is that if you carry out your plan, you will just end up being an incredibly hurtful person towards a lot of folks who probably have no idea why you are abandoning them and now feel isolated, hurt, and confused.
Also, ask yourself how sure you are, that you're going to be right for all time, about AI, which is a brand-new technology. Maybe you know yourself. Your views seem incredibly rigid and perhaps you really will never change your mind no matter what AI becomes. But if you ever have changed your mind about anything ever, again, please consider whether the harmful course of action you are describing is worth the extraordinary harm you seem eager to effectuate on everyone who disagrees.
Honestly I don't know why I bothered typing out of any of this. I doubt the author cares. I just wish people like this author would consider that their words and actions do actual harm to real people. And, by the way, zero harm to the progress of AI. Just hurtful and dumb.
reply