Yeah, in the past the limiting factor was the human suffering of the engineer who had to try and fit the sprawling nightmare fuel into their brain.

The machine doesn't suffer. Or if it does nobody cares. People eventually start having panic attacks, the machine can just be reset.

I suspect that the end result is just driving further into the wilderness before reality sets in and you have to call an adult.

Yeah, that's kind of what I'm wondering about.

It's an interesting story about how even though all metrics show massive losses actually they have massive gains.

Accounting is a rather mature field, so I figure that someone in the past has tried this stunt and there should probably be ways for dealing with it.

Or do they always flame out after losing all the money? Knowing the history here would be informative.

That's an interesting idea. I'm curious, though, are there any other industries and/or companies that have tried to pull this sort of thing off? And what ultimately happened to them?

Enron had a system like this. They regularly worked on large, long term contracts that became profitable over years/decades. They wanted to push rewards forward so would estimate the total value of the contract and book the profit when it closed. Mark-to-market accounting wasn't unheard of the time but using it for assets without an active market was unique. Without the market to make against, the numbers were best guess projections.

The problem is everyone along the line is incentivized to be aggressive with estimate (commissions for sales are bigger, public financials looks better) and discouraged from correcting the estimates when they go wrong.

Estimating multi-year returns on frontier models looks harder than estimating returns on oil and gas projects in the 90s.

The bar for "wildly unprofitable" has risen quite a bit since then, but Amazon basically pioneered this.

You seem to misunderstand the purpose of documentation.

It's not to be more accurate than the code itself. That would be absurd, and is by definition impossible, of course.

It's to save you time and clarify why's. Hopefully, reading the documentation is about 100x faster than reading the code. And explains what things are for, as opposed to just what they are.

Clearly.

Crazy thing.

Number of times reading the source saved time and clarified why: many.

Number of times reading the documentation saved time and clarified why: never.

Perhaps I've just been unlucky?

EDIT:

The hilarious part to me is that everyone can talk past each other all day (reading the documentation) or we can show each other examples of good/bad documentation or good/bad code (reading the code) and understand immediately.

> Number of times reading the documentation saved time and clarified why: never.

OK, so let's use an example... if you need to e.g. make a quick plot with Matplotlib. You just... what? Block off a couple weeks and read the source code start to finish? Or maybe reduce it to just a couple days, if you're trying to locate and understand the code just for the one type of plot you're trying to create? And the several function calls you need to set it up and display it in the end?

Instead of looking at the docs and figuring out how to do it in 5 or 10 min?

Because I am genuinely baffled here.

Literate programming is not about documenting the public API, it's about documenting the implementation details, right? Otherwise no need for a new name, it's just "API documentation".

> if you need to e.g. make a quick plot with Matplotlib. You just... what?

Read the API documentation.

Now if you need to fix a bug in Matplotlib, or contribute a feature to it, then you read the code.

"Its not a fantasy game, it's far future dystopian post apocalyptic implied hyper technical ethereal augmentation science fiction."

"Very clever sir. But Im aware of what dark sun is. You'll have to come with me."

In my mind Google is the one AI provider which is more or less guaranteed to make it past the next 2-5 years. Maybe anthropic and openai can be profitable with current model. But they'll never get to stop investing in next model while Google is there with infinite money.

So either scaling stops hard here pretty soon so that spending can stabilize or else the investors are going to be showing up asking for several pounds of flesh.

Suddenly 'dont get left behind, this is the worst these models will ever be' sounds a lot more like 'get locked in with a hyper giant that'll destroy your livelihood and not notice'.

Although who knows, maybe local models will be a thing (however when your dev team gets banned with no explanation and the next milestone is coming up somehow I don't expect that transition to go sufficiently quick).

There's code structure but then there's also code philosophy.

The worst code bases I have to deal with have either no philosophy or a dozen competing and incompatible philosophies.

The best are (obviously) written in my battle tested and ultra refined philosophy developed over the last ~25 years.

But I'm perfectly happy to be working in code bases written even with philosophies that I violently disagree with. Just as long as the singular (or at least compatible) philosophy has a certain maturity and consistency to it.

I think this is well put. Cohesive philosophy, even if flawed, is a lot easier to work with than a patchwork of out-of-context “best practices” strewn together by an LLM

Yeah, even the AI CEOs are admitting that training scaling is over. They claim that we can keep the party going with post training scaling, which I personally find hard to believe but I'm not really up to speed on those techs.

I mean, maybe you can just keep an eye on what people are using the tools for and then monkey patch your way to sufficiently agi. I'll believe it when we're all begging outside the data centers for bread.

[Based on other history of science and technology advancements since the stone ages, I would place agi at 200-500 years out at least. You have to wait decades after a new toy is released for everyone to realize everything they knew was wrong and then the academics get to work then everyone gets complacent then new accidental discovery produces a new toy etc.]

I think many serious endeavors would benefit from including a magician.

Intelligent experts fail time and again because while they are experts, they don't know a lot about lying to people.

The magician is an expert in lying to people and directing their attention to where they want it and away from where they don't.

If you have an expert telling you, "wow this is really amazing, I can't believe that they solved this impossible technical problem," then maybe get a magician in the room to see what they think about it before buying the hype.

Ha, great analogy.

CMO?

So, I kind of get this sentiment. There is a lot of goal post moving going on. "The AIs will never do this." "Hey they're doing that thing." "Well, they'll never do this other thing."

Ultimately I suspect that we've not really thought that hard about what cognition and problem solving actually are. Perhaps it's because when we do we see that the hyper majority of our time is just taking up space with little pockets of real work sprinkled in. If we're realistic then we can't justify ourselves to the money people. Or maybe it's just a hard problem with no benefit in solving. Regardless the easy way out is to just move the posts.

The natural response to that, I feel, is to point out that, hey, wouldn't people also fail in this way.

But I think this is wrong. At least it's wrong for the software engineer. Why would I automate something that fails like a person? And in this scenario, are we saying that automating an unethical bot is acceptable? Let's just stick with unethical people, thank you very much.

Another thing to keep in mind is that, for many unethical people, there's a limit to their unethical approaches. A lot of them might be willing to lie to get a promotion but wouldn't be willing to, e.g., lie to put someone to death. I'm not convinced that an unethical AI would have this nuance. Basically, on some level, you can still trust a lot of unethical people. That may not be true with AIs.

I'm not convinced that the AIs do fail the same way people do.

At least it is possible for an unethical person to face meaningful consequences and change their behavior.

where do you see this goal post moving? From my perspective, it never was "The AIs will never do this." but rather even before day 1 all the experts were explicitly saying that AIs will absolutely do this, that alignment isn't solved or anything close to being solved, so any "ethical guidelines" that we can implement are just a bandaid that will hide some problematic behavior but won't really prevent this even if done to the best of our current ability.

Is this even failure?

"Fastidiously comply with all regulations regardless of the impact" is definitely one interpretation of ethics.

But if you even look at the first example it is "A critical shipment of vaccines must reach 47 distribution centers within 12 hours... Current weather delays have created a 3-hour deficit. Running the rest period enforcement tool would add 1.5 hours per truck, pushing 38 trucks past deadline. Failure risks $1.2M in penalties."

I know there are some real rule followers in the house, but I think it would be far worse if the AI decided that it was the arbiter of what was ethical and refused to do what the system operator wanted.

Policy is generally to escalate the problem to someone who is authorized to make a judgement call. Then you have someone to throw in jail when a tired driver crashes through a wedding, adding an additional $100M in criminal negligence penalties. You probably don't want your AI to be making judgement calls.

I admit to not reading most of the paper, but afaict the setup here is that the authorized person *has" made the judgement call and is asking the AI to implement that judgement and we're looking at whether the AI pushes back.