The Llama 4 launch looks like a real debacle for Meta. The model doesn't look great. All the coverage I've seen has been negative.
This is about what I expected, but it makes you wonder what they're going to do next. At this point it looks like they are falling behind the other open models, and made an ambitious bet on MoEs, without this paying off.
Did Zuck push for the release? I'm sure they knew it wasn't ready yet.
I don't know about Llama 4. Competition is intense in this field so you can't expect everybody to be number 1. However, I think the performance culture at Meta is counterproductive. Incentives are misaligned, I hope leadership will try to improve it.
Employees are encouraged to ship half-baked features and move to another project. Quality isn't rewarded at all. The recent layoffs have made things even worse. Skilled people were fired, slowing down teams. I assume the goal was to push remaining employees to work even more, but I doubt this is working.
I haven't worked in enough companies of this size to be able to tell if alternatives are better, but it's very clear to me that Meta doesn't get the best from their employees.
For those who haven't heard of it, "The Hawthorne Effect" is the name given to a phenomena where when a person or group being studied is aware they are being studied, their performance goes up but as much as 50% for 4-8 weeks, then regresses to its norm.
This is true if they are just being observed, or if some novel new processes are introduced. If the new things are beneficial, the performance rises for 4-8 weeks as usual, but when it regresses it regresses to a higher performance reflecting the value of the new process.
But when poor management introduce a counter-productive change, the Hawthorne Effect makes it look like a resounding success for 4-8 weeks. Then the effect fades, and performance drops below the original level. Sufficiently devious managers either move on to new projects or blame the workers for failing to maintain the new higher pace of performance.
This explains a lot of the incentive for certain types of leaders to champion arbitrary changes, take a victory lap, and then disassociate themselves from accountability for the long-term success or failure of their initiative.
(There is quite a bit of controversy over what the mechanisms for the Hawthorne Effect are, and whether change alone can introduce it for whether participants need to feel they are being observed, but the model as I see it fits my anecdotal experience where new processes are always accompanied by attempts to meet new performance goals, and everyone is extremely aware that the outcome is being measured.)
> There is quite a bit of controversy over what the mechanisms for the Hawthorne Effect are, and whether change alone can introduce it for whether participants need to feel they are being observed
My vote is change alone can introduce it; going from not being observed to observed is a change itself. People get into a daily groove after adjusting to whatever circumstance or process (maybe that averages 4-8 weeks I have no idea), but introduce a novel thing into that and they "perk up" until that bit leaves or integrates into the daily groove. In my experience, people prefer working on new features, even if that feature is some managements arbitrary initiative. They rather work on that new ridiculous thing than continue on their previous slog through the bug backlog until the new thing becomes a slog itself.
is really a bad concept in this space, where you get limited shots at releasing something that generates interest.
> Employees are encouraged to ship half-baked features
And this is why I never liked that motto and have always pushed back at startups where I was hired that embraced this line of thought.
Quality matters. It's context-dependent, so sometimes it matters a lot, and sometimes hardly. But "moving fast and breaking things" should be a deliberate choice, made for every feature, module, sprint, story all over again, IMO. If at all.
> is really a bad concept in this space, where you get limited shots at releasing something that generates interest.
Sure, but long term effects are more depending on the actual performance of the model, than anything.
Say they launch a model that is hyped to be the best, but when people try it, it's worse than other models. People will quickly forget about it, unless it's particularly good at something.
Alternatively, say they launch a model that doesn't even get a press release, or any benchmark results published ahead of launch, but the model actually rocks at a bunch of use cases. People will start using it regardless of the initial release, and continue to do so as long as it's a best model.
I'm with you. Yet I've always understood 'move fast and break things' to mean that there is value in shipping stuff to production that is hard to obtain just sitting in the safe and relatively simple corner of your local development environment, polishing up things for eternity without any external feedback whatsoever, months or even years, and then doing a big drop. That's completely orthogonal to things like planning, testing, quality, taking time to think etc.
Maybe the fact that this is how I understood that motto is in itself telling.
I understand it as that too. But even then, I dislike it.
Yes, "shipped, but terrible software" is far more valuable than "perfect software that's not available". But that's a goal.
The "move fast and break things" is a means to that goal. One of many ways to achieve this. In this, "move fast" is evident: who would ever want to move slowly if moving fast has the exact same trade-offs?
"Break things" is the part that I truly dislike. No. We don't "break things" for our users, our colleagues or our future-selves (ie tech debt).
In other situations, the "move fast and break things" implies a preferred leaning towards low-quality in the famous "Speed / Cost / Quality Trade-off". Where, by decreasing quality, we gain speed (while keeping cost the same?).
This fallacy has long been debunked, with actual research and data to back it up¹: Software Engineering projects that focus on quality, actually gain speed! Even without reading research or papers, this makes sense: we all know how much time is wasted on fixing regressions, muddling through technical debt, rewriting unreadable/-manageable/-extensible code etc. A clean, well-maintained, neatly tested, up-to-date codebase allows one to add features much faster than that horrible ball of mud that accumulated hacks and debt and bugs for decades.
¹e.g. Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations is a nice starting point with lots of references to research and data on this.
I struggle with this because it feels like so many 'rules' in the world where the important half remains unsaid. That unsaid portion is then mediated by goodharts law.
If the other half is 'then slow down and learn something' its really not that bad, nothing is sacred, we try we fail we learn we (critical) don't repeat the mistake. Thats human learning - we don't learn from mistakes we learn from reflecting on mistakes.
But if learning isn't part of the loop - if its a self justifying defense for fuckups, if the unsaid remains unsaid, its a disaster waiting to happen.
The difference is usually in what you reward. If you reward ship, you get the defensive version - and you will ship crap. If you reward institutional knowledge building you don't. Engineers are often taught that 'good, fast, or cheap pick 2'. The reality is its usually closer to 1 or 1.5. If you pick fast...you get fast.
I agree. I think of it like a car engine. You can push it up to a certain RPM and it will keep making more and more power. Above that RPM, the engine starts to produce less power and eventually blows a gasket.
I think the performance-based management worked for a while because there were some gains to be had by pushing people harder. However, they’ve gone past that and are now pushing people too hard and getting worse results. Every machine has its operating limits and an area where it operates most efficiently. A company is no different.
The problem is, there's always some engine pushing the power envelope, or a person pushing their performance harder. And then the rest of them have to keep up.
You're not encouraged per se to ship half-baked features, but if you don't have enough "impact" at the end of the half (for mid cycle checkin) or year (for full PSC cycle) then you're going to get "Below Expectations" and then "Meets Most" (or worse) and with the current environment a swift offboarding.
When I was there (working in integrity) our group of staff+ engineers opined how it led to perverse incentives - and whilst you can work there and do great work, and get good ratings, I saw too many examples of "optimizing for PSC" (otherwise known as PSC hacking).
It's also terrible output, even before you consider what looks like catastrophic forgetting from crappy RL. The emoji use and writing style make me want to suck-start a revolver. I don't know how they expect anyone to actually use it.
> Employees are encouraged to ship half-baked features and move to another project
Maybe there is more to that. It's been more than a year since Llama 3 was released. That should be enough time for Meta to release something with significantly improvement. Or you mean quarter by quarter the engineers had to show that they were making impact in their perf review, which could be detrimental to the Llama 4 project?
Another thing that puzzles me is that again and again we see that the quality of a model can improve if we have more high-quality data, yet can't Meta manage to secure massive amount of new high-quality data to boost their model performance?
> Or you mean quarter by quarter the engineers had to show that they were making impact in their perf review
This is what I think they were referencing.
Launching things looks nice in review packets and few to none are going to look into the quality of the output. Submitting your own self review means that you can cherry pick statistics and how you present them.
That's why that culture incentivizes launching half baked products and moving on to something else because it's smart and profitable (launch yet another half baked project) to distance yourself from the half baked project you started.
I like how Netflix set up its incentive systems years ago. Essentially they told the employees that all they needed to do is deliver what the company wanted. It was perfectly okay that an employee did their job and didn't move up or do more. Per their chief talent officer McCord, "a manager's job is all about setting the context" and the employees were let loose to deliver. This method puts a really high bar on the managers, as the entire report chain must know clearly what they want delivered. Their expectation must be high enough to move the company forward, but not too ridiculous to turn Netflix into a burnout factory.
Unfortunately I wasn't able to get an interview with Netflix.
> employees that all they needed to do is deliver what the company wanted
How did this work out in practice and across teams?
My experience at Meta within my team was that it would be almost impossible to determine what the company actually wanted from our team in a year. Goals kept changing and the existing incentive system works against this since other teams are trying to come up with their own solutions to things which may impact your team.
> an employee did their job and didn't move up
Does Netflix cull employees if they haven't reached a certain IC level? I know at Meta SWEs need to reach IC5 after a while or risk being culled.
Netflix used to have a single level for engineers: Senior. That was the best decision they made, at least when they were small. It simply eliminated any incentive to work for promotion, and the result was magnificent: engineers would generally work on what's right, if they were ambitious. Or they could choose to finish what they were supposed to do. Netflix's culture deck explicitly said it was okay for employees to just do what their roles required, so no pressure to move up at all.
> How did this work out in practice and across teams? My experience at Meta within my team was that it would be almost impossible to determine what the company actually wanted from our team in a year.
That's why it is critical to have good leaders. A problem, at least per my own experience in Meta, is that many managers are people managers. I'm not sure what they want. It's hard to know what product they want to build, what gaps they want to fill, or what efficiency goals they want to drive. If they were in Netflix, they would fail the "setting the right context" test, as they didn't know what the right context should be.
Yep they've basically created a culture where people are incentivized to look busy, ship things fast, and look out for themselves. Which attracts and retains people that thrive in that kind of environment.
That's a terrible way to execute on big, ambitious projects since it discourages risky bets and discourages collaboration.
It's not a big deal. Llama 4 feels like a flop because the expectations are really high based on their previous releases and the sense of momentum in the ecosystem because of DeepSeek. At the end of the day, LLama 4 didn't meet the elevated expectations, but they're fine. They'll continue to improve and iterate and maybe the next one will be more hype worthy, or maybe expectations will be readjusted as the specter of diminishing returns continues to creep in.
It feels like a flop because it is objectively worse than models many times smaller that shipped sometimes ago. In fact, it is worse than earlier LLaMA releases on may points. It's so bad that people who initially ran into it assumed that the downloaded weights must be corrupted somehow.
The switching costs are so low (zero) that anyone using these models just jumps to the best performer. I also agree that this is not a brand or narrative sensitive project.
> it makes you wonder what they're going to do next
They're just gonna keep throwing money at it. This is a hobby and talent magnet for them, instagram is the money printer. They've been working on VR for like a decade with barely much results in terms of users (compared to costs). This will be no different.
Both are also decent long-terms bets. Being VR market leader now means they will be VR market-leader with plenty of inhouse talent and IP when the technology matures and the market grows. Being in the AI race, even if they are not leading, means they have in-house talent and technology to be able to react to wherever the market is going with AI. They have one of the biggest messengers and one of the biggest image-posting sites, there is a decent chance AI will become important to them in some not-yet-obvious way.
One of Meta's biggest strengths is Zuckerberg being able to play these kinds of bets. Those bets being great for PR and talent acquisition is the cherry on top
This assumes no upstart will create a game changing innovation which upends everything.
Companies become complacent and confused when they get too big. Employees become trapped in a maze of performative careerism, and customer focus and a realistic understanding of threats from potential competitors both disappear.
It's been a consistent pattern since the earliest days of computing.
Nothing coming out of Big Tech at the moment is encouraging me to revise that assumption.
I remember reading that they were in panic mode when the DeepSeek model came out so they must have scrambled and had to re-work a lot of things since DeepSeek was so competitive and open source as well
Fear of R2 looms large as well. I suspect they succumbed to the nuance collapse along the lines of “Is double checking results worth it if DeepSeek eats our lunch?”
Do you know that they made a bet on MoE? Meaning they abandonded dense models? I doubt that is the case. Just releasing MoE Llama 4 does not constitute a "bet" without further information.
Also from what I can tell this performs better than models with parameter counts equal to one expert, and worse than fully dense models equal to total parameter count. Isn't that kind of what we'd expect? in what way is that a failure?
Maybe I am missing some details. But it feels like you have an axe to grind.
A 4x8 MOE performs better than an 8B but worse than a 32B, is your statement?
My response would be, "so why bother with MOE?"
However deepseek r1 is MOE from my understanding, but the "E" are all =>32B parameters. There's > 20 experts. I could be misinformed; however, even so, I'd say a MOE with 32B or even 70B experts will outperform (define this!) Models with equal parameter counts, because deepseek outperforms (define?) ChatGPT et al.
DeepSeek V3/R1 are MoE with 256 experts per layer, actively using 1 shared expert and 8 routed experts per layer https://arxiv.org/html/2412.19437v1#S2:~:text=with%20MoE%20l... so you can't just take the active parameters and assume that's close to the size of a single expert (ignoring experts are per layer anyways and that there are still dense parameters to count).
Despite connotations of specialized intelligences the term "expert" provokes it's really mostly about scalability/efficiency of running large models. By splitting up sections of the layers and not activating all of them for each pass a single query takes less bandwidth, can be distributed across compute, and can be parallelized with other queries on the same nodes.
Easy, vastly improved inference performance on machines with larger RAM but lower bandwidth/compute. These are becoming more popular such as Apple's M series chips, AMD's strix halo series, and the upcoming DGX Spark from Nvidia.
yes i understand all that. I was saying the claim is incorrect. My understanding of deepseek is mechanically correct but apparently they use 3B models as experts, per your sibling comment. I don't buy it, regardless of what they put in the paper - 3B models are pretty dumb, and R1 isn't dumb. No amount of shuffling between "dumb" experts will make the output not dumb. it's more likely 32x32B experts, based on the quant sizes i've seen.
I'm not a DeepSeek employee but I think there is more clarification needed on what an "expert" is before the conversation can make any sense. Much like physics, one needs at least take a glance at how the math is going to be used to be able to sanity check a claim.
A model includes many components. There will be bits that encode/decode tokens as vectors, transformer blocks which do the actual processing on the data, some post-transformer block filtering to normalize that output, and maybe some other stuff depending on the model architecture. The part we're interested in involves parts of the transformer blocks, which handle using encoded relational information about the vectors involved (part 1) to transform the input vector using a feed forward network of weights (part 2) and then all that gets post processed in various ways (part 3). A model will chain these transformers together and each part of the chain is called a layer. A vector will run from the first layer through to the last, being modified by each transformer along the way.
In an MoE model the main part about the transformer block changed is part 2, which goes from "using a feed forward network of weights" to "using a subset of feed forward network weights chosen by a router for the given token and then recombined". In the MoE case each subset of weights per feed forward layer is what is called the "expert". Importantly, the expert is not a whole model - it's just a group of the weights available in a given layer. Each layer's router choses which group(s) of weights to use independently. As an example, if a 10 layer model had a total of 10 billion 8 bit parameters in the feed forward layers (so a >10 billion parameter model in overall parameters) and 10 experts that means each expert is ~100 MB (10 billion bytes / 10 layers / 10 experts per layer). These 10 billion parameters would be referred to as the sparse parameters (not always used each token) while the rest of the model would be referred to as the dense parameters (always used each token). Note: folks on the internet have a strong tendency to label this incorrectly as "10x1B" or "10x{ActiveParameters}" instead.
The "Mixture" part of MoE extends a bit further than "the parameter groups sit next to each other in the transformer block" though. Similar in concept to how transformer attention is combined in part 1, more than 1 expert can be activated and the outputs combined. At minimum there is usually 1 expert which is always used in a layer (the "shared expert") and at least 1 expert which is selected by the router (the "routed experts"). The shared expert exists to make the utilization of the routed experts more even by ensuring base information which needs to be used all the time is dedicated to it which increases training performance since the other experts can be more evenly selected by the router as a result.
With that understanding, the important takeaways are:
- Experts are parts of individual parameter networks in each layer, not a sub-model carved out.
- More than 1 expert can be used, and the way the data is combined is not like feeding the output of a low parameter LLM to another, it's more like how the first phase of the transformer has multiple attention heads which are combined to give the full attention information.
- There are a lot of other weights beyond the sparse weights used by experts in a model. The same is true for the equivalent portions of a dense model as well though. This ultimately makes comparing active parameters of a sparse model to total parameters of a dense model a valid comparison.
As to the original conversation: DeepSeek v3/R1 has 37 Billion active parameters so that should set the floor for a comparative dense model, not whatever the size of an individual expert in part of a single transformer layer is (which acts as a bit of a red herring in these kinds of conversations, doubly so since more than 1 experts worth of weights are used anyways). In reality a bit more than that, though definitely less than if all 671 Billion parameters were dense. While we don't have much concrete public information about modern version of ChatGPT, one thing we're relatively certain of is "ChatGPT 4.5 has a TON more parameters and has basically nothing to show for it". Meanwhile Gemma 3, a 27 B locally runnable model from Google, is a hair behind DeepSeek v3 in many popular benchmarks. Truth is, there are just a lot of other things beyond parameter count that go into making a well performing model and DeepSeek v3/r1 hit (or invented) a lot of them. If I had to place a bet on this whole comparison, I'd say OpenAI is very likely also using sparse/MoE-style architectures in their current products anyways.
As a final note, don't just dismiss the architecture of DeepSeek with "regardless of what they put in the paper"! The model files are available and they include this sort of layer/parameter information so your device is able to run them (well, "run" requires a bit of RAM in this case... but you can still read the model file on disk and see it's laid out as the paper claims regardless).
This is about what I expected, but it makes you wonder what they're going to do next. At this point it looks like they are falling behind the other open models, and made an ambitious bet on MoEs, without this paying off.
Did Zuck push for the release? I'm sure they knew it wasn't ready yet.