Saturday Byte #001

I’ve been wanting to start a newsletter for a while.

So here we are.

Saturday Byte is basically what I’ve been building, reading, thinking about, and getting distracted by during the week.

No 40-link AI news dump.

Just the stuff I found interesting.

Let’s go.

The Byte: Laya

I came across Laya this week.

And I really liked the idea.

We’ve started throwing LLMs at basically everything.

Classify this.

Extract that.

Decide this.

Return JSON.

And then we spend half our time fighting with the model to make it return the right JSON.

Laya takes a different approach.

It doesn't generate text. It makes decisions.

Give it an input and ask something like:

"Which team should handle this?"

or

"Is this customer asking for a refund?"

It gives you a decision and a probability.

No token-by-token generation.

No giant response to parse.

Just a model making a decision.

I think this is going to be an interesting direction for AI.

Maybe every AI feature doesn't need a chatbot sitting behind it.

Thinking about: GPU costs

AI is getting expensive.

And I'm not talking about ChatGPT subscriptions.

I'm talking about the amount of compute we're burning to answer increasingly stupid questions.

Cloud GPUs are great until you actually look at the bill.

Then I saw Apple running a trillion-parameter model across four Mac Studios.

The setup can cost around $20k.

That's insane.

But also kind of interesting.

Because at some point you have to ask:

Do I rent the GPU forever, or just buy the damn thing?

Obviously, this doesn't work for everyone.

But for companies running inference constantly, local compute starts looking a lot more interesting.

The economics of AI are going to matter just as much as the models.

Also thinking about: iPhone Duo

Apple's first foldable iPhone is here.

7.6-inch display when open.

5.4-inch display when closed.

$1,999.

The price is hilarious.

The idea is not.

I'm curious about what happens when your phone can become a much bigger computer without you carrying another device.

We're already carrying ridiculously powerful computers in our pockets.

Now we're just figuring out what shape they should be.

Built: a LangGraph agent

I built a LangGraph agent and actually deployed it.

You can mess with it here:

https://frontend-rouge-eight-19.vercel.app/

Getting an agent to work is pretty easy.

Getting it to stop doing dumb shit is harder.

That's where LangGraph got interesting for me.

You can actually define how the agent moves through a task instead of throwing one giant prompt at an LLM and hoping for the best.

Search something.

Use the result.

Decide what to do next.

Search again if needed.

Eventually finish.

The annoying part is everything that happens when one of those steps goes wrong.

That's the part I enjoyed building.

Deep dive: How LLMs generate text

I wrote this one recently.

One thing I realised while writing it is that LLMs sound way more complicated than they actually are at a high level.

The loop is basically:

tokens → embeddings → attention → logits → sampling → repeat

That's it.

The model predicts the next token.

Then predicts the next one.

Then the next.

Over and over again.

Obviously, what's happening inside those steps is very complicated. Attention, KV caching, temperature, top-p, tokenization, all of that gets interesting pretty quickly.

But the basic idea is surprisingly simple.

I broke the whole thing down here:

https://harshshah.is-a.dev/blog/how-llms-generate-text

If you've ever wondered what actually happens between typing a prompt and getting an answer, this is the rabbit hole.

Rabbit hole: GLM

I somehow ended up reading about GLM's architecture.

This stuff gets crazy quickly.

We're talking about hundreds of billions of parameters, Mixture-of-Experts, sparse attention, and million-token context windows.

The part I found interesting was sparse attention.

If your context is a million tokens long, you probably don't want every token looking at every other token.

That's a ridiculous amount of computation.

So GLM uses an indexer to figure out which tokens are actually worth paying attention to.

GLM-5.3-Flash takes this further with a hybrid of sparse and linear attention, with 320B total parameters but around 18B active parameters.

And this is the part I actually care about.

How do you make these giant models cheaper to run?

Because throwing more GPUs at the problem forever isn't exactly a great business model.

The architecture tricks are getting more interesting than the benchmark numbers.

3 things I'm reading

GLM-5

The repo is hundreds of GB.

Obviously I downloaded it.

Just kidding. I have a life.

Z-AI's GLM Github Repo

Laya

If you want to see what a non-generative AI model looks like, play with this.

Read about Laya

My LLM deep dive

Yes, I'm shamelessly putting my own blog here.

Read it if you want to understand what's actually happening inside an LLM.

How LLMs Actually Generate Text: From Tokens to the Next Word

That's it for this week.

See you next Saturday.