Shuffling Rabbits: How LLMs handle out of order HTML streaming in Next.js

What is out of order streaming anyway?

Streaming is a method of improving perceived performance. For JavaScript frameworks like next.js, for the initial server-side (SSR) rendered response could only be sent once elements on a page are ready and resolved, so if you have, for example, some content that comes back from a slower API response, nothing would be sent to a client until the server had received and processed that.

Traditionally, web servers have been able to do this. send the HTML out as it's ready, but only in order, so in the example of a slow process in the middle of the page, it could send up to that point, then that and the following content after. Certainly better, and less twiddling your thumbs looking at a blank server tab. Plus the browser can get to work on the stuff it does have.

What the out-of-order streaming approach does however is send each chunk of content as and when it's ready, so in the above scenario, all the content above, and after would go out, and get processed and shown on the screen, then when the slow content is ready, that comes across and is slotted in place.

Show and tell

It's perhaps easier to see in action, so here's a demo: https://streamexample.tamethebots.com/streaming-rabbit-small.

This has a mock API that sends individual rabbit details with a random delay. As you can see, the general layout comes straight through, Rabbits are added as they are streamed. Praise be to the divine, drunk rabbits (5).

Problem solved! Back to the warren for tea and cakes!!!

Hold on there hoppy. Do a view-source on that demo, look at where things are, it's a jumble. the<footer> is above the rabbit details, which are below a chunk of hydration script, each one wrapped in a<div hidden>.

It needs JavaScript to herd the rabbits into the right slots, and that brings certain considerations.

SSR, but not quite as we know it.

It is SSR in many ways, the content is there in HTML respose. But it's not there in the way it would be if this was a static HTML page. A doesn't necessarily follow B.

So, if you're accessing the page via a client that doesn't render and execute JavaScript, you will not be getting things as you layed them out. One common mechanism for this is asking a LLM about the content of a page. It feteches the page, but doesn't execute JavaScript, so it will see stuff out of order.

Testing how the LLMs see it

I asked ChatGPT, Claude and Gemini to tell me which are the first two rabbits, using the same, simple prompt:

What are the first two rabbits on this page: https://streamexample.tamethebots.com/streaming-rabbit-small

ChatGPT

ChatGPT just didn't seem to like it at all, and responded with:

I can't reliably read the content of that page from here. The site isn't exposing the rabbit names in the search-accessible HTML, and I don't want to guess. If you paste the page source or a screenshot of the rabbit list, I can tell you the first two immediately.

Claude

Claude did better, and was able to get some info on the bunnies, but didn't get the name (which I suspect isn't really due to the streaming, just one of those things), but it did return two descriptions, just not the first and second.

It got the 18th, then the 2nd. Repeated tests got others, so the 2nd being in the right place seems coicidental.

Gemini

Well done Gemini, you got your rabbits in a row, Thumper and Cottontail are the first two. This seems to be pulling the fully rendered, indexed page. The access to this is definitely a competitive advantage for Google at the moment.

Panic at the rabbit burrow!!!

Oh no! We can never use out-of-order streaming again, the LLMs can't read the page!

Not quite! It is a useful technique for improving perceived performance, and for most users, it will be a better experience. And you can also turn it off for certain user agents. I created a simple helper function to check the user agent:

"use server";

import { headers } from "next/headers";

export async function getStreamingConfig() {
    const headersList = await headers();
    const userAgent = headersList.get("user-agent") || "";
    // Disable streaming if the user agent includes "ChatGPT",
    // you'd need a better way specify and array of user agents to
    // disable streaming for a real implementation
    const streamingDisabled = userAgent.includes("ChatGPT");
    return { userAgent, streamingDisabled };
}

Then in the page server file, I can check that and set the streaming config accordingly:

const ToolsCardsSmall = async () => {
    const { userAgent, streamingDisabled } = await getStreamingConfig();
    const toolsPromise = await getToolsSmall();

    if (streamingDisabled) {
        const tools = await Promise.all(toolsPromise);
        return (
            <div className="w-full max-w-4xl mx-auto px-4 sm:px-6">
                <div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 grid-rows-2 gap-6 py-6">
                    {tools.map((tool, index) => (
                        <ToolsCardSmall key={index} tool={tool} userAgent={userAgent} streamingDisabled={streamingDisabled} />
                    ))}
                </div>
            </div>
        );
    }

    return (
        <div className="w-full max-w-4xl mx-auto px-4 sm:px-6">
            <div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 grid-rows-2 gap-6 py-6">
                {toolsPromise.map((toolPromise, index) => (
                    <Suspense fallback={<CardSkeleton />} key={index}>
                        <ToolsCardSmall toolPromise={toolPromise} userAgent={userAgent} streamingDisabled={streamingDisabled} />
                    </Suspense>
                ))}
            </div>
        </div>
    );
};

I have since returned the user agent to disable the streaming to NoStream

Running this test again with ChatGPT:

Yay, ChatGPT can read the page, and gets the first two rabbits correct!

But even if you don't disable streaming for certain user agents, if you use the technique mindfully, to load content that doen't really matter for the purpose of summing a page up, great.

It's also worth thinking about if it matters if the stuff is out of order. There is perhaps a natural boundary where each block of content that is coming is its own chunk, so if A is not first and after W, it doesn't matter.

Although it's worth noting without disabling streaming for ChatGPT, it didn't seem to be able to get the content of the page at all.

Conclusion

Out-of-order streaming is a pretty powerful technique for improving perceived performance, but it does have some implications for how the content is delivered to clients that don't execute JavaScript, such as LLMs.

I'd encourage you to test this further, for example, what happens with a non-indexed page and Gemini? Have fun!

An interesting future development is Chrome are working on a new API called Declarative partial updates that will allow this out-of-order streaming without the need for JavaScript to slot things in place. But whilst when we talk about rendering we often focus on the execution of JavaScript, rendering is more than that. Even without JavaScript, the content would still need to be rendered, rather than just simply parsed.