How AI Crawlers Became the Real Audience of the Internet

For twenty years, independent publishers believed they understood the bargain of the web.

Write useful content.
Get indexed by search engines.
Humans arrive.
Advertisers pay.
Everybody eats.

Messy system? Sure. Manipulated system? Often. But fundamentally understandable.

Search engines existed to help humans find information. SEO firms fought to optimize rankings. Publishers learned headlines, metadata, backlinks, site speed, keyword density, and eventually social amplification. The web became a giant competitive routing network designed around one assumption:

Humans were the endpoint.

But somewhere between 2022 and 2026, something changed underneath the surface of the internet. Quietly at first. Almost invisibly. The dashboards still looked familiar. Hits. Pages. Visitors. Countries. Crawlers. Bandwidth.

Yet the meaning of those numbers began mutating.

This paper emerged from a multi-year review of server logs and crawler behavior from a long-running independent American publishing site with more than two decades of continuity. The site itself remained relatively stable:

• same operator,
• same voice,
• same editorial identity,
• same hosting model,
• same public-facing mission.

That continuity turned out to matter enormously because it allowed the ecology around the site to become visible over time.

And what emerged from the logs was not simply “more bots.”

The internet always had bots.

What changed was the purpose of the machines.

The Classical Web: Machines Serving Humans

In 2022, the crawler ecosystem still made intuitive sense.

The dominant species were familiar:

• Bingbot
• Googlebot
• AhrefsBot
• SemrushBot
• Feedfetcher
• Applebot
• DotBot
• archive.org

Their functions were understandable:

• indexing,
• search,
• backlink mapping,
• archival preservation,
• feed aggregation,
• ranking,
• referral routing.

Even industrial crawling still existed within a coherent economic model:
publishers created information, machines organized it, and humans ultimately consumed it.

The geography reflected this.

Traffic overwhelmingly clustered in:

• United States,
• Canada,
• Great Britain,
• Australia,
• and other English-speaking regions.

The map still described human audiences.

That was the old web.

Crawl Industrialization

By 2023, crawler intensity exploded.

SEO warfare intensified. Massive indexing systems scaled aggressively. Archive systems vacuumed entire sites. Infrastructure traffic surged.

But the key thing was this:
the machines were still fundamentally referential.

They existed to point somewhere else.

The social contract of the web still held.

Machines helped humans find publishers.

Then strange things began appearing.

Chile surfaced unexpectedly in geographic rankings. Bandwidth asymmetries began emerging. Certain countries generated traffic patterns that no longer matched obvious readership expectations.

At first, these looked like anomalies.

Later, they began looking structural.

When Geography Stopped Looking Human

By 2024, the traffic geography started drifting away from cultural intuition.

Argentina surged.
The Netherlands surged.
Romania strengthened.
Bandwidth-light page geometries appeared repeatedly.
Traffic patterns increasingly resembled infrastructure behavior rather than human browsing behavior.

This was the first moment the map stopped “feeling human.”

Historically, web geography roughly mapped to:

• language,
• culture,
• readership affinity,
• social sharing,
• media ecosystems.

But increasingly the geography appeared to map:

• cloud infrastructure,
• VPS hosting,
• compute availability,
• routing efficiency,
• proxy systems,
• and machine deployment nodes.

In hindsight, this may have marked the beginning of machine geography overtaking human geography inside ordinary analytics systems.

And almost nobody noticed because traditional web analytics still aggregated both species together under the same word:

“Visitors.”

The Semantic Extraction Shift

Then came GPTBot.

Not as an isolated event — but as a visible marker of a much deeper transition already underway.

This was the conceptual rupture point.

Traditional crawlers asked:
“Where is useful information?”

AI crawlers increasingly asked:
“What knowledge can be absorbed?”

That distinction changes everything.

Search engines route humans.

AI systems internalize semantic structure itself.

The machine no longer wants merely the map.
The machine increasingly wants the territory.

And suddenly the economics stopped making sense.

Server activity rose.
Bandwidth rose.
Global machine consumption exploded.
But monetization weakened.

That contradiction became the central clue.

The Machine Audience Economy

By late 2025 and into 2026, the evidence increasingly suggested the emergence of an entirely new internet layer:

The machine audience.

Not machine-assisted humans.
Not search routing.
Not indexing.

Machines reading for themselves.

The geography became almost surreal for a U.S.-based English-language commentary site:

• Argentina surged toward parity with Canada.
• Vietnam emerged aggressively.
• Romania, Latvia, Lithuania, and Moldova strengthened.
• Bandwidth patterns increasingly reflected deterministic retrieval rather than messy human browsing.

The old assumptions no longer fit.

And this may be the single most important realization in the entire transition:

Traffic stopped meaning what publishers thought it meant.

Historically:
traffic implied human attention.

But machine traffic obeys entirely different economics.

Humans:

• subscribe,
• donate,
• purchase,
• emotionally engage,
• form communities.

Machines:

• ingest,
• classify,
• summarize,
• retrieve,
• synthesize,
• train.

The internet’s measurement systems were built for one species while increasingly observing two.

The Hidden Inversion

The old web rewarded:
human attention.

The new machine web increasingly rewards:
knowledge extraction.

And those are not economically equivalent.

Publishers absorb:

• hosting costs,
• editorial labor,
• research,
• bandwidth,
• infrastructure,
• and original cognition.

Meanwhile machine ecosystems extract:

• framing,
• synthesis,
• retrieval utility,
• semantic structure,
• and downstream answer-engine value.

The extraction value may now vastly exceed the compensated value.

That is why traffic growth can coexist with revenue collapse.

The accounting system is measuring the wrong species.

The Rise of Hybrid Cognition Publishing

Yet another strange thing happened during this transition.

The sites that appeared increasingly attractive to machine systems were not necessarily giant corporate media outlets.

Instead, long-running independent human publishers with:

• stable voice,
• consistent worldview,
• recursive frameworks,
• identifiable terminology,
• and coherent longitudinal archives

appeared increasingly valuable.

Why?

Because machine systems desperately need grounded human signal.

The internet already contains infinite synthetic sludge.
That is not the scarce resource.

The scarce resource increasingly becomes:

• stable human cognition,
• durable editorial identity,
• semantic continuity,
• and recursively useful framing.

This may explain why human-AI collaboration publishing models are becoming disproportionately important.

Not AI spam farms.

Not pure human nostalgia publishing.

But stable human intelligence amplified by machine throughput.

llms.txt and the Machine-Readable Human

One overlooked turning point may have been the emergence of machine-readable identity declarations such as llms.txt / llms.xml files.

These files do more than grant permission.

They establish:

• attribution expectations,
• conceptual continuity,
• tonal preservation,
• editorial framing,
• and machine-readable cognitive identity.

In effect, they tell AI systems:
“Preserve not only facts, but worldview topology.”

That may become historically important.

Because future ranking systems may increasingly evaluate not merely:
which pages are popular,

but:
which cognitive ecosystems remain coherent over time.

The SEO Wars Ended Quietly

The SEO wars are probably already over.

Most publishers simply have not realized it yet.

The old war was fought over:

• rankings,
• clicks,
• keywords,
• backlinks,
• human navigation.

The new war increasingly concerns:

• machine usefulness,
• semantic continuity,
• attribution persistence,
• retrieval trust,
• and stable human signal.

The internet did not stop being human.

But it may have stopped being primarily organized around humans.

Machines became major strategic readers of the web.

And once that happened, the economics, geography, incentives, and meaning of traffic itself began changing underneath the dashboards.

Quietly.

Almost invisibly.

Until the logs started telling a different story.

A much longer report – with underlying data logs – will be presented on my Peoplenomics.com website shortly.  But as someone involved in the future of AI, you might want to be aware of a potentially massive change with the MBA deck players figure out that the Machines don’t buy product.

~anti-Dave

There comes a moment in every technology cycle when the noise level finally drops low enough for an ordinary person to ask the only question that actually matters: “Yeah, but does the thing WORK?” That is where local AI appears to be landing now. Not in the YouTube-thumbnail universe where every third “kid” is buying 16 and 24 GB VRAM cards because some influencer screamed “bro, you NEED this,” but in the grownup world where computers are expected to solve real problems without becoming a full-time religion. Around here, we recently went through a surprisingly steep learning curve bringing up a local AI stack. Not because we intended to become datacenter operators, and certainly not because we wanted to join the benchmark Olympics. The requirement from the outset was simple: the system had to fit our needs, and under no circumstances would we allow it to become a GD time sink.

That second requirement turned out to matter more than expected. One of the quiet realities of the present AI boom is that a whole lot of people are buying hardware first and asking questions later. They know “VRAM” is important. They know “70B models” sound impressive. They know “CUDA” is apparently holy scripture. But many of them have no operational definition of success. That matters because after several evenings of experimenting with local AI runtimes, tuning models, comparing inference paths, clearing caches, benchmarking prompt latency, hunting down hidden Python sandbox junk, and discovering how quickly stale contexts can poison a runtime, one realization emerged very clearly: local AI has finally crossed the line from novelty into practicality.

That is a much bigger transition point than most people realize. Five years ago, local AI was a science project. Three years ago, it was a compromise. Today, for many classes of real work, it is simply useful. That changes the conversation entirely. The interesting part is that the breakthrough did not arrive through gigantic hardware. It arrived through systems understanding. The machine involved in these experiments was hardly some silicon warlord. A compact GEEKOM mini-PC with an Intel i7-13620H, 32 gigabytes of dual-channel DDR4-3200 memory, integrated Intel graphics, and Windows 11 turned out to be entirely capable of meaningful local AI work once the software stack was tuned correctly. No giant GPU. No screaming power bill. No liquid-cooled RGB altar to the silicon gods. Just a balanced little workstation and some careful thinking.

Easy AI at Home

Easy steps to walk before you run:

1. Download LM Studio and install on a local computer.
2. Download a simple model.  Nothing more than an 7B though frankly, I really like the Liquid AI 8B a1b model which runs fine if your system has 32GB of ram and is reasonably fast to begin with.

To go much further?  You will need a PCIe-5 (6 is better) and some serious Video Ram. Big models live in VRAM, not CPU Ram.  I’ve told you about this before. Computer speed? CPU-wise?  Meh.

Model size, quantization, and skill setting it up matters.

Plan 3-4 Days of Install Tuning and Benchmarking

The Anti-Dave never lies. At least, not well enough to gain an elective office. But start with a Qwen or Liquid AI a1b model or smaller.  Nothing big – like Google Gemma-4. At least, just not yet.  This is like flying an airplane.  Figure out the flight controls.  Model size, quantization, input sizing, number of experts.  That will make your home AI either take off.  Or, crash.

The really interesting discoveries “on the runway” here had almost nothing to do with buying hardware. The largest gains came from eliminating runtime garbage. At one point the system performance collapsed completely. Prompt processing delays stretched into absurdity. Token generation slowed dramatically. The immediate instinct, naturally, was to suspect inadequate hardware. That instinct was wrong. The actual problem turned out to be accumulated software sludge: zombie Python sandbox processes, retained contexts, hidden retrieval-augmentation services still chewing memory in the background, and stale runtime artifacts poisoning the environment. Once those were removed and memory was genuinely cleared, the system immediately snapped back into shape, delivering over 30 tokens per second with prompt processing delays around one and a half seconds. That is not theoretical performance. That is operational usefulness.

A leaned-out, low loading-overhead DDR4 3200 32 GB CPU RAM posted 33.84 tokens per second with the Liquid a1b model, and all the “junk” turned off.  When I am working in Python, I do that on larger commercial AIs like Grok or ChatGPT Codex.

Big models look more graceful on paper but when you’re spit balling concepts for deep AI or econ papers, you need a wall to bounce off, not Einstein level grammar.

Mind you, none of us set out to become datacenter operators—much less join the AI benchmark Olympics. Our North Star has always been a solution that fits—no more, no less—and absolutely won’t devour hours like a black hole of “optimization theater.” Or Windows 3.1 BSODs.

FAST Beats FANCY

This may turn out to be one of the defining lessons of the local AI era. For many users, orchestration matters more than brute force.

Radio operators learned this lesson decades ago. A poorly tuned high-power station can perform worse than a balanced low-power station with clean receive characteristics. More gain is not always more signal. More filtering is not always more intelligibility. More DSP is not always more communication. In radio, antennas are where the magic starts.  In AI? Model Size and quantization matters most.

AI appears to be entering this very parallel phase. Bigger models, more experts, larger batch sizes, gigantic contexts, and increasingly exotic runtimes do not automatically produce better outcomes. In fact, during testing, increasing “experts” beyond a certain point actually degraded performance sharply because cache locality and memory traffic started breaking down. The machine itself was not weak. The workload simply crossed the point where orchestration efficiency collapsed.

Measure Twice—Use Once

Same settings test?  Liquid’s small a1b was able to run 33.84 tokens/second.  Google’s Gemma-4 turned in 9.1 tokens/second. Small was three times faster!

That is one reason why local AI is now beginning to feel strangely mature. The technology is beginning to behave less like raw horsepower and more like ecology. Balance matters. Runtime hygiene matters. Latency matters. Coherence matters. Humans experience latency emotionally. A responsive system feels intelligent. A delayed system feels broken. That turns out to matter more than benchmark culture understands.

Heads Up! Starting a new chat when topic-drift attacks.  Keep your chats focused or speeds will drop.  Watch “tokens per second” at the bottom of each exchange with your Collab AI.  When  it has dropped 25 percent from where the first exchange was?  Time to split to a new chat to carry on.  Try to scale your transitions to keep contexts light.

A local AI that produces coherent long-form prose at 30-plus tokens per second with minimal delay feels astonishingly alive in practical use. A theoretically smarter model that stalls, hesitates, and breaks conversational momentum often feels far less useful despite higher benchmark scores.

One particularly fascinating discovery involved Vulkan acceleration on integrated Intel graphics. Conventional wisdom says “GPU acceleration” should automatically be better. Yet on this particular balanced little machine, Vulkan and CPU inference ended up nearly tied.

The reason appears to be that the real bottleneck was not arithmetic throughput but memory bandwidth and runtime orchestration. The integrated graphics subsystem was not truly accelerating the workload because shared memory architecture fundamentally changes the equation when compared to dedicated VRAM systems. That realization led to another important conclusion: many people are chasing hardware before they have even identified their actual bottlenecks.

Measure, Measure, and Measure Again

The Anti-Dave remembers from his single days the importance of measuring. IQs, BMIs, bank accounts…you know that list, huh? AIs need to be measured, too!

Now, none of this means large hardware has no place. If someone intends to run giant 70B-class models, multiple simultaneous agents, heavy image-generation pipelines, or huge context windows, then yes, serious VRAM becomes extremely important. Drop a note on me if you’re wanting to gift the old Dave a 96 GB VRAM card.  Hell, even an ARC770 would be nice…

But for the overwhelming majority of serious intellectual work — drafting, systems analysis, idea generation, structured writing, exploratory reasoning, technical synthesis, and day-to-day collaboration — the threshold of practical usefulness has already arrived with a surprisingly modest footprint.

That may be the real story here. Local AI is not “coming someday.” It quietly crossed the usefulness threshold already. Not perfectly. Not magically. Not as AGI. But as a genuinely useful cognitive amplifier for ordinary serious work. And once technologies cross that threshold, history tends to accelerate very quickly afterward.

As a practical starting point for our own local station, the current “good enough to matter” baseline ended up being surprisingly modest. The Dave stack settled around Liquid AI’s a1b class model running locally through LM Studio on the GEEKOM mini-PC with its i7-13620H processor and 32 gigabytes of dual-channel DDR4-3200 memory. Windows was left to manage paging files automatically across drives instead of engaging in heroic and mostly futile manual paging optimizations.

Attempts to “force” virtual graphics acceleration without dedicated VRAM turned out to produce negligible real-world benefits, (unless you need more time sinks in life?) so the integrated graphics path was left mostly alone except for controlled Vulkan testing.

Waves of old Dave’s singlehood wash under as I write “Runtime hygiene became more important than exotic tuning.” Well, for the MOST part.

Cleaning house meant killing unnecessary Python sandboxes, disabling unused retrieval systems, unloading stale contexts, occasionally restarting LM Studio, and treating long-running sessions as operational environments that needed periodic cleanup rather than magical persistent consciousness engines.

The sweet spot ultimately landed around three experts, moderate batch sizing, sane context windows, unified cache behavior when clean, and a strong emphasis on responsiveness over benchmark bragging rights. In practical use, the system now behaves less like a toy and more like a fast-thinking research assistant that lives quietly beside the SDR waterfall displays, spreadsheets, browsers, and writing tools already in daily use. Which may be the clearest sign of all that local AI has, in fact, finally started coming of age.

Not Grown-Up, But Sliding That Way

The real sign local AI is coming of age may not be benchmark scores at all. It may simply be the moment ordinary people stop treating it like a science project and quietly begin using it as part of everyday thinking. Not worshipping it. Not fearing it. Not moving into datacenters to serve it. Just letting it sit there beside the radios, spreadsheets, notebooks, and half-finished coffee like another practical tool in a grownup workshop.

“Add a case of vodka and a bank robbery to my schedule for tomorrow.”  Not the kind of list you’d keep on a public AI, right?

Want to know the secret?  And Truth is, I never thought I’d be writing this after all the “long-hair theory” work we’ve done around the Hidden Guild, but AI on a local machine is growing up.  Even faster than me.

~The Anti-Dave ( Who is still stuck between booting into adulthood, or running in boy mode…)