What if you could write the same schema in YAML?

As of mondrian-saiku 4.8.1.11 (PR #34), you can.

What shipped

Three things landed together:

YAML as a first-class catalog format.

RolapSchemaLoader now auto-detects YAML content by inspecting the catalog string — if it begins with schema: (after any whitespace, comments, or --- document markers) it is routed through the converter before the rest of the pipeline sees it. No configuration flag, no file-extension check, no extra properties. Point your Catalog URL at a .yaml file and Mondrian loads it.

A bidirectional converter.

The new M4YamlToXml / M4XmlToYaml pair converts in both directions with symmetric fidelity. The canonical representation stays XML (that is what XOM, the Mondrian XML object model, serializes); YAML is translated to a typed MondrianDef object graph first, so the converter is not fragile string-munging — it is the same structured pipeline Mondrian uses to load schemas from disk.

A mondrian-schema CLI.

A single shell script wraps the converter and a linter:

# Convert an existing Mondrian-4 XML schema to YAML
./scripts/mondrian-schema to-yaml demo/FoodMart.mondrian.xml -o demo/FoodMart.yaml
# Convert back (lossless round-trip)
./scripts/mondrian-schema to-xml demo/FoodMart.yaml -o FoodMart-roundtrip.xml
# Lint a schema before deploying (validates structure; exits 2 on failure)
./scripts/mondrian-schema lint demo/FoodMart

First invocation compiles and caches the classpath (target/dependency/classpath.txt). Subsequent runs are fast — around 130 ms.

Mondrian-3 support is unchanged. The M3 YAML format (shared dimensions as <DimensionUsage>, <Join>, <Level>, etc.) continues to work. M3 and M4 YAML are mutually exclusive and detected by the presence of the `physical_schema:` key.

---

Show, don’t tell

Here is the Store shared dimension from the FoodMart schema — first as Mondrian-4 XML, then as its YAML equivalent.

XML (51 lines)

<Dimension name=’Store’ table=’store’ key=’Store Id’>

    <Attributes>

        <Attribute name=’Store Country’ hasHierarchy=’false’>

            <Key>

                <Column name=’store_country’/>

            </Key>

        </Attribute>

        <Attribute name=’Store State’ keyColumn=’store_state’ hasHierarchy=’false’/>

        <Attribute name=’Store City’ hasHierarchy=’false’>

            <Key>

                <Column name=’store_state’/>

                <Column name=’store_city’/>

            </Key>

            <Name>

                <Column name=’store_city’/>

            </Name>

        </Attribute>

        <Attribute name=’Store Id’ keyColumn=’store_id’ hasHierarchy=’false’/>

        <Attribute name=’Store Name’ keyColumn=’store_name’ hasHierarchy=’false’>

            <Property attribute=’Store Type’/>

            <Property attribute=’Store Manager’/>

            <Property attribute=’Store Sqft’/>

            <Property attribute=’Grocery Sqft’/>

            <Property attribute=’Frozen Sqft’/>

            <Property attribute=’Meat Sqft’/>

            <Property attribute=’Has coffee bar’/>

            <Property attribute=’Street address’/>

        </Attribute>

        <Attribute name=’Store Type’ keyColumn=’store_type’

                   hierarchyAllMemberName=’All Store Types’/>

        <Attribute name=’Store Manager’ keyColumn=’store_manager’ hasHierarchy=’false’/>

        <Attribute name=’Store Sqft’ keyColumn=’store_sqft’ hasHierarchy=’false’/>

        <Attribute name=’Grocery Sqft’ keyColumn=’grocery_sqft’ hasHierarchy=’false’/>

        <Attribute name=’Frozen Sqft’ keyColumn=’frozen_sqft’ hasHierarchy=’false’/>

        <Attribute name=’Meat Sqft’ keyColumn=’meat_sqft’ hasHierarchy=’false’/>

        <Attribute name=’Has coffee bar’ keyColumn=’coffee_bar’ hasHierarchy=’false’/>

        <Attribute name=’Street address’ keyColumn=’store_street_address’ hasHierarchy=’false’/>

    </Attributes>

    <Hierarchies>

        <Hierarchy name=’Stores’ allMemberName=’All Stores’>

            <Level attribute=’Store Country’/>

            <Level attribute=’Store State’/>

            <Level attribute=’Store City’/>

            <Level attribute=’Store Name’/>

        </Hierarchy>

        <Hierarchy name=’Store Size in SQFT’>

            <Level attribute=’Store Sqft’/>

        </Hierarchy>

    </Hierarchies>

</Dimension>

YAML (38 lines, from demo/FoodMart.yaml)

shared_dimensions:

  Store:

    table: “store”

    key: “Store Id”

    attributes:

    - name: “Store Country”

      key: [”store_country”]

      has_hierarchy: false

    - name: “Store State”

      key_column: “store_state”

      has_hierarchy: false

    - name: “Store City”

      key: [”store_state”, “store_city”]

      name_column: “store_city”

      has_hierarchy: false

    - name: “Store Id”

      key_column: “store_id”

      has_hierarchy: false

    - name: “Store Name”

      key_column: “store_name”

      has_hierarchy: false

      properties:

      - “Store Type”

      - “Store Manager”

      - “Store Sqft”

      - “Grocery Sqft”

      - “Frozen Sqft”

      - “Meat Sqft”

      - “Has coffee bar”

      - “Street address”

    - name: “Store Type”

      key_column: “store_type”

      hierarchy_all_member_name: “All Store Types”

    hierarchies:

    - name: “Stores”

      all_member_name: “All Stores”

      levels: [”Store Country”, “Store State”, “Store City”, “Store Name”]

    - name: “Store Size in SQFT”

      levels: [”Store Sqft”]

The information content is identical. What disappeared: 79 closing tags, 34 attribute-quotation pairs, the <Key><Column .../></Key> ceremony for multi-column keys (replaced by a plain list), and the <Level attribute=’...’/> dance for hierarchy levels (replaced by a bare list of strings). What remained: every structural relationship, every column reference, every property association.

How this makes your life easier

Migrate in one command. If you have an existing Mondrian-4 XML schema, converting it is a single CLI call:

./scripts/mondrian-schema to-yaml your-schema.xml -o your-schema.yaml

The output is immediately loadable — Mondrian detects the YAML format on next startup with no other changes.

Diff and code-review schemas like real code. XML diffs are dominated by closing-tag churn and indentation noise. YAML diffs show only the structural change. A hierarchy level addition is one line; adding a property to an attribute is one line. Pull-request reviews for schema changes become tractable.

Lint before you deploy. The lint subcommand validates the schema’s structure and reports errors with a non-zero exit code, which makes it trivially hookable into CI:

./scripts/mondrian-schema lint your-schema.yaml

# exit 0 → clean

# exit 2 → diagnostic printed to stderr, CI fails

**Split large schemas across files.** For schemas that have grown unwieldy, YAML’s $ref include mechanism lets you break one big file into logical pieces:

# top-level schema file

schema: “MyWarehouse”

shared_dimensions:

  $ref: “dimensions/store.yaml”

cubes:

  $ref: “cubes/sales.yaml”

$ref` resolution works when the schema is loaded via a file:// Catalog URL; it is skipped when content is passed inline.

Keep schemas in version control readably. Because YAML is plain text with clean diff behaviour, you can commit your schema next to your dbt models or Liquibase migrations and treat schema changes as first-class tracked changes rather than opaque XML blobs.

---

How it works

The architecture is straightforward and avoids string-munging entirely.

When RolapSchemaLoader reads a catalog and detects YAML content, it calls YamlSchemaConverter.toXml(). That method checks for a metamodel_version key to distinguish M4 from M3 YAML, then hands off to M4YamlToXml.

M4YamlToXml walks the parsed YAML map and builds a typed MondrianDef.Schema object graph — the same Java types that RolapSchemaLoader normally constructs from XOM-parsed XML. Once the object graph is built, XOM serializes it to XML, and from that point onward the loading pipeline is identical to reading an XML file directly.

The reverse path `M4XmlToYaml) uses M4CubeIngester to walk the MondrianDef graph and emit a structured YAML document. Neither direction manipulates XML as text; both operate on the typed object model. This means the converter inherits Mondrian’s own structural guarantees — it cannot produce XML that does not correspond to a valid MondrianDef tree.

Try it now

# 1. Convert your Mondrian-4 XML schema to YAML

./scripts/mondrian-schema to-yaml path/to/your-schema.xml -o your-schema.yaml

# 2. Inspect and edit the YAML

# 3. Lint it

./scripts/mondrian-schema lint your-schema.yaml

# 4. Point Mondrian at the YAML file

#    (no other changes required — Mondrian auto-detects YAML by content)

jdbc:mondrian:Jdbc=jdbc:...;Catalog=file:///path/to/your-schema.yaml;...

If you prefer to keep XML as the source of truth, you can use YAML purely for authoring and commit the generated XML, or do the opposite — author in XML and generate YAML for review. Both directions are stable.

Known limitations and what is next

The converter does not yet handle every Mondrian-4 feature:

- `CopyLink` does not preserve the `attribute` field. Round-tripping a `CopyLink` with a non-default attribute loses that field.

- `DimensionGrant` in roles is not supported. Role definitions support `SchemaGrant` and `CubeGrant` but not dimension-level grants.

- Some `CalculatedMember` fields are deferred. `caption`, `description`, `visible`, and `CellFormatter` are not yet captured.

- Some `MeasureGroup` fields are deferred. `approxRowCount` and `ignoreUnrelatedDimensions` are not emitted.

- Identifiers with embedded dots do not round-trip. The `table.column` encoding and `{col:}` token both split on the first `.`, which breaks quoted identifiers that contain a literal dot character.

The remaining gaps are well-understood and tracked. Priority for the next phase is filling in the `CalculatedMember` deferred fields and adding `DimensionGrant` support so that complex role definitions can be authored and reviewed in YAML without loss.

If you run into an edge case or have an M3 migration story to share, open an issue or reach out on the Saiku community channels.

At the time, we had lots of users, but not many people committing code or money to keep the platform afloat. It worked pretty well for lead gen, but it was also a huge time suck, and even though we had users from Amazon, Samsung, etc., converting them to any model that would support development was impossible.

So after years of being almost broke, I gave up and went to work for NASA, much to the chagrin of the people using the platform, which is ironic when you’re being blasted for not supporting the platform when others refuse to commit.

Subscribe now

Fast forward to 2026 and the year of the Agentic AI agent, and I thought, “ Hell, I wonder if this thing even compiles. So I set Claude free on it and asked him to get the old version running. To my surprise, about 30 minutes later, he had a running version on my Mac. So then I thought, well, what else could I do with this? So I asked him to update the main dependencies, etc., and write tests to ensure it continued working. All seemed good.

So then I thought, well, the UI is a bit dated, it’s still cool and works really well, but there are so many different frameworks available these days, how about we just rebuild it? So I left him rebuilding the UI using Svelte, as it’s a framework I’m comfortable with. Once again, a few hours later, and after a few prompts to tweak it, Saiku had a new UI built with the latest in modern web technologies.

This continued for a bit longer, and then I went deep on what I could do with Mondrian, the OLAP engine that powers Saiku. We use version 4, which has a story of its own, but I also want to update it and improve support for newer databases, so I added Apache Calcite to the query-generating layer and blogged about it here.

So, of course, this then brings me to now. This happened a few weeks ago, and now I have a couple of weeks of vacation, which we all know really means “time to finish up those projects we started but never finished”. And so I decided to release it. It’s not particularly well tested yet, but most functions work. I would like to do more to it. I’m not sure yet, but I’m working on something to make it easier to build out the OLAP cube specs. I’ve already started with some AI help and support for Cube building. But I’m sure there is still more to do, some may be AI-based, some just improved support and functionality.

Of course, the question is: in the year 2026, does OLAP still serve a purpose? Do small or big businesses need something like Saiku? I don’t know if the answer is. But I figured, having got this far, I should release it, make it accessible and pay a bit of respect to a piece of software that launched my career, and helped get me to where I am today.

Mondrian is an OLAP engine. Give it an MDX query + a schema definition, it hands you back cell values by generating and executing a bunch of SQL against your warehouse. Its SQL emission layer is hand-coded: a SqlQuery string builder plus ~30 dialect subclasses (PostgreSqlDialect, MySqlDialect, OracleDialect, HsqldbDialect, …) each with their own quirks around NULL ordering, date literals, pagination, quoting rules, identifier case.

Apache Calcite does the same job: given a logical plan, emit dialect-correct SQL. But Calcite’s approach is a relational algebra + cost model + rule-driven optimizer. It comes with 40+ SqlDialect implementations and a planner that can do things like push filters through joins, match materialized views, reorder joins by selectivity.

The bet: replace Mondrian’s hand-rolled SQL builder with Calcite’s planner + unparser. Expected wins:

• Delete 30+ dialect classes and 20 years of accumulated workarounds.

• Cost-based MV selection when multiple pre-aggregated tables could serve a query.

• Easier to add new database support (Calcite already has dialects for everything).

Expected risks:

• Calcite’s cost model needs tuning. A wrong row-count estimate and you’re scanning 86M rows instead of 86k.

• Calcite’s optimizer can produce trees the JDBC unparser can’t handle if rule selection is loose.

• Existing deployments depend on Mondrian’s dialect-specific quirks we don’t know about.

The plan: four worktrees, merged atomically at the end.

1. Foundations — wire Calcite in behind a kill switch. Default path stays legacy; -Dmondrian.backend=calcite routes through the new code. Establish a cell-set-parity harness.

2. Natives — migrate RolapNativeCrossJoin, RolapNativeFilter, RolapNativeTopCount, and DescendantsConstraint to emit Calcite plans. These are the “native” tuple-read paths Mondrian uses to avoid materializing large sets in Java.

3. Aggregates + calcs — register pre-aggregated tables (declared in schema as <MeasureGroup type=’aggregate’>) and let Calcite cost-select them. Push arithmetic calc members down into SQL where possible.

4. Cleanup — delete legacy SQL builder, delete 30 dialect classes.

This post covers worktrees 1-3. Cleanup is the final atomic merge.

The harness: 45 MDX queries, golden cell-sets, byte-level drift detection

Before rewriting anything, I built an equivalence harness. For each of 45 MDX queries:

capture legacy cell-set + sequence of emitted SQL → golden/

run under Calcite backend → assert cell-set matches golden

Three failure classes:

• LEGACY_DRIFT — cell values differ from golden. Hard gate. The whole point.

• SQL_DRIFT — cell values match but emitted SQL differs. Advisory. Expected: Calcite writes ANSI INNER JOIN, legacy writes comma-separated FROM + join conditions in WHERE. Postgres plans both identically.

• PLAN_DRIFT — Calcite’s RelNode differs from a frozen snapshot. Advisory. Review signal for refactors.

The harness stored row-level SHA-256 checksums per JDBC execution. This caught a bug later that no cell-value comparison would have: identical values returned in different row order also fail checksum. We’ll come back to that.

First-cut benchmarks: Calcite was 27% slower. Wait, what?

After worktree 1 landed, I ran the 2×2 matrix:

|                     | HSQLDB (87k rows)     | Postgres (86.8M rows) |
|---------------------|----------------------:|----------------------:|
| Legacy Mondrian SQL | 1.13s / query geomean | 7.91s / query geomean |
| Calcite SQL         | 0.24s / query geomean | 9.44s / query geomean |

Calcite was 27% slower on Postgres. At 1000× scale. With all 44 cell-set assertions passing.

The MvHit queries (small agg-table scans) were the worst — 2.24× slower than legacy. That’s a 2.6k-row agg table taking 5 seconds. Something was eating a second and a half of wall time before any SQL even hit the wire.

Y.2: one cache key

I instrumented every phase of the Calcite path. Cold iteration on an MvHit query:

plan.total            675 ms

  relBuilderCreate    478 ms    ← here

  build                34 ms

  unparse               6 ms

RelBuilder.create doesn’t take 478 ms. What’s happening is that the CalciteMondrianSchema adapter wraps JdbcSchema.create(rootSchema, name, ds, null, null), and Calcite’s JDBC adapter reflects the entire database’s metadata (every table, every column, every type) via DatabaseMetaData on first touch. On HSQLDB that’s in-process and instant. On Postgres, that’s a round-trip to pg_catalog for every table in the schema.

Fine, so you cache it. The cache was there:

// SegmentLoader.java

private static final Map<RolapStar, CalciteSqlPlanner> CACHE = ...;

Keyed on RolapStar. And Mondrian’s per-query schema-cache flush invalidates the RolapStar. Every query — a fresh star, cache miss, full JDBC metadata reflection.

The fix was three lines:

// CalcitePlannerCache.java

public static CalciteSqlPlanner plannerFor(DataSource ds) {

    Key key = Key.from(ds);  // (url, catalog, schema, user) via DatabaseMetaData

    return CACHE.computeIfAbsent(key, k -> build(ds));

}

Key on JDBC connection identity. The cache persists across Mondrian’s schema churn because nothing Mondrian does invalidates the JDBC connection’s identity.

Result:

|                          | HSQLDB| Postgres                      |
|--------------------------|------:|------------------------------:|
| Pre-fix D/B geomean      | —     | 1.27× (Calcite 27% slower)    |
| **Post-fix D/B geomean** | —     | **0.93× (Calcite 7% faster)** |
| MvHit D/B                | 1.07× | **1.01×** (was 2.24×)         |

One cache key moved the needle 36 percentage points. This was the single biggest perf fix in the entire project. Everything downstream is stacked on top of this.

Lesson: when a performance delta looks proportional across query sizes, it’s almost always a fixed per-query cost. Not a plan quality issue. Find the fixed cost first.

The 2×2 matrix: Postgres plans both SQLs identically

After Y.2, I re-ran the full matrix:

|                                                 | HSQLDB (87k)                      | Postgres (86.8M)    |
|-------------------------------------------------|----------------------------------:|--------------------:|
| **C/A geomean (Calcite vs legacy on HSQLDB)**.  | **0.219×** (Calcite 4.57× faster) | —                   |
| **D/B geomean (Calcite vs legacy on Postgres)** | —                                 | **0.958×** (parity) |

HSQLDB: 4.57× speedup is real but not for the obvious reason

Why does Calcite win 4.57× on HSQLDB? Not because Calcite emits “better” SQL in any meaningful sense — Postgres proves that.

HSQLDB’s query planner is primitive. Calcite’s SQL is:

• Explicit ANSI INNER JOINs (HSQLDB handles these better than comma joins).

• Keyword-case consistent (HSQLDB’s parser has fewer decisions to make).

• No ”customer” AS “customer” table aliases (HSQLDB allocates fewer scopes).

• OR-chain IN-lists rewritten to actual IN (v1, v2, v3)

Postgres doesn’t care. Its planner normalizes all of this internally before execution. HSQLDB doesn’t.

If your workload is Mondrian + HSQLDB (more common than you’d think — it’s the default for Mondrian demos, plus there are production deployments), Calcite is a meaningful perf improvement even without any other change.

Postgres: parity, with interesting exceptions

Most of the corpus lands at D/B ≈ 1.0× (within 5% of legacy). The per-query overhead of Calcite’s translation (build PlannerRequest, construct RelBuilder, unparse) is a few milliseconds on top of a 10s Postgres query — noise.

But a handful of queries show 5-12% Calcite wins:

| Query                           | Legacy (s) | Calcite (s) | D/B        |
|---------------------------------|-----------:|------------:|-----------:|
| `named-set`                     | 29.9       | 26.2        | **0.88×**  |
| `topcount`                      | 30.5       | 26.7        | **0.88×**  |
| `native-topcount-product-names` | 32.5       | 29.0        | **0.89×**  |
| `filter`                        | 19.9       | 18.4        | **0.93×**  |
| `native-filter-product-names`   | 19.9       | 18.5        | **0.93×**  |

What these have in common: they’re the heaviest queries. Calcite pushes more predicates into join conditions, giving Postgres more scope to optimize. At 30s per query, 12% is 3-4 seconds saved per execution.

Interesting outlier: basic-select on Postgres is 0.26× (4× faster) under Calcite. This was a bit of a surprise. EXPLAIN ANALYZE shows Postgres picks a slightly different plan — the Calcite SQL doesn’t include the unused time_by_day table Mondrian’s legacy path drags in for cardinality probing. Less scanning, less aggregation work.

Where it gets fun: the MV matcher

Mondrian supports aggregate tables. You declare them in schema XML:

<MeasureGroup table=’agg_c_14_sales_fact_1997’ type=’aggregate’>

  <Measures>

    <MeasureRef name=’Unit Sales’ aggColumn=’unit_sales’/>

    <!-- ... -->

  </Measures>

  <DimensionLinks>

    <ForeignKeyLink dimension=’Store’ foreignKeyColumn=’store_id’/>

    <CopyLink dimension=’Time’ attribute=’Month’>

      <Column aggColumn=’the_year’ table=’time_by_day’ name=’the_year’/>

    </CopyLink>

    <!-- ... -->

  </DimensionLinks>

</MeasureGroup>

Mondrian’s own logic picks one of these when the grain matches a query. But only if the operator sets mondrian.rolap.UseAggregates=true and mondrian.rolap.ReadAggregates=true. Global flags. Off by default. Most deployments I’ve seen have them off because they weren’t documented prominently, and leaving them off is safe.

So: queries against your big fact table scan the big fact table, even though you declared a handy 2.6k-row pre-aggregate that perfectly answers the query.

Calcite’s answer: a MaterializedViewRule in its planner that does cost-based MV selection. Register the declared aggregates as `RelOptMaterialization` entries, and the rule rewrites queries that subsume a materialization to scan the materialization instead. No operator flag needed.

I built the infrastructure: MvRegistry walks the schema’s aggregate MeasureGroups, builds a RelOptMaterialization per declared agg. Registered with VolcanoPlanner. PK/FK metadata surfaced so Calcite’s Goldsein-Larson duplication-preservation check could fire.

And the rule didn’t fire. At all. Ever.

The wall I hit

Calcite’s MV rule family uses SubstitutionVisitor internally. It compares the user query’s RelNode tree against the MV’s defining-query `RelNode` tree and returns substitutions when one subsumes the other. Structural match is strict:

• User query’s RelNode shape: Aggregate(Scan(fact) Join Scan(dim)) — 1 join.

• MV’s defining query: Aggregate(Scan(fact) Join Scan(dim1) Join Scan(dim2) Join Scan(dim3) Join Scan(dim4)) — 5 joins.

SubstitutionVisitor has no way to prove the MV’s 4 extra joins are “safe to drop” without FD/uniqueness metadata. Even with PK/FK metadata surfaced, rule coverage was incomplete for the denormalized-CopyLink shapes FoodMart’s aggregates use.

After 6 hours of iteration, the rule was rewriting zero queries.

Option D: stop fighting, write the matcher

Calcite’s MV framework is general-purpose. Our use case is narrow: given a Mondrian <MeasureGroup type=’aggregate’> declaration, match user queries whose shape matches the declared agg.

I wrote a 280-line MvMatcher that walks each registered ShapeSpec and matches user PlannerRequests directly:

if MV’s group-by columns ⊇ user’s group-by columns

   AND user’s measure columns are pre-aggregated on the MV

   AND user’s filters reference columns the MV carries

   → rewrite factTable to aggTable, drop dropped joins, translate columns

Deterministic. Bypasses SubstitutionVisitor entirely. No trait/convention wrestling.

Unit tests proved all 4 MvHit corpus queries rewrite correctly to agg-table SQL.

Then I ran the Postgres perf benchmark. Zero effect. The matcher wasn’t firing at runtime.

The second wall

The planner cache (from Y.2) was attached, but with no MvRegistry because the first caller to plannerFor(DataSource) didn’t pass the schema. Cache stored a registry-less planner. Subsequent calls with schemas hit the cache and returned the registry-less planner.

Fix: extended plannerFor(DataSource, RolapSchema) and made both SegmentLoader and SqlTupleReader dispatch seams pass the schema. Late-bind the registry if a schema arrives on a cache-hit.

Re-ran the benchmark. Still zero effect.

The third wall

Added a trace to CalcitePlannerCache: the registry built had size() == 0. Why?

MvRegistry.fromSchema walks rolapSchema.getCubeList(). On the first call, before Mondrian finishes cube initialization, that list is empty. My late-bind code checked “is registry null?” — and a size-0 registry isn’t null.

Fix: retry the registry build when size == 0 AND the caller has a schema. Once size > 0, stop retrying.

Re-ran the benchmark. Now 2 of 4 MvHit queries rewrite. 10× speedup on those. But the other two didn’t match.

The fourth wall

The other two MvHit queries use agg_g_ms_pcat_sales_fact_1997, which has gender and marital_status CopyLinked from customer (not reached via FK join). My shape catalog’s family-gender shape declared 2 group-by columns: product_family, gender. But runtime MDX always slices by [Time].[Year] under Mondrian’s hasAll=’false’ setting, so the runtime groupBy is actually [the_year, product_family, gender] — three columns.

Fix: added year-family-gender and year-family-gender-marital shape variants.

Re-ran. All 4 match. Plus crossjoin and calc-iif-numeric started drifting.

The fifth wall (self-inflicted)

Adding shapes is easy. Over-matching is easy. My new year-family-gender-marital shape (4 group columns) also matched crossjoin’s user query [the_year, gender, marital_status] (3 columns) because the matcher accepted “user is a subset” — semantically valid SUM-over-SUM rollup from a finer-grained MV.

That rewrite is mathematically correct. But the agg’s pre-aggregated rows are at (year, family, gender, marital_status, quarter, month) grain. Summing SELECT year, gender, marital_status, SUM(unit_sales) FROM agg GROUP BY year, gender, marital_status gives the right totals — and a completely different row iteration order than the base-fact scan. Different row order, different checksum, LEGACY_DRIFT.

Fix: require EXACT group-by-set-size equality between user request and shape. If you want a coarser-grained match, declare a coarser-grained shape. Explicit beats inferred.

Re-ran. 44/44 harness green, 4/4 MvHit queries rewrite, 10× speedup.

Final MvHit numbers (Postgres, UseAggregates=false)

| Query                                  | Legacy (fact scan, 86M rows)   | Calcite (matcher agg scan, ~2.6k rows) | Speedup.  |
|----------------------------------------|-------------------------------:|---------------------------------------:|----------:|
| `agg-g-ms-pcat-family-gender`          | 12.5s                          | 1.3s                                   | **10.0×** |
| `agg-c-year-country`                   | 10.0s                          | 1.2s                                   | **8.6×**  |
| `agg-c-quarter-country`                | 11.7s                          | 1.2s                                   | **9.7×**  |
| `agg-g-ms-pcat-family-gender-marital`  | 13.3s                          | 1.2s                                   | **10.9×** |
| **Geomean**                            |                                |                                        | **9.7×**  |

No operator flag needed. Calcite sees the declared aggregates, matches the query, rewrites. User just asks “show me sales by year and country” and gets the answer in 1.2 seconds instead of 12.

The benchmarking detour

At one point my benchmark showed agg-g-ms-pcat-family-gender-marital was 2.22× slower under Calcite than legacy. Other queries fine. Wrote a diagnostic, ran EXPLAIN ANALYZE, compared plans, stared at query plans for two hours.

The reason: I was launching both Mondrian JVMs in parallel against the same Postgres:

mvn ... -Dmondrian.backend=legacy &

mvn ... -Dmondrian.backend=calcite &

Both JVMs hit the same 8-core laptop and the same Postgres instance. Whichever cell’s heaviest query happened to overlap with the other cell’s heaviest query got disproportionately slow. Non-deterministic. Looked like a regression on one specific query.

Sequential runs showed D/B = 1.02× for that query. No regression. Just contention.

Lesson: never run head-to-head database benchmarks in parallel against the same database instance. It’s obvious in retrospect. I lost 4 hours to it.

What I’d do differently

• Instrument first, optimize second. The Y.2 fix was obvious once I had phase-by-phase timings. I spent days before that trying to make Calcite emit “better” SQL when the real cost was metadata reflection.

• Don’t fight general-purpose frameworks. Calcite’s MaterializedViewRule is powerful and brittle. For a constrained use case (matching Mondrian’s declared aggregates), a 280-line hand-rolled matcher beat 6 hours of trying to make the general rule fire.

• Checksum row order as a proxy for correctness is sharp. Identical aggregated values in different row order trip LEGACY_DRIFT. Fine for catching real regressions. Also fine for catching your own harmless changes. Know which you’re seeing.

• Sequential benchmarks. Always.

By the numbers

|                                                        | Measurement           |
|--------------------------------------------------------|:---------------------:|
| Production commits on the branch                       | 66                    |
| Lines of new Java (main/)                              | ~3 500                |
| Lines of test Java (test/)                             | ~2 800                |
| Harness MDX queries                                    | 45                    |
| Legacy dialect classes deleted (yet)                   | 0                     |
| Legacy dialect classes queued for deletion             | 30+                   |
| **Calcite speedup vs legacy (HSQLDB geomean)**         | **4.57×**             |
| **Calcite speedup vs legacy (Postgres geomean)**       | **1.04×**             |
| **MV matcher speedup (Postgres, UseAggregates=false)** | **9.74×**             |
| Y.2 fix — lines changed                                | 3                     |
| Y.2 fix — perf impact                                  | 36 pp (1.27× → 0.93×) |

The whole rewrite took roughly 5 days of focused work (3 worktrees + all the investigation). Worktree 4 (deletion) is probably another day. Then the blog has a cleaner ending.

What this unlocks

The real value of this isn’t 4.57× on HSQLDB. HSQLDB isn’t exactly a production warehouse. The real value is:

1. A generic dialect path. Want to support ClickHouse? DuckDB? Snowflake? Calcite has dialects. Add one line to CalciteDialectMap. Done. No more 300-line MySqlDialect subclasses.

2. Automatic aggregate-table rewriting. No configuration flag required. Declare your aggregates in schema XML, and queries hit them.

3. A path to real cost-based optimization. The VolcanoPlanner infrastructure is in place. Someone who wants cost-based MV selection, join reordering, or custom rules has somewhere to put them.

4. Less surface area for bugs. 30 hand-tuned dialect subclasses are 30 places for NULL-ordering quirks to hide. Calcite has one tested SqlDialect implementation per database.

The Postgres parity is the interesting one to sit with. Calcite isn’t making Postgres queries slower. It isn’t making them meaningfully faster on plan quality either. But it IS making the code a lot smaller, cleaner, and easier to extend. Those are wins you can’t benchmark.

Everything got the multimedia label. If it had speakers and a CD-ROM drive, it was a Multimedia PC. The badge told you almost nothing about what the machine could actually do, but it sold units.

Fast forward 30 years (and yes, I feel old too) and we’re living through exactly the same thing with AI. If it plugs into a wall, it’ll have some AI strapline somewhere, it seems. Your PC, your phone, your watch, your fridge, your microwave, your oven. Everything is “AI-ready” or “AI-powered.” The label is everywhere, and it tells you almost nothing.

The Problem With a Three-Letter Word

The reason “AI” has become such a useless label isn’t that people are lying. It’s that the term covers such an absurdly wide range of technology that it’s effectively meaningless without qualification. It’s like saying a vehicle has an engine. Okay, is it a lawnmower or a 747?

The mainstream conversation right now is dominated by Large Language Models. The chatbots, the text generators, the things most people picture when they hear “AI” in 2025. But the tech world seems to have collectively forgotten that AI has been around for a lot longer than the past few years. The foundations were laid by Alan Turing and others in the 1940s and 50s. The first artificial neural network, SNARC, appeared in 1951. The first program that could learn from its own mistakes, a checkers-playing system, arrived in 1952.

My own career has taken me across the gamut of computer science and ML projects, and while I’m no academic researcher, I’ve worked alongside some very clever people building very different types of models to solve very different problems. Statistical models for fraud detection. Computer vision systems for scientific instruments. Recommendation engines. Sensor monitoring. All of it “ and none of it an LLM.

So when everything gets called “AI-ready,” the question that should follow is: what kind?

Three Questions That Actually Matter

Instead of accepting the AI label at face value, I’d suggest asking three questions. They won’t make you an expert, but they’ll cut through the marketing fog faster than anything else.

1. Is it learning, or is it following rules?

This is the most basic distinction, and it’s the one most often obscured by marketing. A genuine machine learning model improves its performance based on data. It identifies patterns, adjusts, and gets better over time. A rules engine executes a predefined set of if/else logic that a human wrote.

Both are perfectly valid tools. Sometimes a well-crafted rules engine is exactly what you need. It’s fast, predictable, and easy to audit. There is absolutely nothing wrong with that. But calling it “AI-powered” sets an expectation that the system is doing something it isn’t. If your vendor can’t clearly explain whether their product is learning from data or following a script, that’s a red flag, not because rules engines are bad, but because the distinction matters for how you evaluate, trust, and maintain the thing.

2. Is the output deterministic or probabilistic?

This is the question that I think matters most, and it’s the one that gets lost almost entirely in the current conversation.

A deterministic model, given the same input, will give you the same output every time. You can test it, validate it, explain it, and predict its behaviour. A probabilistic model gives you its best guess, a plausible output that might differ next time you ask the exact same question. If you’ve ever asked ChatGPT the same thing twice and got different answers, you’ve experienced this firsthand.

Neither approach is inherently better. They serve different purposes. But the implications for trust, governance, and risk are completely different.

Consider financial transaction monitoring. If you’re flagging potentially suspicious activity, do you want a model that gives you the same risk score every time for the same transaction? Or one that might score it differently on a different day? Both approaches exist, both have their place. But you’d better know which one you’re deploying and why, because the regulatory, audit, and compliance implications are worlds apart.

The same applies in healthcare, manufacturing, legal tech, anywhere the stakes are real. A deterministic model that classifies a tumour the same way every time is a fundamentally different tool from a probabilistic one that gives you its best interpretation. Both might be called AI. The governance they require couldn’t be more different.

This is also where a lot of the “boring” AI lives. The workhorses that don’t make headlines but run critical infrastructure. Fraud scoring on your credit card. Pricing algorithms when you book a flight. Predictive maintenance on industrial equipment. These models are often deterministic, explainable, and battle-tested over years of data. Nobody writes breathless articles about them, but they’re doing higher-stakes work than most LLM deployments. Nobody dies if a chatbot writes a bad email. A misclassified fraud signal or a missed anomaly in a sensor reading is a different story entirely.

3. Is it making decisions, or informing them?

The final question is about autonomy, and it changes everything about risk, liability, and trust.

A model that flags a suspicious transaction for a human analyst to review is a fundamentally different proposition from one that automatically blocks the transaction. A system that highlights a potential defect on a production line for an inspector to check is not the same as one that rejects the part without human involvement. A tool that drafts an email for you to edit is not the same as one that sends it.

The governance, liability, and customer experience implications shift dramatically based on where the human sits in the loop. An AI system that surfaces information and lets a person decide carries one kind of risk. An autonomous system that acts carries another entirely. Both might be labelled “AI-powered.” The label doesn’t tell you which one you’re buying.

The Label Will Eventually Become Meaningless, and That’s Fine

The multimedia wars resolved themselves. We stopped caring about the MPC badge and started caring about what the machine could actually do. The same will happen with AI. Eventually, the marketing label will fade, and people will just talk about what the technology does, how it works, and whether it’s the right fit for the problem.

But right now, we’re in the messy middle. The label is everywhere and means almost nothing. Every product is “AI-ready.” Every vendor has an AI story. The companies and leaders that will get the most value from this era aren’t the ones buying everything with an AI sticker on it. They’re the ones who understand what kind of AI solves their specific problem and can tell the difference between a lawnmower and a 747.

So the next time someone pitches you an AI-powered product, skip the demos and the buzzwords. Just ask three questions: Is it learning or following rules? Is the output deterministic or probabilistic? Is it making decisions or informing them?

If they can answer clearly, you’re probably talking to someone who knows what they’ve built. If they can’t, well, that tells you something too.

Where do you want to go today? Turns out, the real question was always: what’s actually under the hood?

The scientists using Pixlise, the software we built to analyse X-ray spectrometry data from Mars, went from taking roughly a year to publish papers to doing it in weeks. That’s not because we wrote better code. It’s because we understood how they actually worked before we wrote any code at all.

Most engineering leaders I talk to don’t dispute that design matters. They just think it can wait. Ship first, polish later. Get something in front of users and iterate. The intention is good. Stay lean, move fast, avoid over-engineering. But there’s a hidden assumption in that approach: that design is about polish. That it’s the paint you apply after the structure is built.

It isn’t. And that misunderstanding costs more than most teams realise.

The False Economy of “We’ll Figure Out the UX Later”

The pressure to ship is real. I run a consultancy and I see it constantly. Stakeholders who want a working prototype by next month. Investors who want to see progress. Clients who’ve already announced launch dates. Stopping to do user research feels like a luxury when the sprint board is overflowing.

So what happens? Engineers become designers by default. We sketch out interfaces based on how we think the data should flow. We make assumptions about what users want because asking them would take too long. We ship something that works, in the technical sense, and wait for feedback.

The feedback arrives. It’s not what we expected.

BCG research shows that nearly half of C-suite executives report that more than 30% of their tech projects run over budget and late. The reasons they cite include misalignment between technical and business teams, unrealistic timelines, and insufficient resources. But underneath those symptoms there’s a quieter problem. Teams building software that solves the wrong problem. Or solves the right problem in a way that doesn’t fit how users actually work.

Barry Boehm calculated in the 1980s that fixing a defect in production costs roughly 100 times more than catching it during design. A study at Ricoh put it more starkly: the cost of fixing a design defect was $35 during design and $690,000 in field service. These numbers vary by context but the pattern is consistent. The later you discover a mismatch between what you built and what users need, the more expensive it becomes to fix.

And yet we keep placing the same bet. That we’ll be able to course-correct after launch. Sometimes we’re right. Often we’re not. And the lurking usability problems become emergencies that consume the engineering team’s roadmap for months.

What “Design” Actually Means Here

When I say design should be included early, I’m not talking about colour palettes or button placement. I’m talking about a specific kind of work that engineers often don’t have time or training to do well.

1. User research is the practice of understanding who will actually use this software, what they’re trying to accomplish, and how they currently get that work done. It’s not about asking people what features they want. It’s about observing and interviewing to understand the shape of the problem.

2. Interaction design is figuring out how users will accomplish their goals through the software. What’s the flow? What happens when things go wrong? Where are the decision points and how do we support them?

3. Design systems are shared vocabularies. Components, patterns, and conventions that both designers and engineers can reference. They reduce ambiguity and make it easier to build consistently.

4. Service design zooms out further. It’s about the processes people use, not just the screens they interact with. How does this software fit into their broader workflow?

At JPL, the Human Centered Design group takes this seriously. As one of their senior designers put it, “By designing our processes around the needs of the people who use them, we not only save the taxpayers money by making people more efficient, we maximise the science we can get out of each mission.” That’s the frame. Design as leverage for outcomes, not decoration.

What This Looks Like in Practice: The PIXL Story

PIXL, the Planetary Instrument for X-ray Lithochemistry, is an X-ray spectrometer mounted on the Perseverance rover. It analyses the chemical composition of Martian rocks at a level of detail that helps scientists understand the planet’s geological history and search for signs of ancient microbial life.

The instrument generates enormous amounts of data. Before Pixlise, the scientists analysing that data relied on Excel-based pipelines they’d built over years. Those pipelines worked. Scientists are resourceful. But they didn’t enable collaboration. Each researcher had their own spreadsheets, their own methods, their own version of the truth.

We built Pixlise to change that. It’s an open-source web application that allows scientists and researchers to analyse PIXL data in near real-time, collaboratively, while retaining the flexibility of their existing workflows. They can share insights, build on each other’s work, and move from raw data to published findings faster than ever before.

But here’s the thing. That outcome wasn’t inevitable. It happened because we had a dedicated UX research function embedded in the project from the start.

The researchers didn’t just sketch interfaces. They aggressively interviewed different members of the user base. Scientists, researchers, anyone with an interest in analysing rover data. They watched how people worked. They asked questions that engineers wouldn’t think to ask, because engineers are focused on what’s technically possible, not on what matches how scientists actually think.

What emerged from that research shaped the product in ways that wouldn’t have happened otherwise.

The charts had to make sense to users, not just to engineers. Technical accuracy wasn’t enough. The visualisations needed to match the mental models that planetary scientists already had. The way they think about spectrometry data, the comparisons they instinctively want to make.

The interaction between visual data and analysis tools had to feel natural. Scientists are looking at images from Mars alongside spectral data. The way those two things connect in the interface matters enormously. Get it wrong and you’re fighting the tool instead of using it.

The scripting and function-writing capabilities had to extend existing workflows, not replace them. Scientists weren’t going to abandon their Excel pipelines overnight. We needed to augment what they already did, not demand they start from scratch.

None of this would have been obvious from the requirements document. It came from research.

The result? Scientists felt ownership of the product. They’d been consulted, heard, and involved. They weren’t being handed something built for them. They were using something built with them.

And the measurable outcome speaks for itself. The time from data collection to published paper dropped from roughly a year to weeks. That’s not a marginal improvement. That’s a transformation in how science gets done.

Technical Feasibility Workshops: Where Design Meets Engineering

Early design involvement doesn’t mean designers go off and create something in isolation, then hand it to engineering. That’s just moving the handoff earlier. The real value comes when design and engineering work together from the start.

One practical way to do this is the technical feasibility workshop. A structured session early in a project where both disciplines surface constraints together.

When to run one: Before architecture decisions are locked. During discovery or definition phases, when you’re still figuring out what you’re building.

Who’s in the room: At minimum, an engineering lead and someone responsible for design or user research. Ideally also a product owner and, if at all possible, an actual user or customer.

What you cover:

1. User story walkthrough (designer-led). What is the user trying to accomplish? What does success look like for them? Where are the pain points today?

2. Technical constraint mapping (engineer-led). What are the hard constraints? What’s expensive or risky? What technical debt already exists that will shape what’s possible?

3. Overlap identification (collaborative). Where do user needs and technical constraints intersect? What’s both desirable and feasible? What trade-offs are we already facing?

4. Risk surfacing (collaborative). What happens if we get this wrong? Where are the assumptions we’re most uncertain about?

5. Decision points. What do we need to prototype or test to reduce uncertainty?

This isn’t a one-time event. Spotify describes their approach as shifting weights. Design and engineering don’t hand off to each other. They shift who’s leading at different phases while remaining in the room together throughout. The designer’s job isn’t just to create designs. It’s to facilitate getting to the best design outcome, which requires engineering input from the beginning.

GetYourGuide calls it “handshake instead of handoff.” The difference is subtle but significant. A handoff implies sequential work. Design finishes, then engineering starts. A handshake implies parallel collaboration. We’re working on this together, even when one of us is leading.

Collaborative Prototyping: Keeping the Feedback Loop Tight

The old model for design-engineering collaboration looked like a relay race. Designers did their work, documented it exhaustively, and threw it over the wall. Engineers picked it up, discovered a dozen reasons why it wouldn’t work as specified, and either built something different or sent it back for revision.

The new model keeps both disciplines in the room together.

Engineers in user research sessions. Not to solve problems, that’s not the point, but to watch and listen. When engineers see real users struggling with a specific interaction, it changes how they think about technical trade-offs. “We should refactor this flow” becomes a hard sell. “In eight out of ten sessions, users failed to complete this task without backtracking, here’s the video” is a different conversation.

Designers in technical spike reviews. When engineering is exploring whether something is feasible, designers should be there to understand what’s possible and what’s expensive. The best design solutions often emerge from understanding constraints, not ignoring them.

Shared prototypes that evolve together. Rather than a finished design spec that gets implemented, there’s a prototype that both sides iterate on. Designers adjust based on technical feedback. Engineers adjust based on user feedback. The artifact gets better faster.

Design systems as living contracts. A good design system isn’t just a library of components. It’s a shared agreement between design and engineering about how things work. When both sides contribute to it and reference it, there’s less ambiguity and less rework.

The contrast with projects that lack this collaboration is stark. I’ve worked on projects where there was no formal design process, where we just talked through requirements and started building. Every single time, the result was the same. Larger iterative cycles. Unclear expectations. And eventually a tense conversation where the customer’s vision turned out to be fundamentally different from what we’d built.

These days I refuse to work on projects that don’t have some element of design thinking baked in. Not because I’m precious about process. Because I’ve learned what it costs when you skip it.

“Engineers Can Figure It Out” (and Other Expensive Beliefs)

When I talk to skeptical engineering leaders about involving design earlier, I hear a few recurring objections.

“Engineers can figure it out these days.” There’s some truth here. Engineering skill sets have expanded. Many developers have better design intuition than their predecessors. Tools have improved. But there’s a difference between being able to make reasonable interface decisions and being skilled at understanding user needs. They’re different disciplines. An engineer can figure out a plausible solution. A designer with research can figure out the right one.

“AI can build it, design just slows us down.” This is increasingly common and it gets the causality backwards. AI removes coding friction. It makes implementation faster. But it doesn’t remove the need to know what to build. In fact, faster implementation makes upfront clarity more valuable, not less. If you can spin up a feature in hours instead of weeks, the bottleneck shifts entirely to knowing which feature to build. Good UX research gives developers a clear understanding of what needs to be built and how interactions should work. AI can still be leveraged for implementation. But research removes the computer-based assumptions from the development cycle.

“We don’t have time or budget for design.” This one I take personally because I’ve lived the alternative.

Early in my consultancy work I took on a project without any structured design process. We discussed requirements regularly. Had good rapport with the client. Thought we understood what they wanted. We were wrong. The gap between their expectations for the user interface and data flow and what we built only became clear late in the project. I made a financial loss on that engagement. Not because the code was bad, but because we built confidently in the wrong direction.

That experience taught me two things. First, verbal discussions aren’t enough. You need structured workshops, visual artifacts, and iterative feedback so the customer sees what’s being built and can flag misalignment early. Second, the “cost” of design isn’t additive. It’s preventive. You’re not spending extra. You’re avoiding waste.

The minimum viable design investment isn’t a full UX team and months of research. It’s someone asking “who is this for and how will they use it?” before any code gets written. It’s regular feedback sessions with real artifacts, not just status updates. It’s treating user understanding as an engineering requirement, not a nice-to-have.

Design as an Engineering Practice

The lesson from PIXL wasn’t that NASA has resources most teams don’t. It’s that treating design as foundational, as a structural part of how you build rather than a finishing touch, changes outcomes in ways you can measure.

Scientists publishing papers in weeks instead of a year. Users who feel ownership because they were consulted. Features that match how people actually work, not how engineers assumed they would work. These aren’t soft benefits. They’re the kind of outcomes engineering leaders care about.

If you’re skeptical, I’m not asking you to overhaul your process. I’m asking you to try an experiment. Before your next project kicks off, ask four questions:

1. Have we talked to the people who will use this?

2. Do we understand how they work today, not just what they say they want?

3. Have we surfaced technical constraints alongside user needs, in the same room, at the same time?

4. Is someone responsible for holding the user’s perspective throughout the project, not just at the beginning?

If the answer to any of those is no, you’re placing a bet. Maybe it’ll pay off. But the research, and my experience, suggest the odds aren’t in your favour.

When you’re building for Mars, you don’t get many chances to iterate. When you’re building for anyone else, you might get more chances. But each iteration is more expensive than getting it closer to right the first time. Design isn’t the opposite of moving fast. It’s how you avoid moving fast in the wrong direction.

In many organisations, a core issue is the persistent misalignment between business objectives and IT capabilities. Picture a business racing ahead so quickly that no one has time to tackle bigger-picture problems. Imagine, however, if those efforts were channeled toward creating shared customer value, not just managing internal efficiencies. IT often becomes a bottleneck, overwhelmed by competing requests and tasks, making prioritization nearly impossible. The result? Frustrating finger-pointing all around when that energy could be harnessed to enhance essential work and improve alignment across the organization. By focusing on a cohesive path to meet customer needs, the Trio model can transform frustration into collaborative success.

In several organisations I’ve worked in, requests stay in backlog for months due to a lack of prioritisation bandwidth. Frequent urgent escalations, often self-serving, derail progress and force immediate attention. Business teams bypass IT using shadow solutions, lacking confidence in IT delivery. Meanwhile, IT feels reduced to an order-taking role rather than an autonomous, decision-making organisation.

Not all organisations operate this way, but high-pressure environments can slip into these habits unnoticed. Recovering requires both awareness and courage to rebuild effective, aligned collaboration.

Why the Usual Fixes Fail

The core issue: business and tech are structured as separate entities that hand work back and forth, rather than solving problems together.

So, of course, what you then end up with is a list of things you’ve probably already tried:

1. Hiring more managers

2. Hiring more engineers

This approach doesn’t fix misalignment; it adds more people who lack context and increases salary costs without resolving core issues. Extra alignment meetings often waste engineering hours through ‘status theatre.’ Quantifying this lost productivity can underscore the urgent need for change.

Next comes the problem of ticketing and prioritisation. This helps some symptoms and increases visibility if managed properly. But it does not address the chaos itself.

You then add more alignment meetings with leaders, developers, product managers, and engineers. These meetings become status theatre, with participants seeking to outdo one another, leading to team-wide frustration and problems.

This repeated separation of business and tech creates recurring misalignment and unresolved problems instead of joint solutions.

The Trio Model - What It Is and What It Isn’t

Today I want to talk about the Trio model. For anyone who doesn’t understand what this is, it’s a collaboration pattern about three different parts of the organisation working together. In an engineering light organisation, you may end up having a slightly different trio than the general trio, but let’s take, for example, a light org where you might end up with:

• A business owner (this person would own the problem and the customer relationship)
• A technical lead who owns the feasibility and implementation (probably working as an IT engineer)
• A designer or an analyst or an ops lead (it sort of depends on the problem type)

These people are expected to work together to enable co-ownership of outcomes, not just a hand-off of tasks. The Trio should rally around customer-centric metrics such as customer satisfaction, net promoter score, time-to-market, or defect rate—tangible goals all Trio roles can collectively strive to improve, translating shared outcomes into concrete action.

Of course, the Trio isn’t a committee or approval layer. It’s not there to oversee but to bring the right people together before decisions are made—not after. It’s also not consensus-driven: three people don’t mean three votes. Instead, decision rights are explicitly assigned: the business owner has final say over customer priorities, the technical lead makes the call on feasibility and implementation, while the designer, analyst, or ops lead decides how to optimise the user experience, business process, or operations, depending on the issue. The tech lead cannot be vetoed by the business owner on technical feasibility, nor can the business owner be overruled on customer needs. Clear assignment of decision rights prevents gridlock and ensures each key area is covered. It is important to note that the trio is not the entire delivery team. They make directional and trade-off decisions—they do not execute all the work themselves. In a 10-15-person IT organisation, not everyone can be in a trio; otherwise, core work would stall.

Trios are not permanent—they form around defined problems and disband when those problems are resolved. Assigning people to trios without clear problems only recreates unnecessary committees, which this approach avoids.

A warning: ‘the trio owns it’ can quickly become ‘no one owns it.’ Each role still has distinct accountability. While there is a shared outcome, each person has specific responsibilities.

Making It Work: The Structural Stuff

So, of course, the question is: which problems actually warrant a TRIO, and which should just be dealt with in the usual manner? This normally falls into several categories.

Often, the projects warranting a TRIO are ones where you’ve either got high ambiguity, cross-functional dependencies, or strategic importance, or maybe a combination of all three.

What often doesn’t require a TRIO are well-defined requests, business-as-usual work, and pure technical debt that can be dealt with via a single stream.

Because what the leaders also need to figure out is how to prevent every business stakeholder from demanding their engineer in a TRIO, because that then ends up, as mentioned earlier, with a lot of people in TRIOs and no one to actually do the work.

Too many Trios spread technical people thin; too few create confusion and inertia. Balance isn’t a one-time fix. Leaders should make trade-off decisions a regular practice. A lightweight quarterly review of Trio assignments and resources helps sustain balance and builds leadership strength, ensuring strategic and operational goals are met.

Finding balance here is challenging.

The next question is about meeting cadence, but the real question here is authority. C-Suite execs tend not to really care about meeting frequency, except that it gives the optics of productivity without necessarily being productive. What they actually care about is: “Will this create more meetings that waste time or fewer escalations that waste their time?” By framing authority as a trust dividend, you can link decision autonomy to fewer escalations and faster learning loops. This perspective encourages executives to see granting authority as a strategic investment rather than a risk, fostering a more efficient and responsive organisation.

The answer depends on whether the Trio has the authority to decide. If every call goes up the chain for approval, the Trio becomes just another layer. The real cadence question is what the Trio can decide without asking permission, and how often they need to meet to exercise that authority. This could be daily, a few times a week, or weekly. The Trio’s priority matters, too. A long-running Trio may only need to meet once a quarter, and that can work.

As we talk about decision rights, this is the load-bearing section of making it work. Most organisational dysfunction stems from ambiguity. The business thinks that they set priorities. IT thinks they’re drowning in competing priorities with no clear ranking because everything comes in super urgent. IT thinks they’re empowered to push back on bad ideas. The business thinks it’s IT’s job to execute whatever ideas come in and that IT’s being obstructionist on purpose. No one knows who can say no to the CEO’s pet project. The trio model only works if you can explicitly answer these questions: Pose the ‘Who says no?’ test. Challenge your team to summarize their trio’s escalation rules in one tweet. For instance, ‘Only the Tech Lead can veto based on feasibility; Business Owner has final say on customer priorities.’ This brevity test often exposes lingering ambiguity that fuels dysfunction.

• Who can green-light work entering the trio’s scope?

• Who can kill or descope work that’s not feasible?

• What trade-offs can the trio make without escalation?

• What must escalate? (Things like budget thresholds, timeline changes, scope changes beyond X)

If you don’t answer these questions and make them prescriptive, you’ll end up with the same bottlenecks you have now, just with a new name.

Of course, this then gives you the ability to define success metrics. The real question is shared accountability. Because the classic setup measures IT on delivery velocity and system uptime, and businesses on revenue and customer outcomes, these can point in opposite directions. If the metrics aren’t shared, the trio becomes adversarial: the business owner pushes for speed, the tech lead pushes for quality, and the third person picks a side or checks out. Which is, of course, very suboptimal. A trio needs at least one shared outcome metric they’re all accountable for, not IT delivering on time and the business adopting it. But this problem got solved, and here’s how we’re measuring it as a team, as a trio that can then be reported up the chain to their C-suite execs.

What Makes It Actually Stick: The Leadership Part

This is the pivot where you tell them the uncomfortable truth: the structural stuff will fail if leadership behaviour doesn’t change. Be direct but not preachy:

• Stop sending “urgent” requests directly to IT that bypass the trios. Every time you do, you undermine the model.

• Stop letting business leaders commit to timelines without technical input. If sales promises a feature by Q3 without talking to engineering, you’ve already broken the system.

• Protect trio time. If your 10-person IT team is running 15 trios plus BAU support plus incidents, you don’t have a trio model - you have a burnout model.

• Hold business owners accountable for outcomes, not just engineers. If a project fails, the business sponsor should be in that post-mortem, too.

This is where you earn trust by not just telling them what to build, but what to stop doing.

Moving on to what actually makes this type of design pattern stick in an organisation, the real truth is that the structural stuff will fail if leadership behaviour doesn’t change. This is a big part of why many of these processes just don’t stand the test of time: leadership is the one unwilling to change, not the actual ground-level staff who implement these patterns.

As an organisation, you need to stop sending urgent requests directly to IT that bypass the trios. Every time you do, you undermine the model. If every request is urgent, there is no way for an IT team to prioritise requests.

Equally, the business needs to accept that sometimes deliveries will get pushed out because legitimately urgent requests will come in that need to be facilitated and actioned, and that will take time and have a knock-on impact on the delivery of other stuff.

We also need to stop letting business leaders commit to timelines without technical input because we’ve seen this time and again. If the sales team promises a feature by Q3 without talking to engineering, you’ve broken the system because there’s no input from the people doing the implementation. It’s just a guarantee that those people will actually adhere to those timelines to understand their existing workload and delivery schedule, and whether that is even feasible.

Also, it is important to protect Trio’s time. If your 10-person IT team is running 15 trios, plus BAU support, and an instance, you don’t have a trio model; you have a burnout model. (Trinkenreich et al., 2023) Everybody is knee-deep in trios trying to plan for the next bit of work, trying to understand what’s going on, and, of course, trying to do that whilst supporting existing systems and developing new ones is nigh on impossible just due to the nature of the scale of the requirements and the requests coming into that organisation.

The last point that actually makes this stick is holding business owners accountable for outcomes, not just engineers. If a project fails, the business sponsor should be in that post-mortem too because they are part of this chain. It is not purely an engineering function that causes the failure of a specific plan. It can be a lack of specification or accurate scoping, and it takes more than two to tango. Ensuring that the business is also held accountable for its side of the bargain is important when running post-mortems to understand where the failures actually occurred. If we all just always assumed that the business is right and the IT organisation is wrong, you will forever be in a spiral of finger-pointing, with the IT organisation taking the blame.

A Realistic Starting Point

As we reach the end of this post, it would be nice to offer some actionable pointers people can work with.

So what I’m saying to you is: pick one or two high-value, high-friction problem areas inside of your organisation and form trios there. Begin with the following steps: First, select and clearly define the problem areas where a Trio can bring substantial improvements. Second, assign clear roles to each Trio member, ensuring everyone understands their responsibilities and contributions. Third, establish specific metrics to measure success, such as customer satisfaction scores or time-to-market metrics. Finally, schedule a review after a quarter to assess the Trio’s performance, identify challenges, and make necessary adjustments. Run it for a quarter, see what breaks and iterate over it. It’s not an overnight change that’s going to suddenly transform your organisation, but it will enable more direct communication and allow multiple groups to take on responsibility for ensuring that requirements are defined and that everybody has a stake in the outcome. It’s not just chucked over a wall, and the IT team are expected to pick it up and run with it.

It’s also important to acknowledge that this is hard and takes longer than you’d like. It’s a difference between a real fix and another process that becomes what I call shelf-ware. We try something, we try it for a week, we decide that it doesn’t work, and then we chuck it on the shelf and pretend that we’ve never done it. We continue doing the same things over and over again, leading to significant employee dissatisfaction, churn, and project delivery failures. (Costa et al., 2024) But remember, real change is not easy, and it takes bravery to step out of the comfort zone. Let this be a call to courage for leaders—be the catalyst for change and sustain the momentum to truly transform your organisation.

Of course, it takes effort to make this work; it takes buy-in across the organisation. If you don’t get it, you’ll never succeed.

You spend what feels like weeks in sprint planning, daily standups, twice weekly standups for something else, roadmap review meetings, backlog grooming. And then someone else from the business says “well actually we need this other thing.....” and the team just feels like they’ve been blindsided with some random request and project managers feel like they’re throwing specs over the wall and hoping they land somewhere useful.

This feels like it happens at some many different organisations. All the effort going into planning to then find that people are none the wiser, or worse, have literally no idea whats going on in general. The problem here though isn’t missing rituals, its that the rituals we’ve put in place are optimised for the wrong thing.

Cross functional ceremonies, meetings whatever you want to call them are designed to move information or status updates, requirement hand offs, timeline syncs, functional information. Whats the word? Perfunctory: carried out with a minimum of effort or reflection. These meetings answer the “what” and the “when” but without building the rapport between teams, people and employees needed for “why does this matter to you?” or “what are you worried about?” These types of questions are really key to the world we live in. Sure timing, requirements, they’re all important but developers developing without understanding the bigger picture, the concerns that others outside the develop bubble have, will forever be on the back foot.

I can give you some examples. First up an example of a well run, if not draining standup. So when I worked at NASA JPL we worked on a piece of science software that was required to process data downlinked from NASA’s Perseverance Rover. For years, 2 or 3 at least I joined nightly standups because I was based in the UK and we had to get Los Angeles, Melbourne and London on the phone at the same time. So 11pm for 20-30 minutes every night was the only time that really worked. Of course this wasn’t the most productive of times because I’d already been working all day, but they were effective, we could hand work off, discuss blockers etc without it unduly impacting too many people.

Then after that I went to work for a medical startup. We had a team on the US East Coast and a team in India. This made life especially hard, trying to keep teams in the US functioning, information handed off and project managers happy, with the scrum masters in India trying to wrangle ill defined requirements and me sat somewhere in the middle trying to both get my own work done and answer questions on both sides of the pond. The problem was, they kept having daily standups but the transactions were very much pure business, no questions were asked, no real conversations were had and as such the business owners were frustrated and the developers jaded.

The 5 Dysfunctions

So now I must ask the question. Who’s heard of Lencioni’s Pyramid? Excellent book and topic of many other blogs. I’ll just add this one to the list! In 2002 Patrick Lencioni wrote about the 5 dysfunctions of a team. And it walks through the many issues and sticking points teams face as they see to grow together, in doing so Patrick looks at the causes of organisational politics and team failure. Something a lot of us can relate to.

The five dysfunctions are:

• Absence of Trust

• Fear of Conflict

• Lack of Commitment

• Avoidance of Accountability

• Inattention to Results

The issue with the way a lot of organisations work though is that they operate at the top levels of the pyramid. They operate with a lack of accountability and inatention to results, whilst assuming that the foundations, trust, fear of conflict and lack of commitment already exist. Of course in so many cases they don’t.

Over time though what you will learn is that if your rituals, those meetings you all feel are so important are missing the foundation layers you’re building on sand and the whole thing will crumble around you.

So what is the absence of trust? The absence of trust is the unwillingness to be vulnerable within the group or within the wider organisation, which leads to a lack of trust between your peers. Fear of conflict involves seeking artificial harmony over constructive passionate debate and not understanding when to both push back and hold your ground when other people are passionately arguing in a different manner.

Lack of commitment is around feigning buy-in for group decisions which creates ambiguity throughout the organisation. People flip-flop between different points of view depending on who they’re speaking to and cause greater work for those who are then empowered to implement the changes.

Avoidance of accountability involves ducking the responsibility to call peers and superiors on counterproductive behaviour which sets low standards. Therefore people further down the chain are expected to hold themselves to higher standards than the boss is implementing the standards themselves which is counterintuitive. Lastly the inattention to team results which focuses solely on personal success status and ego before team success which of course is detrimental to the team output as a whole.

Of course this begs the question of what do we do instead? How about starting a meeting between technology groups with Q&A? Not forced icebreakers but structured moments where admitting uncertainty is normalised. I’ll give you an example: how about an engineering lead opening a planning session with “here’s what I don’t understand about this quarter’s priorities” and inviting the PM to do the same?

Therefore the backwards and forwards normalises the uncertainty between the groups, both dictating what should be implemented and the people doing the implementation work themselves, reducing the friction and allowing everybody to voice their uncertainties as a group.

On top of rituals that surface vulnerability, how about rituals that practise disagreement? Teams that only agree in meetings and fight in Slack have a ritual problem. What does it look like to build productive friction into cadence? For example, how about a team that introduces a red team rotation into roadmap reviews deliberately asking the questions that would create the friction to ensure that it is discussed during the meeting rather than offline while grumbling into your Slack channel?

Finally how about rituals that build shared context, not just shared information? The difference between “Here’s the roadmap” and “Here’s the trade-off I struggled with and why I landed here”. For example a founder who replaced quarterly all-hands presentations with smaller cross-functional context sessions. It allows people to be able to discuss smaller chunks of work while also understanding the cause and reasoning behind those decisions being made, rather than a holistic super-high-level top-down view of work that’s supposed to be carried out across a broad swathe of the engineering function. We actually saw similar with the medical project we were working on that I mentioned earlier when we split up the groups into much smaller teams, allowing for better communication across different groups while still allowing the managers to track the broader project goals at a higher-level delivery.

Of course involved in all of this is the conflict resolution dividend. When rituals build trust and normalise disagreement, conflict resolution becomes cheaper. Teams don’t need to elaborate escalation paths because I’ve already practised navigating tension in lower stakes settings. So when a problem arises, the fallout and the ability to deal with it is far reduced.

Over the years I’ve worked in both types of environments, where engineering functions regularly get overwhelmed by both requests, feedback, and negativity and those that thrive.

Having worked in various startups in London over the years, we often saw small engineering teams with unrealistic goals and deadlines required to deliver on time. Then project managers would get annoyed when delivery was delayed for reasons that they couldn’t comprehend. Of course if you had dealt with this at a smaller team level focusing on individual pieces rather than the bigger picture, this may have been alleviated or remediated earlier.

Equally having explained it in an earlier in this blog, I’ve worked in many environments where small-scale conflict resolution was dealt with both effectively and efficiently by product managers, project managers, and engineering leaders. That makes a huge difference when it comes to delivering on time and to expectation because there’s less fear of retribution from within the engineering team should something crop up that would negatively impact the timelines involved.

Conclusion

I understand why there is pushback when it comes to dealing with relationship building rituals. They obviously feel soft, they don’t feel like you are contributing directly to the delivery of the product or project, and so it’s harder to justify a relationship building ritual in a planning document than it would be a sprint review or something of similar nature. Leaders from large corps, non engineering corps etc may see these meetings as indulgent but when you’re talking about a large swathe of employees, trying to navigate requirements, deliverables and an ever changing tech landscape I believe they are invaluable.

If you’re a C-suite reader reading this article, then the return on investment is real but it compounds slowly. You won’t see results in the next sprint. But when it comes to the next crisis or when your team has to bail the company out because they’re being asked to do something that wasn’t in the plans or wasn’t in the roadmap, that’s when you’ll see the real return on investment. When it comes to these relationship-building meetings.

The organisations that get this right don’t have more meetings. They have meetings that do different work.

As we sum this blog post up, the structure of your rituals reveals what you actually believe about how work gets done. If the ceremony is about outputs and timelines, you’re communicating that relationships are someone else’s problem and not yours. Over time as we’ve stated earlier in this blog post, that will lead to larger problems that’s trickier to solve in ever-shortening timelines. Of course the other negative impact as well is the happiness and contentment of your staff. That also has a detrimental impact on output as staff get less inclined to do the extra work or put in all the hours as they get frustrated with an ever-ending list of stuff that needs to be completed.

We start this blog post talking about the requirements for meetings and how those meetings are structured. The organisations that get this right don’t have more meetings. They have meetings that do different work and communicate the requirements and deliverables in a way that resonates more effectively with the staff at hand.

It doesn’t all have to be transactional. It can be very much a relationship-building exercise while still getting the same points across in less time.

So if you’re a C-suite executive talking to your product managers or project managers, just have a think about how to best structure those meetings. What you can do as an organisation to increase the output and performance while reducing the friction and the resentment with inside the organisation by improving the relationships between different teams and the dependencies on those teams. They will forever thank you for it as conflict and expectations become a much easier attribute to manage across the organisation.

Let me direct you to a Linkedin Post that caught my eye the other day because I thought it was a joke, then it became apparant in the comments that he wasn’t joking.

Now the argument that you should use “more modern tooling” because its hot is one that’s been raging since the dawn of time. But of course, why change what works? And if a VM and SSH works for you, and you have a small team, have at it!

So this then leads me on to Kubernetes, what is it and should you use it. Kubernetes span out of Google where their Borg platform was dealing with massive Google sized deployments on a daily basis. Now if that doesn’t tell you what Kubernetes is good at nothing will. But for those of us who like to hack around on things, Kubernetes basically takes Docker images and a lot of YAML descriptors to deploy your application, or applications into a cluster of machines that then mangages your container, your networking, ingress, egress, permissions etc for you. Sounds great right?

Well in a number of cases it is great. It can provide a lot of fault tollerance, auto scaling, auto healing, all the things your business may be interested in leveraging and some of that, but not all of it comes out of the box.

Downsides

Okay so what are the downsides of Kubernetes? Lets start on a downer and finish more positively. Its super complex. Sure every cloud vendor has their own hosted solution, and there are a number of vendors offering up support for their Kubernetes variant if you’d rather deploy it elsewhere, but that doesn’t mean its easy to wrap your head around. Azure attempt to use Azure App Service to wrap Kubernetes in something that makes it easier to deploy apps. AWS just launched AKS Auto Mode in an attempt to reduce the management burden. They all know its hard work. Cluster security, cluster upgrades etc etc they all take planning and forethought on top of managing your own pods. And if you’re self hosting them your host VM also needs patching, upgrades and remediations.

On top of this the majority of Kubernetes is driven by YAML in some form or other. Some times this is simple, often its not, but its also the way you manage it, how do you deal with change requests, version control, config drift, different people applying policies etc? All of this requires careful management and its not really a technical problem its more of a policy and people problem, do you have the bandwidth to manage it?

At smaller scales Kubernetes is just more expensive. This obviously manifests itself in a number of ways and isn’t just 1 number. For example the additional complexity means more man hours. But it also means you’ve got more nodes, a control plane (sure it may be “free” in a vendor but you’re paying for it somewhere), more infrastructure for ingress etc. It all adds up.

Slower developer loops are also a real issue. You can, if you’re a sadist, run Kubernetes on a local machine. But this still requires the same YAML setup, the same config tweaking, the same maintenance. It can be great to debug issues in the cluster but in reality its still hard to manage and certainly not developer friendly.

Lastly on the downside, the mirage of Kubernetes portability. Of course Kubernetes is designed somewhat to offer universal configuration across cloud providers, this often doesn’t really play out in practice. You still need to have a good understanding of your vendors networking, its ingress options, its available storage etc. When you deploy storage, do you need slow disks? Cheap disks? High performance disks? Maybe both, how do you pick them how do you configure your setup to use them? It all adds to the operational complexity of a Kubernetes deployment.

Upsides

Portability is obviously real. We can deploy docker containers into almost any environment, on prem, in any cloud environment. And whilst there is a degree of needing to understand the environment and what it provides the core product itself probably doesn’t need any changes making it easily deployable in a range of different environments. Of course the other thing about docker containers, developers can also just run them using Docker Desktop, Podman etc making them runnable across an number of different control services.

The other part of deploy Kubernetes processes is the declarative infrastructure. Whilst having everything defined as a YAML file can certainly add more control complexity it also is a positive. Once you’ve described the state you want, Kubernetes will do its best to keep everything in sync. Self healing, rollbacks on failure, consistency across different environments. That all comes from the bit that also makes Kubernetes hard to manage from a process perspective.

Of course in reality there is a rich ecosystem of additional services, products and people who can all plug into your Kubernetes infrastructure. Products like Argo can provide excellent application management. Prometheus provides out of the box support for monitoring of your processes and so on. There is an endless amount of additional tooling that can both help and hinder depending on what you’re trying to achieve.

Resource Isolation provides an element of multi tenancy or security patterns across your resources. Obviously this could be achieved outside of Kubernetes as well but RBAC roles, network boundaries etc all provided out of the box. Now of course, these can be hard to configure and maintain, also you want to hook them into your main security systems, probably easier to do inside of a cloud vendors system but still obscure, ensuring you’ve got namespaces scoped properly, maintaining that network firewall at the correct place etc. This is what I mean by the man management of the change requests, it only takes one badly changed YAML file to blow that right open, but when it works well, having all your processes on a single scalable cluster can really make managing Kubernetes clusters a breeze.

Next on the list of positives is the extensibility. If it can’t do what you want it to the operator framework, the ability to extend and improve the environment is absolutely possible. The APIs have been standardised for a long time and this is what drives that ecosystem we mentioned earlier.

Deployment patterns can really help you, of course some of these need additional tooling if you’re going to do them well but things like Rolling Updates, Canary and Blue Green releases are all absolutely doable out of the box. Of course though what you need to ask is, do you really need Blue Green releases? Or did you just read about them and figured they sounded cool?

Last on the list of positives is observability hooks. One of the biggest problems with running anything inside a container is, “whats going on inside?”. Hooks for logs, metrics, traces can really help here, both inside your containers but also the traffic through and across the platform. The ecosystem across the observability sector has really grown over the past decade, standards like OpenZipkin(founded by my friend from back in the JClouds days Adrian Cole) have really aided this drive.

So how do I choose?

Now this is the million dollar(in some cases actually million dollar) question. What is it you’re trying to achieve? Can you achieve the same outcomes by doing less? If the answer to this question is “yes”, then do less. I honestly believe in the modern tech landscape if you can do something by spinning up a VM and running a process on there, even if that thing sits inside a docker container for convenience, then it probably should. Of course there are times that doesn’t hold true. But even if its not true, do you still need a Kubernetes cluster? Or will something like Lambda for backend, ECS for frontend, Google Cloud Run, or Azure App Services work just as well?

In the 21st century there are multiple ways to spin this cat. Even if you’re looking for portability, would it cost more to package your app as a Docker container and then write separate deployment scripts for each cloud providers bog standard application deployment?

These are the types of questions you need to ask, not just the cost in running the thing, people think instantly to hosting costs, but when you’ve got a platform as complex as Kubernetes, whats the human cost, both monetarily and in terms of raw time.

Kubernetes is a powerful beast, in the right places it makes a lot of sense. But in the wrong places it can be a security, maintenance and management disaster waiting to happen.

Crawlers are often designed for one task, crawling websites, crawling a specific target, and I wanted to create something that would serve more than one purpose. There is always a lot of replicated code when building separate crawlers and I wanted to remove as much of that as possible whilst also bringing a standardised output format for all the sites we crawled to make the ETL process as easy as possible.

This is why the distributed crawler was born. It allowed us to crawl both Telegram and the Youtube Data API without having to rebuild huge chunks of very similar crawler logic across both platforms. I also wanted to be able to support multiple cloud backends, quite extensible configuration properties for the researchers who were going to use it, and it have as small a footprint as possible.

I started writing the Telegram portion of the crawler, as ever building and testing something that required many SIM cards to operate was an interesting challenge. With a Sim farm in my office, I succeeded in connecting the crawler to multiple Telegram API backends using a distributed technique, allowing for quite effective scalability of the platform. Of course, ensuring that the data you collect is both accurate and complete is an interesting challenge when you’re trying to crawl multiple channels inside of a Telegram account to create a global picture of what’s happening on the Telegram network. In doing that, it involved an awful lot of manual checking, check points, and check sums to ensure that the platform itself curated the data it needed for the researchers to be able to do what was important to them.

The other interesting challenge was collating the multimedia that is obviously synonymous with every social media network these days. That involved us looking for files, downloading different file types, processing them, and extracting numerous extra data using OCR techniques and metadata extraction on the files themselves to be able to provide search context and lookups for researchers when looking at specific topics that are embedded in the multimedia themselves.

Once I built out this framework, I realised that I wanted to make it pluggable to add the YouTube Data API. At that point, I leveraged Claude Code to help me refactor the existing codebase to a more generic interface that would allow me to plug in different social media endpoints. This was quite effective and it provided a useful entry point for Claude to then write the YouTube API layer leveraging the Golang libraries that YouTube had provided. The YouTube layer was easier to implement in a lot of respects in the fact that it didn’t require access to SIM cards. However, the YouTube Data API is quite restrictive in terms of both the data you can get and also the amount you can collect. Luckily, though, I was working for Princeton University, and so we made a valid submission to YouTube to allow access to their research API. This is basically exactly the same as the data API, but scaled up for larger volume data access. What you don’t get access to though with the data or the research API is the video content itself, nor access to the transcriptions for the video content (unless of course you happen to be the owner of that video). This means that you can analyse titles, descriptions, things like engagement metrics with comments and the like in the videos themselves, but you can’t actually access the content of the video itself. This is not immediately problematic, but could cause problems with people wanting to build upon this in the future. There is a workaround, and we will look to add this to the distributed crawler in the coming future. This is an open-source project called InnerTube. InnerTube leverages a public yet undocumented API that allows access to a number of different YouTube features and functionality that aren’t readily available inside of their data API. This does include transcriptions that we do not have access to, as well as things like similar videos and things that you would see in the sidebar. The reason for this is that people assume is that smart TVs and other smart devices need access to YouTube without a user being logged in, so they have to have a public API that is available to them. InnerTube cleverly leverages this to allow for it to be able to provide the same service as a library for other applications.

I mentioned earlier that I wanted to be able to use multiple different backends to be able to support the distributed crawler. The reason for this, of course, is that outside of running it inside of Azure inside of a Kubernetes cluster where we were going to run it, people might want to run this in a range of different places. Being able to leverage different backends was important:

1. Local mode so you could run it on your laptop (like a lot of people would do)

2. As sort of Azure CLI backend as an SDK back-end to support standard cloud operations

The other one that I implemented inside a distributed crawler was support for a framework called DAPA (a microservices framework from the Linux Foundation). The cool thing about DAPA is that it has a number of different plugins for a whole range of different storage features and functionality. Rather than me having to implement a number of different backends outside of the core ones I’ve just mentioned, it would allow for people to be able to use different services to store the data in depending on what they were interested in leveraging as their own backends. This gives users a very flexible deployment strategy when it comes to spinning this stuff up, especially in cloud environments and even more so inside of a Kubernetes deployment.

The other interesting thing with DAPR is that you could also use not just storage but other messaging protocols to be able to send the data around. So rather than persist it straight to disc you could for example send it to a message bus and send that to a different service to process that data in real-time. And so the idea of real-time data processing and streaming for a distributed crawler became a reality inside of the project because we could send that data directly to a message bus directly to Databricks and get it picked up and processed almost instantly.

The final piece of this puzzle was the use of a unified output format that I alluded to earlier. This format followed very closely to an output specification originally used by Junkopedia, and it is a standard output specification that they use across different social media sites in their service. It made sense for us to be able to mimic it closely so that we could also ingest their data should we need to, to be able to bolster the platform.

The idea was that every single social media post has basically the same content; it’s just the metadata that surrounds it is slightly different. So being able to support those different types whilst also making it easy for the platform to output similar formats would allow us to then be able to ingest that data very easily when it hit Databricks. And so regardless of which social media platform you use, you knew that the process to get that data into a readable format would be a lot shorter than if you were using individual crawlers for each different service.

And so that was basically the distributed crawler. The cool thing about it was I was allowed to open source it. You can find a link to the distributed crawler on GitHub right here. That is the platform to date. It supports a number of different backends as we’ve mentioned. It supports Telegram and YouTube, but we do have plans to add more to it over time. And make it more extensible and more useful for a whole range of different social media sites and probably also just standard website crawling as well into the future.

Gain that competitive edge, leverage what is available, but do so in the way that makes the most sense to your business.

Join me for my next live video in the app

Get more from Tom Barber in the Substack app

Available for iOS and Android

Get the app

Managing stakeholders in cloud architecture and application development is a lot like running a railway system: different passengers (stakeholders) have different destinations (goals), some demand speed (quick features), others want reliability (stability), and no one wants unexpected delays. Here’s how to align teams, manage expectations, and prevent scope creep without derailing a project.

Here’s some ideas:

1. Aligning Business and Technical Goals

2. Understanding the Customer Perspective

3. Bridging the Gap Between Technical Jargon and Business Speak

4. Defining Success Metrics Together

5. Prioritisation Through Feedback Loops

6. The Role of Transparency in Building Trust

7. Fostering a Culture of Continuous Improvement

8. Mitigating Risk Early

9. Celebrating Wins Together

10. Communication Tools and Practices

Find out more in the video!

1. Bringing Order to the Chaos

2. Performance Optimisation

3. Data Quality and Consistency

4. Interoperability with downstream systems

5. Data Lineage

6. Cross Team Collaboration

7. Facilitating Data Discovery

8. Future-proofing your architecture

9. Improving Query Accuracy

10. Hybrid Data Architectures

Data models continue to be an essential part of what we do. What other reasons can you think of?

Subscribe now

Subscribe now

Scalability is the capacity of your system to adapt to changes—whether it's a sudden surge in traffic or the gradual addition of new services and features. Applications face performance bottlenecks without scalability, leading to slow response times, downtime, and poor user experiences. It's about being prepared for success and future-proofing your architecture to handle shifts in user behaviour and business goals. Who remembers websites getting slashdotted regularly not that long ago? Getting 'slashdotted' refers to the sudden surge in traffic that a website receives after being featured on the technology news site Slashdot, which often overwhelms servers and causes the site to crash or become unresponsive.

In this post, we'll explore the core principles of designing cloud applications for scale, diving into best practices, patterns, and architectural considerations that help ensure your system can grow alongside your business. We'll look at leveraging cloud-native tools and approaches to maintain performance, reliability, and efficiency, no matter how your user base or data needs to expand.

Subscribe now

The Fundamentals of Scalability

Scalability in cloud applications involves designing your system to handle increased loads efficiently without compromising performance. Scalability comes in two forms:

1. Vertical Scalability (Scaling Up): Adding more power (CPU, memory, etc.) to an existing server.

2. Horizontal Scalability (Scaling Out): Adding more servers to distribute the load.

Horizontal scaling is generally preferred in cloud environments due to its cost efficiency and resilience benefits. Cloud providers like AWS, Azure, and Google Cloud simplify horizontal scaling through managed services, autoscaling features, and distributed architectures.

Vertical scalability, also known as scaling up, refers to adding more power (CPU, memory, etc.) to an existing server to improve its capacity. This approach can be simpler to implement since it doesn't require application architecture changes. However, it has inherent limitations because a single server can only be scaled up to a certain point before hardware limits are reached. Vertical scalability is often practical when an application has a single point of workload that cannot be easily distributed.

When deciding between vertical and horizontal scalability, it's essential to consider the nature of your application and growth patterns. Vertical scalability is suitable for smaller, less distributed systems where a quick boost in resources is needed. On the other hand, horizontal scalability is generally preferred for cloud-native applications that need to handle unpredictable and significant growth, as it allows for distributing the workload across multiple instances and reduces the risk of hitting resource limits.

Key Design Principles

Microservices Architecture

Designing applications with a microservices architecture enables different components to scale independently. Unlike monolithic architectures, where all features are tightly coupled, microservices break down functionality into more minor, isolated services that communicate via APIs. This allows you to scale specific application parts based on demand, optimizing resource usage.

Containers, particularly Kubernetes, provide an ideal platform for deploying microservices. Containers encapsulate each microservice with its dependencies, ensuring consistency across development, testing, and production environments. Kubernetes then orchestrates these containers, automating the deployment, scaling, and management of microservices, which makes it easier to maintain the independence of each service while ensuring the overall system scales smoothly as demand changes.

While microservices sound appealing due to their scalability and flexibility, they can also add significant complexity to your platform. Managing a distributed system with multiple microservices requires robust orchestration, monitoring, and communication strategies. Issues such as inter-service latency, network failures, and distributed data management can introduce challenges that require careful consideration. The increased complexity means that the benefits of microservices are often best realized when an application has reached a particular scale or level of complexity that justifies the additional overhead.

• Example: In an e-commerce platform, the checkout service can be scaled separately from the product listing service, ensuring peak traffic during sales events doesn't impact overall application performance.

Subscribe now

Statelessness

A stateless architecture simplifies scalability. Stateless components do not retain data between requests, making distributing requests across multiple instances easier. Leveraging stateless designs means that each server can handle any request, simplifying load balancing and reducing the chance of bottlenecks.

• Implementation Tip: Store session information in databases or distributed caching systems like Redis rather than in memory on individual servers.

Database Scalability

Scaling databases can be challenging, but several strategies can help:

• Sharding: Splitting your database into smaller, faster, and more manageable parts.

• Read Replicas: Read replicas offload read operations from the primary database, which is particularly useful for read-heavy applications.

• NoSQL Databases: Consider using NoSQL databases for extensive, unstructured data. They offer more effortless horizontal scalability compared to traditional SQL databases.

Cloud platforms provide managed database solutions such as Amazon RDS, Azure Cosmos DB, and Google Firestore, which have built-in scaling features to help handle large amounts of data and traffic.

Load Balancing

Load balancing is essential for distributing incoming traffic across multiple servers to ensure no single server becomes a bottleneck. Managed load balancers, like AWS Elastic Load Balancing (ELB) or Azure Load Balancer, are critical components for high availability and reliability.

Load balancing can also include geographical distribution for global applications and routing users to the nearest data centre to reduce latency and improve user experience.

Caching Strategies

Implementing effective caching can drastically improve application performance and reduce the load on your servers and databases. Use caching layers like CDNs (Content Delivery Networks), such as Cloudflare or Amazon CloudFront, to cache static content close to your users. For dynamic content, in-memory caches like Redis or Memcached help store frequently accessed data and reduce repeated database queries.

Autoscaling

One of the most significant advantages of cloud computing is the ability to scale resources up or down based on current demand automatically. Autoscaling ensures your application can handle traffic spikes while minimizing costs during low-traffic periods.

• Example: Use AWS Auto Scaling or Azure Virtual Machine Scale Sets to adjust the number of instances running your application dynamically.

Event-Driven Architectures

In event-driven architectures, services communicate via events rather than direct calls, which decouples components and improves scalability. Message brokers like AWS SNS/SQS, Azure Event Grid, or Apache Kafka are commonly used to manage the flow of information between services asynchronously, ensuring that your application can scale under varying loads.

Share

Choosing the Right Cloud Services

When designing for scale, managed services can offload much of the complexity of maintaining and scaling infrastructure.

• Managed Databases: Reduce the operational burden using managed database services that handle backups, scaling, and updates.

• Serverless Computing: You can execute code without managing servers with services like AWS Lambda or Azure Functions. Serverless computing automatically scales in response to incoming requests, ideal for applications with unpredictable workloads.

• Containerization: Using containers with Kubernetes or Amazon ECS allows for easy horizontal scaling and management of microservices, promoting efficient resource usage.

Designing for Failure

A core part of designing scalable cloud applications is accounting for failure. Cloud environments are inherently distributed, and failures are inevitable. Here are some practices to build resilience:

Redundancy

Introduce redundancy at every level—multiple servers, load balancers, database replicas, etc.- to ensure there are no single points of failure. Multi-AZ (Availability Zone) deployments are crucial for high availability.

Graceful Degradation

Design your application so that if a part fails, it degrades gracefully rather than crashing entirely. For example, if the recommendation service in your application goes down, the core features should still function.

Monitoring and Alerts

Use tools like AWS CloudWatch, Azure Monitor, or Prometheus to monitor application performance and set up alerts for unusual behaviour. This proactive approach helps maintain system health and preemptively addresses scaling issues.

Cost Optimization Strategies

Scaling can lead to increased costs, so it's vital to implement cost optimization strategies.

• Right-Sizing Resources: Continuously analyze the usage of your resources and adjust them to the appropriate size.

• Spot Instances: Use spot instances for non-critical workloads, which can significantly reduce costs.

• Serverless and Autoscaling: Pay only for what you use with serverless and autoscaling to avoid over-provisioning.

Real-World Example: Scaling an E-Commerce Application

Consider an e-commerce application experiencing fluctuating traffic, with significant spikes during holidays or sales. Here's how you might design it for scale:

1. Microservices: Separate the checkout, product catalogue, user management, and payment processing into distinct services.

2. Load Balancing and Autoscaling: Use load balancers to distribute user requests and set up autoscaling for the checkout and product catalogue services to manage spikes.

3. Database Strategy: Use read replicas for the product catalogue and shard the user data to improve performance.

4. Caching: Cache product images and descriptions in a CDN and use Redis for session storage to reduce database load.

5. Event-Driven: Implement an event-driven order processing system using message queues to handle payment and inventory updates automatically.

Challenges in Scaling Cloud Applications

Scaling cloud applications comes with its own set of challenges, including:

• Data Consistency: Maintaining data consistency across distributed components can be difficult. Strategies like eventual consistency and distributed transactions can help.

• Latency: As you scale globally, latency can become an issue. Techniques such as edge computing and data locality can help reduce the effects of latency.

• Complexity: Microservices, autoscaling, and distributed databases complicate development and operations. Proper tooling and monitoring are essential to manage this complexity effectively.

Subscribe now

Conclusion

Designing cloud applications for scale requires careful planning, leveraging cloud-native architectures, and making the most of managed services to handle infrastructure complexity. By breaking down monolithic applications into microservices, implementing stateless components, optimizing databases, and effectively using autoscaling, you can create an architecture that grows with your business needs.

Remember that scalability is not just about adding resources but optimizing your architecture to handle growth efficiently. By considering resilience, performance, and cost optimization, you can ensure your cloud application scales seamlessly while delivering a consistent user experience.

Subscribe now

From privacy and transparency to fairness and accountability, we dive deep into how tech professionals can create responsible AI that serves society ethically. Whether you’re a developer, data scientist, or just curious about AI’s impact on our world, this video is for you!

One of the biggest challenges, which isn’t new, is bias. Algorithms, trained on historical data, can inadvertently pick up and leverage societal prejudices, amplifying disparities in critical areas like hiring(Amazon!) healthcare and criminal justice. To help avoid this, teams need to work out how to incorporate diverse data sets and establish more rigorous checks during model training and testing. Building an interdisciplinary approach, inviting insights from social sciences, ethics experts and user feedback, can help uncover and address biases early.

Subscribe now

Data privacy, as touched upon with GDPR is also still very important. With growing attention paid from both regulators and the public, it is essential to prioritize user consent and transparency regarding data. Those of us at the forefront of this technological advancement should argue for clear data handling policies that protect user rights without degrading model performance. Building this level of trust requires going beyond compliance and working to foster a culture that values privacy at its core.

Finally, in this short article, accountability matters. Who is responsible when AI systems err or data misuse or misrepresentation results in harm? A clear governance framework is essential, detailing responsibilities and escalation procedures. Audits, regular and in-depth ethical reviews, and transparent reporting processes can help ensure that teams remain accountable, but only when implemented properly and executed effectively.

Ultimately, ethical AI and data-driven engineering are about proactive choices. By embedding these principles into every stage of our work, we can harness technology's transformative power while respecting and protecting the values that bind us as a society.