Everything got the multimedia label. If it had speakers and a CD-ROM drive, it was a Multimedia PC. The badge told you almost nothing about what the machine could actually do, but it sold units.

Fast forward 30 years (and yes, I feel old too) and we’re living through exactly the same thing with AI. If it plugs into a wall, it’ll have some AI strapline somewhere, it seems. Your PC, your phone, your watch, your fridge, your microwave, your oven. Everything is “AI-ready” or “AI-powered.” The label is everywhere, and it tells you almost nothing.

The Problem With a Three-Letter Word

The reason “AI” has become such a useless label isn’t that people are lying. It’s that the term covers such an absurdly wide range of technology that it’s effectively meaningless without qualification. It’s like saying a vehicle has an engine. Okay, is it a lawnmower or a 747?

The mainstream conversation right now is dominated by Large Language Models. The chatbots, the text generators, the things most people picture when they hear “AI” in 2025. But the tech world seems to have collectively forgotten that AI has been around for a lot longer than the past few years. The foundations were laid by Alan Turing and others in the 1940s and 50s. The first artificial neural network, SNARC, appeared in 1951. The first program that could learn from its own mistakes, a checkers-playing system, arrived in 1952.

My own career has taken me across the gamut of computer science and ML projects, and while I’m no academic researcher, I’ve worked alongside some very clever people building very different types of models to solve very different problems. Statistical models for fraud detection. Computer vision systems for scientific instruments. Recommendation engines. Sensor monitoring. All of it “ and none of it an LLM.

So when everything gets called “AI-ready,” the question that should follow is: what kind?

Three Questions That Actually Matter

Instead of accepting the AI label at face value, I’d suggest asking three questions. They won’t make you an expert, but they’ll cut through the marketing fog faster than anything else.

1. Is it learning, or is it following rules?

This is the most basic distinction, and it’s the one most often obscured by marketing. A genuine machine learning model improves its performance based on data. It identifies patterns, adjusts, and gets better over time. A rules engine executes a predefined set of if/else logic that a human wrote.

Both are perfectly valid tools. Sometimes a well-crafted rules engine is exactly what you need. It’s fast, predictable, and easy to audit. There is absolutely nothing wrong with that. But calling it “AI-powered” sets an expectation that the system is doing something it isn’t. If your vendor can’t clearly explain whether their product is learning from data or following a script, that’s a red flag, not because rules engines are bad, but because the distinction matters for how you evaluate, trust, and maintain the thing.

2. Is the output deterministic or probabilistic?

This is the question that I think matters most, and it’s the one that gets lost almost entirely in the current conversation.

A deterministic model, given the same input, will give you the same output every time. You can test it, validate it, explain it, and predict its behaviour. A probabilistic model gives you its best guess, a plausible output that might differ next time you ask the exact same question. If you’ve ever asked ChatGPT the same thing twice and got different answers, you’ve experienced this firsthand.

Neither approach is inherently better. They serve different purposes. But the implications for trust, governance, and risk are completely different.

Consider financial transaction monitoring. If you’re flagging potentially suspicious activity, do you want a model that gives you the same risk score every time for the same transaction? Or one that might score it differently on a different day? Both approaches exist, both have their place. But you’d better know which one you’re deploying and why, because the regulatory, audit, and compliance implications are worlds apart.

The same applies in healthcare, manufacturing, legal tech, anywhere the stakes are real. A deterministic model that classifies a tumour the same way every time is a fundamentally different tool from a probabilistic one that gives you its best interpretation. Both might be called AI. The governance they require couldn’t be more different.

This is also where a lot of the “boring” AI lives. The workhorses that don’t make headlines but run critical infrastructure. Fraud scoring on your credit card. Pricing algorithms when you book a flight. Predictive maintenance on industrial equipment. These models are often deterministic, explainable, and battle-tested over years of data. Nobody writes breathless articles about them, but they’re doing higher-stakes work than most LLM deployments. Nobody dies if a chatbot writes a bad email. A misclassified fraud signal or a missed anomaly in a sensor reading is a different story entirely.

3. Is it making decisions, or informing them?

The final question is about autonomy, and it changes everything about risk, liability, and trust.

A model that flags a suspicious transaction for a human analyst to review is a fundamentally different proposition from one that automatically blocks the transaction. A system that highlights a potential defect on a production line for an inspector to check is not the same as one that rejects the part without human involvement. A tool that drafts an email for you to edit is not the same as one that sends it.

The governance, liability, and customer experience implications shift dramatically based on where the human sits in the loop. An AI system that surfaces information and lets a person decide carries one kind of risk. An autonomous system that acts carries another entirely. Both might be labelled “AI-powered.” The label doesn’t tell you which one you’re buying.

The Label Will Eventually Become Meaningless, and That’s Fine

The multimedia wars resolved themselves. We stopped caring about the MPC badge and started caring about what the machine could actually do. The same will happen with AI. Eventually, the marketing label will fade, and people will just talk about what the technology does, how it works, and whether it’s the right fit for the problem.

But right now, we’re in the messy middle. The label is everywhere and means almost nothing. Every product is “AI-ready.” Every vendor has an AI story. The companies and leaders that will get the most value from this era aren’t the ones buying everything with an AI sticker on it. They’re the ones who understand what kind of AI solves their specific problem and can tell the difference between a lawnmower and a 747.

So the next time someone pitches you an AI-powered product, skip the demos and the buzzwords. Just ask three questions: Is it learning or following rules? Is the output deterministic or probabilistic? Is it making decisions or informing them?

If they can answer clearly, you’re probably talking to someone who knows what they’ve built. If they can’t, well, that tells you something too.

Where do you want to go today? Turns out, the real question was always: what’s actually under the hood?

The scientists using Pixlise, the software we built to analyse X-ray spectrometry data from Mars, went from taking roughly a year to publish papers to doing it in weeks. That’s not because we wrote better code. It’s because we understood how they actually worked before we wrote any code at all.

Most engineering leaders I talk to don’t dispute that design matters. They just think it can wait. Ship first, polish later. Get something in front of users and iterate. The intention is good. Stay lean, move fast, avoid over-engineering. But there’s a hidden assumption in that approach: that design is about polish. That it’s the paint you apply after the structure is built.

It isn’t. And that misunderstanding costs more than most teams realise.

The False Economy of “We’ll Figure Out the UX Later”

The pressure to ship is real. I run a consultancy and I see it constantly. Stakeholders who want a working prototype by next month. Investors who want to see progress. Clients who’ve already announced launch dates. Stopping to do user research feels like a luxury when the sprint board is overflowing.

So what happens? Engineers become designers by default. We sketch out interfaces based on how we think the data should flow. We make assumptions about what users want because asking them would take too long. We ship something that works, in the technical sense, and wait for feedback.

The feedback arrives. It’s not what we expected.

BCG research shows that nearly half of C-suite executives report that more than 30% of their tech projects run over budget and late. The reasons they cite include misalignment between technical and business teams, unrealistic timelines, and insufficient resources. But underneath those symptoms there’s a quieter problem. Teams building software that solves the wrong problem. Or solves the right problem in a way that doesn’t fit how users actually work.

Barry Boehm calculated in the 1980s that fixing a defect in production costs roughly 100 times more than catching it during design. A study at Ricoh put it more starkly: the cost of fixing a design defect was $35 during design and $690,000 in field service. These numbers vary by context but the pattern is consistent. The later you discover a mismatch between what you built and what users need, the more expensive it becomes to fix.

And yet we keep placing the same bet. That we’ll be able to course-correct after launch. Sometimes we’re right. Often we’re not. And the lurking usability problems become emergencies that consume the engineering team’s roadmap for months.

What “Design” Actually Means Here

When I say design should be included early, I’m not talking about colour palettes or button placement. I’m talking about a specific kind of work that engineers often don’t have time or training to do well.

1. User research is the practice of understanding who will actually use this software, what they’re trying to accomplish, and how they currently get that work done. It’s not about asking people what features they want. It’s about observing and interviewing to understand the shape of the problem.

2. Interaction design is figuring out how users will accomplish their goals through the software. What’s the flow? What happens when things go wrong? Where are the decision points and how do we support them?

3. Design systems are shared vocabularies. Components, patterns, and conventions that both designers and engineers can reference. They reduce ambiguity and make it easier to build consistently.

4. Service design zooms out further. It’s about the processes people use, not just the screens they interact with. How does this software fit into their broader workflow?

At JPL, the Human Centered Design group takes this seriously. As one of their senior designers put it, “By designing our processes around the needs of the people who use them, we not only save the taxpayers money by making people more efficient, we maximise the science we can get out of each mission.” That’s the frame. Design as leverage for outcomes, not decoration.

What This Looks Like in Practice: The PIXL Story

PIXL, the Planetary Instrument for X-ray Lithochemistry, is an X-ray spectrometer mounted on the Perseverance rover. It analyses the chemical composition of Martian rocks at a level of detail that helps scientists understand the planet’s geological history and search for signs of ancient microbial life.

The instrument generates enormous amounts of data. Before Pixlise, the scientists analysing that data relied on Excel-based pipelines they’d built over years. Those pipelines worked. Scientists are resourceful. But they didn’t enable collaboration. Each researcher had their own spreadsheets, their own methods, their own version of the truth.

We built Pixlise to change that. It’s an open-source web application that allows scientists and researchers to analyse PIXL data in near real-time, collaboratively, while retaining the flexibility of their existing workflows. They can share insights, build on each other’s work, and move from raw data to published findings faster than ever before.

But here’s the thing. That outcome wasn’t inevitable. It happened because we had a dedicated UX research function embedded in the project from the start.

The researchers didn’t just sketch interfaces. They aggressively interviewed different members of the user base. Scientists, researchers, anyone with an interest in analysing rover data. They watched how people worked. They asked questions that engineers wouldn’t think to ask, because engineers are focused on what’s technically possible, not on what matches how scientists actually think.

What emerged from that research shaped the product in ways that wouldn’t have happened otherwise.

The charts had to make sense to users, not just to engineers. Technical accuracy wasn’t enough. The visualisations needed to match the mental models that planetary scientists already had. The way they think about spectrometry data, the comparisons they instinctively want to make.

The interaction between visual data and analysis tools had to feel natural. Scientists are looking at images from Mars alongside spectral data. The way those two things connect in the interface matters enormously. Get it wrong and you’re fighting the tool instead of using it.

The scripting and function-writing capabilities had to extend existing workflows, not replace them. Scientists weren’t going to abandon their Excel pipelines overnight. We needed to augment what they already did, not demand they start from scratch.

None of this would have been obvious from the requirements document. It came from research.

The result? Scientists felt ownership of the product. They’d been consulted, heard, and involved. They weren’t being handed something built for them. They were using something built with them.

And the measurable outcome speaks for itself. The time from data collection to published paper dropped from roughly a year to weeks. That’s not a marginal improvement. That’s a transformation in how science gets done.

Technical Feasibility Workshops: Where Design Meets Engineering

Early design involvement doesn’t mean designers go off and create something in isolation, then hand it to engineering. That’s just moving the handoff earlier. The real value comes when design and engineering work together from the start.

One practical way to do this is the technical feasibility workshop. A structured session early in a project where both disciplines surface constraints together.

When to run one: Before architecture decisions are locked. During discovery or definition phases, when you’re still figuring out what you’re building.

Who’s in the room: At minimum, an engineering lead and someone responsible for design or user research. Ideally also a product owner and, if at all possible, an actual user or customer.

What you cover:

1. User story walkthrough (designer-led). What is the user trying to accomplish? What does success look like for them? Where are the pain points today?

2. Technical constraint mapping (engineer-led). What are the hard constraints? What’s expensive or risky? What technical debt already exists that will shape what’s possible?

3. Overlap identification (collaborative). Where do user needs and technical constraints intersect? What’s both desirable and feasible? What trade-offs are we already facing?

4. Risk surfacing (collaborative). What happens if we get this wrong? Where are the assumptions we’re most uncertain about?

5. Decision points. What do we need to prototype or test to reduce uncertainty?

This isn’t a one-time event. Spotify describes their approach as shifting weights. Design and engineering don’t hand off to each other. They shift who’s leading at different phases while remaining in the room together throughout. The designer’s job isn’t just to create designs. It’s to facilitate getting to the best design outcome, which requires engineering input from the beginning.

GetYourGuide calls it “handshake instead of handoff.” The difference is subtle but significant. A handoff implies sequential work. Design finishes, then engineering starts. A handshake implies parallel collaboration. We’re working on this together, even when one of us is leading.

Collaborative Prototyping: Keeping the Feedback Loop Tight

The old model for design-engineering collaboration looked like a relay race. Designers did their work, documented it exhaustively, and threw it over the wall. Engineers picked it up, discovered a dozen reasons why it wouldn’t work as specified, and either built something different or sent it back for revision.

The new model keeps both disciplines in the room together.

Engineers in user research sessions. Not to solve problems, that’s not the point, but to watch and listen. When engineers see real users struggling with a specific interaction, it changes how they think about technical trade-offs. “We should refactor this flow” becomes a hard sell. “In eight out of ten sessions, users failed to complete this task without backtracking, here’s the video” is a different conversation.

Designers in technical spike reviews. When engineering is exploring whether something is feasible, designers should be there to understand what’s possible and what’s expensive. The best design solutions often emerge from understanding constraints, not ignoring them.

Shared prototypes that evolve together. Rather than a finished design spec that gets implemented, there’s a prototype that both sides iterate on. Designers adjust based on technical feedback. Engineers adjust based on user feedback. The artifact gets better faster.

Design systems as living contracts. A good design system isn’t just a library of components. It’s a shared agreement between design and engineering about how things work. When both sides contribute to it and reference it, there’s less ambiguity and less rework.

The contrast with projects that lack this collaboration is stark. I’ve worked on projects where there was no formal design process, where we just talked through requirements and started building. Every single time, the result was the same. Larger iterative cycles. Unclear expectations. And eventually a tense conversation where the customer’s vision turned out to be fundamentally different from what we’d built.

These days I refuse to work on projects that don’t have some element of design thinking baked in. Not because I’m precious about process. Because I’ve learned what it costs when you skip it.

“Engineers Can Figure It Out” (and Other Expensive Beliefs)

When I talk to skeptical engineering leaders about involving design earlier, I hear a few recurring objections.

“Engineers can figure it out these days.” There’s some truth here. Engineering skill sets have expanded. Many developers have better design intuition than their predecessors. Tools have improved. But there’s a difference between being able to make reasonable interface decisions and being skilled at understanding user needs. They’re different disciplines. An engineer can figure out a plausible solution. A designer with research can figure out the right one.

“AI can build it, design just slows us down.” This is increasingly common and it gets the causality backwards. AI removes coding friction. It makes implementation faster. But it doesn’t remove the need to know what to build. In fact, faster implementation makes upfront clarity more valuable, not less. If you can spin up a feature in hours instead of weeks, the bottleneck shifts entirely to knowing which feature to build. Good UX research gives developers a clear understanding of what needs to be built and how interactions should work. AI can still be leveraged for implementation. But research removes the computer-based assumptions from the development cycle.

“We don’t have time or budget for design.” This one I take personally because I’ve lived the alternative.

Early in my consultancy work I took on a project without any structured design process. We discussed requirements regularly. Had good rapport with the client. Thought we understood what they wanted. We were wrong. The gap between their expectations for the user interface and data flow and what we built only became clear late in the project. I made a financial loss on that engagement. Not because the code was bad, but because we built confidently in the wrong direction.

That experience taught me two things. First, verbal discussions aren’t enough. You need structured workshops, visual artifacts, and iterative feedback so the customer sees what’s being built and can flag misalignment early. Second, the “cost” of design isn’t additive. It’s preventive. You’re not spending extra. You’re avoiding waste.

The minimum viable design investment isn’t a full UX team and months of research. It’s someone asking “who is this for and how will they use it?” before any code gets written. It’s regular feedback sessions with real artifacts, not just status updates. It’s treating user understanding as an engineering requirement, not a nice-to-have.

Design as an Engineering Practice

The lesson from PIXL wasn’t that NASA has resources most teams don’t. It’s that treating design as foundational, as a structural part of how you build rather than a finishing touch, changes outcomes in ways you can measure.

Scientists publishing papers in weeks instead of a year. Users who feel ownership because they were consulted. Features that match how people actually work, not how engineers assumed they would work. These aren’t soft benefits. They’re the kind of outcomes engineering leaders care about.

If you’re skeptical, I’m not asking you to overhaul your process. I’m asking you to try an experiment. Before your next project kicks off, ask four questions:

1. Have we talked to the people who will use this?

2. Do we understand how they work today, not just what they say they want?

3. Have we surfaced technical constraints alongside user needs, in the same room, at the same time?

4. Is someone responsible for holding the user’s perspective throughout the project, not just at the beginning?

If the answer to any of those is no, you’re placing a bet. Maybe it’ll pay off. But the research, and my experience, suggest the odds aren’t in your favour.

When you’re building for Mars, you don’t get many chances to iterate. When you’re building for anyone else, you might get more chances. But each iteration is more expensive than getting it closer to right the first time. Design isn’t the opposite of moving fast. It’s how you avoid moving fast in the wrong direction.

In many organisations, a core issue is the persistent misalignment between business objectives and IT capabilities. Picture a business racing ahead so quickly that no one has time to tackle bigger-picture problems. Imagine, however, if those efforts were channeled toward creating shared customer value, not just managing internal efficiencies. IT often becomes a bottleneck, overwhelmed by competing requests and tasks, making prioritization nearly impossible. The result? Frustrating finger-pointing all around when that energy could be harnessed to enhance essential work and improve alignment across the organization. By focusing on a cohesive path to meet customer needs, the Trio model can transform frustration into collaborative success.

In several organisations I’ve worked in, requests stay in backlog for months due to a lack of prioritisation bandwidth. Frequent urgent escalations, often self-serving, derail progress and force immediate attention. Business teams bypass IT using shadow solutions, lacking confidence in IT delivery. Meanwhile, IT feels reduced to an order-taking role rather than an autonomous, decision-making organisation.

Not all organisations operate this way, but high-pressure environments can slip into these habits unnoticed. Recovering requires both awareness and courage to rebuild effective, aligned collaboration.

Why the Usual Fixes Fail

The core issue: business and tech are structured as separate entities that hand work back and forth, rather than solving problems together.

So, of course, what you then end up with is a list of things you’ve probably already tried:

1. Hiring more managers

2. Hiring more engineers

This approach doesn’t fix misalignment; it adds more people who lack context and increases salary costs without resolving core issues. Extra alignment meetings often waste engineering hours through ‘status theatre.’ Quantifying this lost productivity can underscore the urgent need for change.

Next comes the problem of ticketing and prioritisation. This helps some symptoms and increases visibility if managed properly. But it does not address the chaos itself.

You then add more alignment meetings with leaders, developers, product managers, and engineers. These meetings become status theatre, with participants seeking to outdo one another, leading to team-wide frustration and problems.

This repeated separation of business and tech creates recurring misalignment and unresolved problems instead of joint solutions.

The Trio Model - What It Is and What It Isn’t

Today I want to talk about the Trio model. For anyone who doesn’t understand what this is, it’s a collaboration pattern about three different parts of the organisation working together. In an engineering light organisation, you may end up having a slightly different trio than the general trio, but let’s take, for example, a light org where you might end up with:

• A business owner (this person would own the problem and the customer relationship)
• A technical lead who owns the feasibility and implementation (probably working as an IT engineer)
• A designer or an analyst or an ops lead (it sort of depends on the problem type)

These people are expected to work together to enable co-ownership of outcomes, not just a hand-off of tasks. The Trio should rally around customer-centric metrics such as customer satisfaction, net promoter score, time-to-market, or defect rate—tangible goals all Trio roles can collectively strive to improve, translating shared outcomes into concrete action.

Of course, the Trio isn’t a committee or approval layer. It’s not there to oversee but to bring the right people together before decisions are made—not after. It’s also not consensus-driven: three people don’t mean three votes. Instead, decision rights are explicitly assigned: the business owner has final say over customer priorities, the technical lead makes the call on feasibility and implementation, while the designer, analyst, or ops lead decides how to optimise the user experience, business process, or operations, depending on the issue. The tech lead cannot be vetoed by the business owner on technical feasibility, nor can the business owner be overruled on customer needs. Clear assignment of decision rights prevents gridlock and ensures each key area is covered. It is important to note that the trio is not the entire delivery team. They make directional and trade-off decisions—they do not execute all the work themselves. In a 10-15-person IT organisation, not everyone can be in a trio; otherwise, core work would stall.

Trios are not permanent—they form around defined problems and disband when those problems are resolved. Assigning people to trios without clear problems only recreates unnecessary committees, which this approach avoids.

A warning: ‘the trio owns it’ can quickly become ‘no one owns it.’ Each role still has distinct accountability. While there is a shared outcome, each person has specific responsibilities.

Making It Work: The Structural Stuff

So, of course, the question is: which problems actually warrant a TRIO, and which should just be dealt with in the usual manner? This normally falls into several categories.

Often, the projects warranting a TRIO are ones where you’ve either got high ambiguity, cross-functional dependencies, or strategic importance, or maybe a combination of all three.

What often doesn’t require a TRIO are well-defined requests, business-as-usual work, and pure technical debt that can be dealt with via a single stream.

Because what the leaders also need to figure out is how to prevent every business stakeholder from demanding their engineer in a TRIO, because that then ends up, as mentioned earlier, with a lot of people in TRIOs and no one to actually do the work.

Too many Trios spread technical people thin; too few create confusion and inertia. Balance isn’t a one-time fix. Leaders should make trade-off decisions a regular practice. A lightweight quarterly review of Trio assignments and resources helps sustain balance and builds leadership strength, ensuring strategic and operational goals are met.

Finding balance here is challenging.

The next question is about meeting cadence, but the real question here is authority. C-Suite execs tend not to really care about meeting frequency, except that it gives the optics of productivity without necessarily being productive. What they actually care about is: “Will this create more meetings that waste time or fewer escalations that waste their time?” By framing authority as a trust dividend, you can link decision autonomy to fewer escalations and faster learning loops. This perspective encourages executives to see granting authority as a strategic investment rather than a risk, fostering a more efficient and responsive organisation.

The answer depends on whether the Trio has the authority to decide. If every call goes up the chain for approval, the Trio becomes just another layer. The real cadence question is what the Trio can decide without asking permission, and how often they need to meet to exercise that authority. This could be daily, a few times a week, or weekly. The Trio’s priority matters, too. A long-running Trio may only need to meet once a quarter, and that can work.

As we talk about decision rights, this is the load-bearing section of making it work. Most organisational dysfunction stems from ambiguity. The business thinks that they set priorities. IT thinks they’re drowning in competing priorities with no clear ranking because everything comes in super urgent. IT thinks they’re empowered to push back on bad ideas. The business thinks it’s IT’s job to execute whatever ideas come in and that IT’s being obstructionist on purpose. No one knows who can say no to the CEO’s pet project. The trio model only works if you can explicitly answer these questions: Pose the ‘Who says no?’ test. Challenge your team to summarize their trio’s escalation rules in one tweet. For instance, ‘Only the Tech Lead can veto based on feasibility; Business Owner has final say on customer priorities.’ This brevity test often exposes lingering ambiguity that fuels dysfunction.

• Who can green-light work entering the trio’s scope?

• Who can kill or descope work that’s not feasible?

• What trade-offs can the trio make without escalation?

• What must escalate? (Things like budget thresholds, timeline changes, scope changes beyond X)

If you don’t answer these questions and make them prescriptive, you’ll end up with the same bottlenecks you have now, just with a new name.

Of course, this then gives you the ability to define success metrics. The real question is shared accountability. Because the classic setup measures IT on delivery velocity and system uptime, and businesses on revenue and customer outcomes, these can point in opposite directions. If the metrics aren’t shared, the trio becomes adversarial: the business owner pushes for speed, the tech lead pushes for quality, and the third person picks a side or checks out. Which is, of course, very suboptimal. A trio needs at least one shared outcome metric they’re all accountable for, not IT delivering on time and the business adopting it. But this problem got solved, and here’s how we’re measuring it as a team, as a trio that can then be reported up the chain to their C-suite execs.

What Makes It Actually Stick: The Leadership Part

This is the pivot where you tell them the uncomfortable truth: the structural stuff will fail if leadership behaviour doesn’t change. Be direct but not preachy:

• Stop sending “urgent” requests directly to IT that bypass the trios. Every time you do, you undermine the model.

• Stop letting business leaders commit to timelines without technical input. If sales promises a feature by Q3 without talking to engineering, you’ve already broken the system.

• Protect trio time. If your 10-person IT team is running 15 trios plus BAU support plus incidents, you don’t have a trio model - you have a burnout model.

• Hold business owners accountable for outcomes, not just engineers. If a project fails, the business sponsor should be in that post-mortem, too.

This is where you earn trust by not just telling them what to build, but what to stop doing.

Moving on to what actually makes this type of design pattern stick in an organisation, the real truth is that the structural stuff will fail if leadership behaviour doesn’t change. This is a big part of why many of these processes just don’t stand the test of time: leadership is the one unwilling to change, not the actual ground-level staff who implement these patterns.

As an organisation, you need to stop sending urgent requests directly to IT that bypass the trios. Every time you do, you undermine the model. If every request is urgent, there is no way for an IT team to prioritise requests.

Equally, the business needs to accept that sometimes deliveries will get pushed out because legitimately urgent requests will come in that need to be facilitated and actioned, and that will take time and have a knock-on impact on the delivery of other stuff.

We also need to stop letting business leaders commit to timelines without technical input because we’ve seen this time and again. If the sales team promises a feature by Q3 without talking to engineering, you’ve broken the system because there’s no input from the people doing the implementation. It’s just a guarantee that those people will actually adhere to those timelines to understand their existing workload and delivery schedule, and whether that is even feasible.

Also, it is important to protect Trio’s time. If your 10-person IT team is running 15 trios, plus BAU support, and an instance, you don’t have a trio model; you have a burnout model. (Trinkenreich et al., 2023) Everybody is knee-deep in trios trying to plan for the next bit of work, trying to understand what’s going on, and, of course, trying to do that whilst supporting existing systems and developing new ones is nigh on impossible just due to the nature of the scale of the requirements and the requests coming into that organisation.

The last point that actually makes this stick is holding business owners accountable for outcomes, not just engineers. If a project fails, the business sponsor should be in that post-mortem too because they are part of this chain. It is not purely an engineering function that causes the failure of a specific plan. It can be a lack of specification or accurate scoping, and it takes more than two to tango. Ensuring that the business is also held accountable for its side of the bargain is important when running post-mortems to understand where the failures actually occurred. If we all just always assumed that the business is right and the IT organisation is wrong, you will forever be in a spiral of finger-pointing, with the IT organisation taking the blame.

A Realistic Starting Point

As we reach the end of this post, it would be nice to offer some actionable pointers people can work with.

So what I’m saying to you is: pick one or two high-value, high-friction problem areas inside of your organisation and form trios there. Begin with the following steps: First, select and clearly define the problem areas where a Trio can bring substantial improvements. Second, assign clear roles to each Trio member, ensuring everyone understands their responsibilities and contributions. Third, establish specific metrics to measure success, such as customer satisfaction scores or time-to-market metrics. Finally, schedule a review after a quarter to assess the Trio’s performance, identify challenges, and make necessary adjustments. Run it for a quarter, see what breaks and iterate over it. It’s not an overnight change that’s going to suddenly transform your organisation, but it will enable more direct communication and allow multiple groups to take on responsibility for ensuring that requirements are defined and that everybody has a stake in the outcome. It’s not just chucked over a wall, and the IT team are expected to pick it up and run with it.

It’s also important to acknowledge that this is hard and takes longer than you’d like. It’s a difference between a real fix and another process that becomes what I call shelf-ware. We try something, we try it for a week, we decide that it doesn’t work, and then we chuck it on the shelf and pretend that we’ve never done it. We continue doing the same things over and over again, leading to significant employee dissatisfaction, churn, and project delivery failures. (Costa et al., 2024) But remember, real change is not easy, and it takes bravery to step out of the comfort zone. Let this be a call to courage for leaders—be the catalyst for change and sustain the momentum to truly transform your organisation.

Of course, it takes effort to make this work; it takes buy-in across the organisation. If you don’t get it, you’ll never succeed.

You spend what feels like weeks in sprint planning, daily standups, twice weekly standups for something else, roadmap review meetings, backlog grooming. And then someone else from the business says “well actually we need this other thing.....” and the team just feels like they’ve been blindsided with some random request and project managers feel like they’re throwing specs over the wall and hoping they land somewhere useful.

This feels like it happens at some many different organisations. All the effort going into planning to then find that people are none the wiser, or worse, have literally no idea whats going on in general. The problem here though isn’t missing rituals, its that the rituals we’ve put in place are optimised for the wrong thing.

Cross functional ceremonies, meetings whatever you want to call them are designed to move information or status updates, requirement hand offs, timeline syncs, functional information. Whats the word? Perfunctory: carried out with a minimum of effort or reflection. These meetings answer the “what” and the “when” but without building the rapport between teams, people and employees needed for “why does this matter to you?” or “what are you worried about?” These types of questions are really key to the world we live in. Sure timing, requirements, they’re all important but developers developing without understanding the bigger picture, the concerns that others outside the develop bubble have, will forever be on the back foot.

I can give you some examples. First up an example of a well run, if not draining standup. So when I worked at NASA JPL we worked on a piece of science software that was required to process data downlinked from NASA’s Perseverance Rover. For years, 2 or 3 at least I joined nightly standups because I was based in the UK and we had to get Los Angeles, Melbourne and London on the phone at the same time. So 11pm for 20-30 minutes every night was the only time that really worked. Of course this wasn’t the most productive of times because I’d already been working all day, but they were effective, we could hand work off, discuss blockers etc without it unduly impacting too many people.

Then after that I went to work for a medical startup. We had a team on the US East Coast and a team in India. This made life especially hard, trying to keep teams in the US functioning, information handed off and project managers happy, with the scrum masters in India trying to wrangle ill defined requirements and me sat somewhere in the middle trying to both get my own work done and answer questions on both sides of the pond. The problem was, they kept having daily standups but the transactions were very much pure business, no questions were asked, no real conversations were had and as such the business owners were frustrated and the developers jaded.

The 5 Dysfunctions

So now I must ask the question. Who’s heard of Lencioni’s Pyramid? Excellent book and topic of many other blogs. I’ll just add this one to the list! In 2002 Patrick Lencioni wrote about the 5 dysfunctions of a team. And it walks through the many issues and sticking points teams face as they see to grow together, in doing so Patrick looks at the causes of organisational politics and team failure. Something a lot of us can relate to.

The five dysfunctions are:

• Absence of Trust

• Fear of Conflict

• Lack of Commitment

• Avoidance of Accountability

• Inattention to Results

The issue with the way a lot of organisations work though is that they operate at the top levels of the pyramid. They operate with a lack of accountability and inatention to results, whilst assuming that the foundations, trust, fear of conflict and lack of commitment already exist. Of course in so many cases they don’t.

Over time though what you will learn is that if your rituals, those meetings you all feel are so important are missing the foundation layers you’re building on sand and the whole thing will crumble around you.

So what is the absence of trust? The absence of trust is the unwillingness to be vulnerable within the group or within the wider organisation, which leads to a lack of trust between your peers. Fear of conflict involves seeking artificial harmony over constructive passionate debate and not understanding when to both push back and hold your ground when other people are passionately arguing in a different manner.

Lack of commitment is around feigning buy-in for group decisions which creates ambiguity throughout the organisation. People flip-flop between different points of view depending on who they’re speaking to and cause greater work for those who are then empowered to implement the changes.

Avoidance of accountability involves ducking the responsibility to call peers and superiors on counterproductive behaviour which sets low standards. Therefore people further down the chain are expected to hold themselves to higher standards than the boss is implementing the standards themselves which is counterintuitive. Lastly the inattention to team results which focuses solely on personal success status and ego before team success which of course is detrimental to the team output as a whole.

Of course this begs the question of what do we do instead? How about starting a meeting between technology groups with Q&A? Not forced icebreakers but structured moments where admitting uncertainty is normalised. I’ll give you an example: how about an engineering lead opening a planning session with “here’s what I don’t understand about this quarter’s priorities” and inviting the PM to do the same?

Therefore the backwards and forwards normalises the uncertainty between the groups, both dictating what should be implemented and the people doing the implementation work themselves, reducing the friction and allowing everybody to voice their uncertainties as a group.

On top of rituals that surface vulnerability, how about rituals that practise disagreement? Teams that only agree in meetings and fight in Slack have a ritual problem. What does it look like to build productive friction into cadence? For example, how about a team that introduces a red team rotation into roadmap reviews deliberately asking the questions that would create the friction to ensure that it is discussed during the meeting rather than offline while grumbling into your Slack channel?

Finally how about rituals that build shared context, not just shared information? The difference between “Here’s the roadmap” and “Here’s the trade-off I struggled with and why I landed here”. For example a founder who replaced quarterly all-hands presentations with smaller cross-functional context sessions. It allows people to be able to discuss smaller chunks of work while also understanding the cause and reasoning behind those decisions being made, rather than a holistic super-high-level top-down view of work that’s supposed to be carried out across a broad swathe of the engineering function. We actually saw similar with the medical project we were working on that I mentioned earlier when we split up the groups into much smaller teams, allowing for better communication across different groups while still allowing the managers to track the broader project goals at a higher-level delivery.

Of course involved in all of this is the conflict resolution dividend. When rituals build trust and normalise disagreement, conflict resolution becomes cheaper. Teams don’t need to elaborate escalation paths because I’ve already practised navigating tension in lower stakes settings. So when a problem arises, the fallout and the ability to deal with it is far reduced.

Over the years I’ve worked in both types of environments, where engineering functions regularly get overwhelmed by both requests, feedback, and negativity and those that thrive.

Having worked in various startups in London over the years, we often saw small engineering teams with unrealistic goals and deadlines required to deliver on time. Then project managers would get annoyed when delivery was delayed for reasons that they couldn’t comprehend. Of course if you had dealt with this at a smaller team level focusing on individual pieces rather than the bigger picture, this may have been alleviated or remediated earlier.

Equally having explained it in an earlier in this blog, I’ve worked in many environments where small-scale conflict resolution was dealt with both effectively and efficiently by product managers, project managers, and engineering leaders. That makes a huge difference when it comes to delivering on time and to expectation because there’s less fear of retribution from within the engineering team should something crop up that would negatively impact the timelines involved.

Conclusion

I understand why there is pushback when it comes to dealing with relationship building rituals. They obviously feel soft, they don’t feel like you are contributing directly to the delivery of the product or project, and so it’s harder to justify a relationship building ritual in a planning document than it would be a sprint review or something of similar nature. Leaders from large corps, non engineering corps etc may see these meetings as indulgent but when you’re talking about a large swathe of employees, trying to navigate requirements, deliverables and an ever changing tech landscape I believe they are invaluable.

If you’re a C-suite reader reading this article, then the return on investment is real but it compounds slowly. You won’t see results in the next sprint. But when it comes to the next crisis or when your team has to bail the company out because they’re being asked to do something that wasn’t in the plans or wasn’t in the roadmap, that’s when you’ll see the real return on investment. When it comes to these relationship-building meetings.

The organisations that get this right don’t have more meetings. They have meetings that do different work.

As we sum this blog post up, the structure of your rituals reveals what you actually believe about how work gets done. If the ceremony is about outputs and timelines, you’re communicating that relationships are someone else’s problem and not yours. Over time as we’ve stated earlier in this blog post, that will lead to larger problems that’s trickier to solve in ever-shortening timelines. Of course the other negative impact as well is the happiness and contentment of your staff. That also has a detrimental impact on output as staff get less inclined to do the extra work or put in all the hours as they get frustrated with an ever-ending list of stuff that needs to be completed.

We start this blog post talking about the requirements for meetings and how those meetings are structured. The organisations that get this right don’t have more meetings. They have meetings that do different work and communicate the requirements and deliverables in a way that resonates more effectively with the staff at hand.

It doesn’t all have to be transactional. It can be very much a relationship-building exercise while still getting the same points across in less time.

So if you’re a C-suite executive talking to your product managers or project managers, just have a think about how to best structure those meetings. What you can do as an organisation to increase the output and performance while reducing the friction and the resentment with inside the organisation by improving the relationships between different teams and the dependencies on those teams. They will forever thank you for it as conflict and expectations become a much easier attribute to manage across the organisation.

Let me direct you to a Linkedin Post that caught my eye the other day because I thought it was a joke, then it became apparant in the comments that he wasn’t joking.

Now the argument that you should use “more modern tooling” because its hot is one that’s been raging since the dawn of time. But of course, why change what works? And if a VM and SSH works for you, and you have a small team, have at it!

So this then leads me on to Kubernetes, what is it and should you use it. Kubernetes span out of Google where their Borg platform was dealing with massive Google sized deployments on a daily basis. Now if that doesn’t tell you what Kubernetes is good at nothing will. But for those of us who like to hack around on things, Kubernetes basically takes Docker images and a lot of YAML descriptors to deploy your application, or applications into a cluster of machines that then mangages your container, your networking, ingress, egress, permissions etc for you. Sounds great right?

Well in a number of cases it is great. It can provide a lot of fault tollerance, auto scaling, auto healing, all the things your business may be interested in leveraging and some of that, but not all of it comes out of the box.

Downsides

Okay so what are the downsides of Kubernetes? Lets start on a downer and finish more positively. Its super complex. Sure every cloud vendor has their own hosted solution, and there are a number of vendors offering up support for their Kubernetes variant if you’d rather deploy it elsewhere, but that doesn’t mean its easy to wrap your head around. Azure attempt to use Azure App Service to wrap Kubernetes in something that makes it easier to deploy apps. AWS just launched AKS Auto Mode in an attempt to reduce the management burden. They all know its hard work. Cluster security, cluster upgrades etc etc they all take planning and forethought on top of managing your own pods. And if you’re self hosting them your host VM also needs patching, upgrades and remediations.

On top of this the majority of Kubernetes is driven by YAML in some form or other. Some times this is simple, often its not, but its also the way you manage it, how do you deal with change requests, version control, config drift, different people applying policies etc? All of this requires careful management and its not really a technical problem its more of a policy and people problem, do you have the bandwidth to manage it?

At smaller scales Kubernetes is just more expensive. This obviously manifests itself in a number of ways and isn’t just 1 number. For example the additional complexity means more man hours. But it also means you’ve got more nodes, a control plane (sure it may be “free” in a vendor but you’re paying for it somewhere), more infrastructure for ingress etc. It all adds up.

Slower developer loops are also a real issue. You can, if you’re a sadist, run Kubernetes on a local machine. But this still requires the same YAML setup, the same config tweaking, the same maintenance. It can be great to debug issues in the cluster but in reality its still hard to manage and certainly not developer friendly.

Lastly on the downside, the mirage of Kubernetes portability. Of course Kubernetes is designed somewhat to offer universal configuration across cloud providers, this often doesn’t really play out in practice. You still need to have a good understanding of your vendors networking, its ingress options, its available storage etc. When you deploy storage, do you need slow disks? Cheap disks? High performance disks? Maybe both, how do you pick them how do you configure your setup to use them? It all adds to the operational complexity of a Kubernetes deployment.

Upsides

Portability is obviously real. We can deploy docker containers into almost any environment, on prem, in any cloud environment. And whilst there is a degree of needing to understand the environment and what it provides the core product itself probably doesn’t need any changes making it easily deployable in a range of different environments. Of course the other thing about docker containers, developers can also just run them using Docker Desktop, Podman etc making them runnable across an number of different control services.

The other part of deploy Kubernetes processes is the declarative infrastructure. Whilst having everything defined as a YAML file can certainly add more control complexity it also is a positive. Once you’ve described the state you want, Kubernetes will do its best to keep everything in sync. Self healing, rollbacks on failure, consistency across different environments. That all comes from the bit that also makes Kubernetes hard to manage from a process perspective.

Of course in reality there is a rich ecosystem of additional services, products and people who can all plug into your Kubernetes infrastructure. Products like Argo can provide excellent application management. Prometheus provides out of the box support for monitoring of your processes and so on. There is an endless amount of additional tooling that can both help and hinder depending on what you’re trying to achieve.

Resource Isolation provides an element of multi tenancy or security patterns across your resources. Obviously this could be achieved outside of Kubernetes as well but RBAC roles, network boundaries etc all provided out of the box. Now of course, these can be hard to configure and maintain, also you want to hook them into your main security systems, probably easier to do inside of a cloud vendors system but still obscure, ensuring you’ve got namespaces scoped properly, maintaining that network firewall at the correct place etc. This is what I mean by the man management of the change requests, it only takes one badly changed YAML file to blow that right open, but when it works well, having all your processes on a single scalable cluster can really make managing Kubernetes clusters a breeze.

Next on the list of positives is the extensibility. If it can’t do what you want it to the operator framework, the ability to extend and improve the environment is absolutely possible. The APIs have been standardised for a long time and this is what drives that ecosystem we mentioned earlier.

Deployment patterns can really help you, of course some of these need additional tooling if you’re going to do them well but things like Rolling Updates, Canary and Blue Green releases are all absolutely doable out of the box. Of course though what you need to ask is, do you really need Blue Green releases? Or did you just read about them and figured they sounded cool?

Last on the list of positives is observability hooks. One of the biggest problems with running anything inside a container is, “whats going on inside?”. Hooks for logs, metrics, traces can really help here, both inside your containers but also the traffic through and across the platform. The ecosystem across the observability sector has really grown over the past decade, standards like OpenZipkin(founded by my friend from back in the JClouds days Adrian Cole) have really aided this drive.

So how do I choose?

Now this is the million dollar(in some cases actually million dollar) question. What is it you’re trying to achieve? Can you achieve the same outcomes by doing less? If the answer to this question is “yes”, then do less. I honestly believe in the modern tech landscape if you can do something by spinning up a VM and running a process on there, even if that thing sits inside a docker container for convenience, then it probably should. Of course there are times that doesn’t hold true. But even if its not true, do you still need a Kubernetes cluster? Or will something like Lambda for backend, ECS for frontend, Google Cloud Run, or Azure App Services work just as well?

In the 21st century there are multiple ways to spin this cat. Even if you’re looking for portability, would it cost more to package your app as a Docker container and then write separate deployment scripts for each cloud providers bog standard application deployment?

These are the types of questions you need to ask, not just the cost in running the thing, people think instantly to hosting costs, but when you’ve got a platform as complex as Kubernetes, whats the human cost, both monetarily and in terms of raw time.

Kubernetes is a powerful beast, in the right places it makes a lot of sense. But in the wrong places it can be a security, maintenance and management disaster waiting to happen.

Crawlers are often designed for one task, crawling websites, crawling a specific target, and I wanted to create something that would serve more than one purpose. There is always a lot of replicated code when building separate crawlers and I wanted to remove as much of that as possible whilst also bringing a standardised output format for all the sites we crawled to make the ETL process as easy as possible.

This is why the distributed crawler was born. It allowed us to crawl both Telegram and the Youtube Data API without having to rebuild huge chunks of very similar crawler logic across both platforms. I also wanted to be able to support multiple cloud backends, quite extensible configuration properties for the researchers who were going to use it, and it have as small a footprint as possible.

I started writing the Telegram portion of the crawler, as ever building and testing something that required many SIM cards to operate was an interesting challenge. With a Sim farm in my office, I succeeded in connecting the crawler to multiple Telegram API backends using a distributed technique, allowing for quite effective scalability of the platform. Of course, ensuring that the data you collect is both accurate and complete is an interesting challenge when you’re trying to crawl multiple channels inside of a Telegram account to create a global picture of what’s happening on the Telegram network. In doing that, it involved an awful lot of manual checking, check points, and check sums to ensure that the platform itself curated the data it needed for the researchers to be able to do what was important to them.

The other interesting challenge was collating the multimedia that is obviously synonymous with every social media network these days. That involved us looking for files, downloading different file types, processing them, and extracting numerous extra data using OCR techniques and metadata extraction on the files themselves to be able to provide search context and lookups for researchers when looking at specific topics that are embedded in the multimedia themselves.

Once I built out this framework, I realised that I wanted to make it pluggable to add the YouTube Data API. At that point, I leveraged Claude Code to help me refactor the existing codebase to a more generic interface that would allow me to plug in different social media endpoints. This was quite effective and it provided a useful entry point for Claude to then write the YouTube API layer leveraging the Golang libraries that YouTube had provided. The YouTube layer was easier to implement in a lot of respects in the fact that it didn’t require access to SIM cards. However, the YouTube Data API is quite restrictive in terms of both the data you can get and also the amount you can collect. Luckily, though, I was working for Princeton University, and so we made a valid submission to YouTube to allow access to their research API. This is basically exactly the same as the data API, but scaled up for larger volume data access. What you don’t get access to though with the data or the research API is the video content itself, nor access to the transcriptions for the video content (unless of course you happen to be the owner of that video). This means that you can analyse titles, descriptions, things like engagement metrics with comments and the like in the videos themselves, but you can’t actually access the content of the video itself. This is not immediately problematic, but could cause problems with people wanting to build upon this in the future. There is a workaround, and we will look to add this to the distributed crawler in the coming future. This is an open-source project called InnerTube. InnerTube leverages a public yet undocumented API that allows access to a number of different YouTube features and functionality that aren’t readily available inside of their data API. This does include transcriptions that we do not have access to, as well as things like similar videos and things that you would see in the sidebar. The reason for this is that people assume is that smart TVs and other smart devices need access to YouTube without a user being logged in, so they have to have a public API that is available to them. InnerTube cleverly leverages this to allow for it to be able to provide the same service as a library for other applications.

I mentioned earlier that I wanted to be able to use multiple different backends to be able to support the distributed crawler. The reason for this, of course, is that outside of running it inside of Azure inside of a Kubernetes cluster where we were going to run it, people might want to run this in a range of different places. Being able to leverage different backends was important:

1. Local mode so you could run it on your laptop (like a lot of people would do)

2. As sort of Azure CLI backend as an SDK back-end to support standard cloud operations

The other one that I implemented inside a distributed crawler was support for a framework called DAPA (a microservices framework from the Linux Foundation). The cool thing about DAPA is that it has a number of different plugins for a whole range of different storage features and functionality. Rather than me having to implement a number of different backends outside of the core ones I’ve just mentioned, it would allow for people to be able to use different services to store the data in depending on what they were interested in leveraging as their own backends. This gives users a very flexible deployment strategy when it comes to spinning this stuff up, especially in cloud environments and even more so inside of a Kubernetes deployment.

The other interesting thing with DAPR is that you could also use not just storage but other messaging protocols to be able to send the data around. So rather than persist it straight to disc you could for example send it to a message bus and send that to a different service to process that data in real-time. And so the idea of real-time data processing and streaming for a distributed crawler became a reality inside of the project because we could send that data directly to a message bus directly to Databricks and get it picked up and processed almost instantly.

The final piece of this puzzle was the use of a unified output format that I alluded to earlier. This format followed very closely to an output specification originally used by Junkopedia, and it is a standard output specification that they use across different social media sites in their service. It made sense for us to be able to mimic it closely so that we could also ingest their data should we need to, to be able to bolster the platform.

The idea was that every single social media post has basically the same content; it’s just the metadata that surrounds it is slightly different. So being able to support those different types whilst also making it easy for the platform to output similar formats would allow us to then be able to ingest that data very easily when it hit Databricks. And so regardless of which social media platform you use, you knew that the process to get that data into a readable format would be a lot shorter than if you were using individual crawlers for each different service.

And so that was basically the distributed crawler. The cool thing about it was I was allowed to open source it. You can find a link to the distributed crawler on GitHub right here. That is the platform to date. It supports a number of different backends as we’ve mentioned. It supports Telegram and YouTube, but we do have plans to add more to it over time. And make it more extensible and more useful for a whole range of different social media sites and probably also just standard website crawling as well into the future.

Gain that competitive edge, leverage what is available, but do so in the way that makes the most sense to your business.

Join me for my next live video in the app

Get more from Tom Barber in the Substack app

Available for iOS and Android

Get the app

Managing stakeholders in cloud architecture and application development is a lot like running a railway system: different passengers (stakeholders) have different destinations (goals), some demand speed (quick features), others want reliability (stability), and no one wants unexpected delays. Here’s how to align teams, manage expectations, and prevent scope creep without derailing a project.

Here’s some ideas:

1. Aligning Business and Technical Goals

2. Understanding the Customer Perspective

3. Bridging the Gap Between Technical Jargon and Business Speak

4. Defining Success Metrics Together

5. Prioritisation Through Feedback Loops

6. The Role of Transparency in Building Trust

7. Fostering a Culture of Continuous Improvement

8. Mitigating Risk Early

9. Celebrating Wins Together

10. Communication Tools and Practices

Find out more in the video!

1. Bringing Order to the Chaos

2. Performance Optimisation

3. Data Quality and Consistency

4. Interoperability with downstream systems

5. Data Lineage

6. Cross Team Collaboration

7. Facilitating Data Discovery

8. Future-proofing your architecture

9. Improving Query Accuracy

10. Hybrid Data Architectures

Data models continue to be an essential part of what we do. What other reasons can you think of?

Subscribe now

Subscribe now

Scalability is the capacity of your system to adapt to changes—whether it's a sudden surge in traffic or the gradual addition of new services and features. Applications face performance bottlenecks without scalability, leading to slow response times, downtime, and poor user experiences. It's about being prepared for success and future-proofing your architecture to handle shifts in user behaviour and business goals. Who remembers websites getting slashdotted regularly not that long ago? Getting 'slashdotted' refers to the sudden surge in traffic that a website receives after being featured on the technology news site Slashdot, which often overwhelms servers and causes the site to crash or become unresponsive.

In this post, we'll explore the core principles of designing cloud applications for scale, diving into best practices, patterns, and architectural considerations that help ensure your system can grow alongside your business. We'll look at leveraging cloud-native tools and approaches to maintain performance, reliability, and efficiency, no matter how your user base or data needs to expand.

Subscribe now

The Fundamentals of Scalability

Scalability in cloud applications involves designing your system to handle increased loads efficiently without compromising performance. Scalability comes in two forms:

1. Vertical Scalability (Scaling Up): Adding more power (CPU, memory, etc.) to an existing server.

2. Horizontal Scalability (Scaling Out): Adding more servers to distribute the load.

Horizontal scaling is generally preferred in cloud environments due to its cost efficiency and resilience benefits. Cloud providers like AWS, Azure, and Google Cloud simplify horizontal scaling through managed services, autoscaling features, and distributed architectures.

Vertical scalability, also known as scaling up, refers to adding more power (CPU, memory, etc.) to an existing server to improve its capacity. This approach can be simpler to implement since it doesn't require application architecture changes. However, it has inherent limitations because a single server can only be scaled up to a certain point before hardware limits are reached. Vertical scalability is often practical when an application has a single point of workload that cannot be easily distributed.

When deciding between vertical and horizontal scalability, it's essential to consider the nature of your application and growth patterns. Vertical scalability is suitable for smaller, less distributed systems where a quick boost in resources is needed. On the other hand, horizontal scalability is generally preferred for cloud-native applications that need to handle unpredictable and significant growth, as it allows for distributing the workload across multiple instances and reduces the risk of hitting resource limits.

Key Design Principles

Microservices Architecture

Designing applications with a microservices architecture enables different components to scale independently. Unlike monolithic architectures, where all features are tightly coupled, microservices break down functionality into more minor, isolated services that communicate via APIs. This allows you to scale specific application parts based on demand, optimizing resource usage.

Containers, particularly Kubernetes, provide an ideal platform for deploying microservices. Containers encapsulate each microservice with its dependencies, ensuring consistency across development, testing, and production environments. Kubernetes then orchestrates these containers, automating the deployment, scaling, and management of microservices, which makes it easier to maintain the independence of each service while ensuring the overall system scales smoothly as demand changes.

While microservices sound appealing due to their scalability and flexibility, they can also add significant complexity to your platform. Managing a distributed system with multiple microservices requires robust orchestration, monitoring, and communication strategies. Issues such as inter-service latency, network failures, and distributed data management can introduce challenges that require careful consideration. The increased complexity means that the benefits of microservices are often best realized when an application has reached a particular scale or level of complexity that justifies the additional overhead.

• Example: In an e-commerce platform, the checkout service can be scaled separately from the product listing service, ensuring peak traffic during sales events doesn't impact overall application performance.

Subscribe now

Statelessness

A stateless architecture simplifies scalability. Stateless components do not retain data between requests, making distributing requests across multiple instances easier. Leveraging stateless designs means that each server can handle any request, simplifying load balancing and reducing the chance of bottlenecks.

• Implementation Tip: Store session information in databases or distributed caching systems like Redis rather than in memory on individual servers.

Database Scalability

Scaling databases can be challenging, but several strategies can help:

• Sharding: Splitting your database into smaller, faster, and more manageable parts.

• Read Replicas: Read replicas offload read operations from the primary database, which is particularly useful for read-heavy applications.

• NoSQL Databases: Consider using NoSQL databases for extensive, unstructured data. They offer more effortless horizontal scalability compared to traditional SQL databases.

Cloud platforms provide managed database solutions such as Amazon RDS, Azure Cosmos DB, and Google Firestore, which have built-in scaling features to help handle large amounts of data and traffic.

Load Balancing

Load balancing is essential for distributing incoming traffic across multiple servers to ensure no single server becomes a bottleneck. Managed load balancers, like AWS Elastic Load Balancing (ELB) or Azure Load Balancer, are critical components for high availability and reliability.

Load balancing can also include geographical distribution for global applications and routing users to the nearest data centre to reduce latency and improve user experience.

Caching Strategies

Implementing effective caching can drastically improve application performance and reduce the load on your servers and databases. Use caching layers like CDNs (Content Delivery Networks), such as Cloudflare or Amazon CloudFront, to cache static content close to your users. For dynamic content, in-memory caches like Redis or Memcached help store frequently accessed data and reduce repeated database queries.

Autoscaling

One of the most significant advantages of cloud computing is the ability to scale resources up or down based on current demand automatically. Autoscaling ensures your application can handle traffic spikes while minimizing costs during low-traffic periods.

• Example: Use AWS Auto Scaling or Azure Virtual Machine Scale Sets to adjust the number of instances running your application dynamically.

Event-Driven Architectures

In event-driven architectures, services communicate via events rather than direct calls, which decouples components and improves scalability. Message brokers like AWS SNS/SQS, Azure Event Grid, or Apache Kafka are commonly used to manage the flow of information between services asynchronously, ensuring that your application can scale under varying loads.

Share

Choosing the Right Cloud Services

When designing for scale, managed services can offload much of the complexity of maintaining and scaling infrastructure.

• Managed Databases: Reduce the operational burden using managed database services that handle backups, scaling, and updates.

• Serverless Computing: You can execute code without managing servers with services like AWS Lambda or Azure Functions. Serverless computing automatically scales in response to incoming requests, ideal for applications with unpredictable workloads.

• Containerization: Using containers with Kubernetes or Amazon ECS allows for easy horizontal scaling and management of microservices, promoting efficient resource usage.

Designing for Failure

A core part of designing scalable cloud applications is accounting for failure. Cloud environments are inherently distributed, and failures are inevitable. Here are some practices to build resilience:

Redundancy

Introduce redundancy at every level—multiple servers, load balancers, database replicas, etc.- to ensure there are no single points of failure. Multi-AZ (Availability Zone) deployments are crucial for high availability.

Graceful Degradation

Design your application so that if a part fails, it degrades gracefully rather than crashing entirely. For example, if the recommendation service in your application goes down, the core features should still function.

Monitoring and Alerts

Use tools like AWS CloudWatch, Azure Monitor, or Prometheus to monitor application performance and set up alerts for unusual behaviour. This proactive approach helps maintain system health and preemptively addresses scaling issues.

Cost Optimization Strategies

Scaling can lead to increased costs, so it's vital to implement cost optimization strategies.

• Right-Sizing Resources: Continuously analyze the usage of your resources and adjust them to the appropriate size.

• Spot Instances: Use spot instances for non-critical workloads, which can significantly reduce costs.

• Serverless and Autoscaling: Pay only for what you use with serverless and autoscaling to avoid over-provisioning.

Real-World Example: Scaling an E-Commerce Application

Consider an e-commerce application experiencing fluctuating traffic, with significant spikes during holidays or sales. Here's how you might design it for scale:

1. Microservices: Separate the checkout, product catalogue, user management, and payment processing into distinct services.

2. Load Balancing and Autoscaling: Use load balancers to distribute user requests and set up autoscaling for the checkout and product catalogue services to manage spikes.

3. Database Strategy: Use read replicas for the product catalogue and shard the user data to improve performance.

4. Caching: Cache product images and descriptions in a CDN and use Redis for session storage to reduce database load.

5. Event-Driven: Implement an event-driven order processing system using message queues to handle payment and inventory updates automatically.

Challenges in Scaling Cloud Applications

Scaling cloud applications comes with its own set of challenges, including:

• Data Consistency: Maintaining data consistency across distributed components can be difficult. Strategies like eventual consistency and distributed transactions can help.

• Latency: As you scale globally, latency can become an issue. Techniques such as edge computing and data locality can help reduce the effects of latency.

• Complexity: Microservices, autoscaling, and distributed databases complicate development and operations. Proper tooling and monitoring are essential to manage this complexity effectively.

Subscribe now

Conclusion

Designing cloud applications for scale requires careful planning, leveraging cloud-native architectures, and making the most of managed services to handle infrastructure complexity. By breaking down monolithic applications into microservices, implementing stateless components, optimizing databases, and effectively using autoscaling, you can create an architecture that grows with your business needs.

Remember that scalability is not just about adding resources but optimizing your architecture to handle growth efficiently. By considering resilience, performance, and cost optimization, you can ensure your cloud application scales seamlessly while delivering a consistent user experience.

Subscribe now

From privacy and transparency to fairness and accountability, we dive deep into how tech professionals can create responsible AI that serves society ethically. Whether you’re a developer, data scientist, or just curious about AI’s impact on our world, this video is for you!

One of the biggest challenges, which isn’t new, is bias. Algorithms, trained on historical data, can inadvertently pick up and leverage societal prejudices, amplifying disparities in critical areas like hiring(Amazon!) healthcare and criminal justice. To help avoid this, teams need to work out how to incorporate diverse data sets and establish more rigorous checks during model training and testing. Building an interdisciplinary approach, inviting insights from social sciences, ethics experts and user feedback, can help uncover and address biases early.

Subscribe now

Data privacy, as touched upon with GDPR is also still very important. With growing attention paid from both regulators and the public, it is essential to prioritize user consent and transparency regarding data. Those of us at the forefront of this technological advancement should argue for clear data handling policies that protect user rights without degrading model performance. Building this level of trust requires going beyond compliance and working to foster a culture that values privacy at its core.

Finally, in this short article, accountability matters. Who is responsible when AI systems err or data misuse or misrepresentation results in harm? A clear governance framework is essential, detailing responsibilities and escalation procedures. Audits, regular and in-depth ethical reviews, and transparent reporting processes can help ensure that teams remain accountable, but only when implemented properly and executed effectively.

Ultimately, ethical AI and data-driven engineering are about proactive choices. By embedding these principles into every stage of our work, we can harness technology's transformative power while respecting and protecting the values that bind us as a society.

How has data sharing changed? Obviously, inside a business, self-service business intelligence, reporting, and analytics tools are far more commonplace than they were a couple of decades ago. Gone are the static reports and dashboards that we were used to in those days. Self-service or AI-generated analytics is far more commonplace in a business to drive business needs and answers on a near real-time basis.

Of course, it's not just internal to businesses that data is now shared. You can use many open datasets to do your own research and augment your existing datasets with additional information. Some are free and easily downloadable, while others are commercially available. Companies spend a lot of money yearly buying third-party data to make better business decisions.

Subscribe now

The other big change we've seen is the speed of connectivity. Data sets back in the day were limited by the amount of data that you could ship over the Internet, over a connection, or on written media. These days, with superfast connections around the globe, the locality of that data is less of a concern (except when it comes to the legality of using that data and the jurisdiction in which that data was created).

These days, with modern data legal frameworks like the GDPR and the user's right to be forgotten, companies have to pay more care and attention to the data they collect, process, and use to ensure that they don't violate GDPR compliance and similar data processing restrictions in other areas of the globe.

The Privacy Paradox: Balancing Transparency and Data Security

So, just what are GDPR and CCPA? The GDPR was brought in by the European Union, and it provides a number of securities for Internet users, including their right to be forgotten, along with who their data is shared with, and how it is shared.

The General Data Protection Regulation (GDPR) represents one of the most significant data protection and privacy shifts in modern history, certainly in the European Union. GDPR started in the early days of data regulation within the EU, beginning with the Data Protection Directive of 1995.

The Data Protection Directive was part of the European Union's early efforts to regulate how personal data was handled and shared, establishing a baseline for data privacy across member states. However, technology's rapid advancements and the growth of the digital economy quickly outpaced these regulations. Does that sound familiar?

Initial proposals for the GDPR began in 2012 with a regulation that would enforce uniform data protection across all EU member states. After four years of negotiations and discussions between stakeholders, legislators, and lobbyists, the GDPR was finally enacted in 2016.

The regulation was designed to empower European Union citizens with control over their personal data, mandate transparency in data handling, and establish hefty fines for non-compliance. We've seen a few; British Airways was fined 20m GBP in 2020 for a data breach.

So here are five essential points businesses need to be aware of to ensure GDPR compliance:

• Lawful Basis for Data Processing: Businesses must establish a legal basis for collecting and processing personal data, such as consent, contractual necessity, or legitimate interest. Without a valid legal basis, data processing activities are non-compliant.

• Data Subject Rights: GDPR grants individuals specific rights, including the right to access, correct, delete, and restrict the processing of their data. Businesses must have mechanisms to address and respond to these requests promptly.

• Data Protection Impact Assessments (DPIAs): Businesses are required to conduct DPIAs for high-risk data processing activities to identify and mitigate privacy risks. DPIAs are crucial for evaluating the impact on individuals' privacy before launching new data-driven projects.

• Breach Notification Requirements: In the event of a data breach, businesses must notify the relevant supervisory authority within 72 hours if there is a risk to individuals' rights and freedoms. Notifications to affected individuals may also be necessary under certain conditions.

• Accountability and Documentation: GDPR mandates that businesses maintain comprehensive records of their data processing activities. This documentation must include details about data types, processing purposes, retention periods, and data-sharing practices and be available for auditing.

The GDPR isn't the only legislation dealing with data privacy. There is also the CCPA (California Consumer Privacy Act).

Share

The CCPA is slightly different, and here are some critical distinctions in geographic scope and applicability. The CCPA applies specifically to businesses operating in California (as the name suggests) or doing business with California residents. It also has criteria based on annual revenue, data processing volume, and revenue derived from data sales, making it more focused on larger businesses. Unlike the GDPR, the CCPA doesn't require a legal basis for data processing in the same way. Instead, it focuses on giving consumers control over their personal data's sale, allowing them to opt out of data sales rather than explicitly consenting to data collection or processing.

Regarding individual rights, the CCPA provides rights for California residents, such as the right to know what personal data is collected, the right to delete personal data, and the right to opt out of data sales. However, the CCPA doesn't offer a set of rights as extensive as the GDPR, especially regarding data portability and the objection to processing.

When it comes to fines, they are worlds apart. The GDPR offers penalties of up to 4% of the company's global annual revenue or €20 million, whichever is higher. These really make it obvious how strict the GDPR's emphasis on data protection is.

The CCPA, in comparison, has much lower fines and penalties, which are capped at $2500 per violation or $7500 for intentional violations.

A distinctive feature of the CCPA is its focus on data sales. It allows consumers to opt out of selling their personal information. It requires businesses to display a Do Not Sell My Personal Information link on their websites, if applicable. The GDPR, in comparison, does not specifically address the sale of data but instead regulates the processing of personal data more broadly, including storage, collection, and sharing, based on lawful processing requirements.

Interoperability and Standards: Breaking Down Data Silos

Another aspect of data sharing in the modern era that has made a massive difference is the open standards of data. Historically, these would have been CSV files or other text-based files for processing and transferring data. Of course, these days, there's much more emphasis on the ability to move data around in a binary fashion while still maintaining interoperability between systems.

Parquet tables are a primary example of this. They have a binary format that is readable by many different database systems and is becoming the de facto standard for data processing at volume in the modern era.

Of course, many other examples of similar situations exist, such as JSON with a valid JSON schema, XML with an XML schema, and textual base representations of data that must conform to a schema. But there are also other...

Several formats allow different programming languages, operating systems, and APIs to produce and consume data when transferring data over the wire. For example, Google's Protobuf format has been around for several years. It allows data serialization in several different programming languages while being read by different programming languages. There's also more modern technology for data-specific processing, like Apache Arrows, which allows for the serialization of data and in-memory analytics, as well as the serialization of data for processing and transfer over the wire again across platforms and operating systems.

Of course, one of the things that drove this was the advent of business use cases for open-source technology. Twenty years ago, access to open-source technology was in its infancy, and so people used commercial vendors and competing file formats and storage formats. These days, access to open-source technology via the Apache Software Foundation or just getting it off GitHub makes it far more viable for businesses to leverage open formats.

Subscribe now

The Role of Data Sharing in Driving Innovation and Policy

Open datasets and open data can really drive forward innovation and policy making. One of the projects I'm currently working on is collecting social media data to help organizations and governments make better policy decisions regarding social media platforms.

Open datasets and leveraging these datasets to augment existing data that businesses, organizations, and governments may have to allow for a better understanding of the wider environment and the information environment we live in today. It allows for better decision-making when it comes to targeting customers, government policy, or how your local authority decides which day to collect your bin!

Of course, open standards also play a part in this. Different organizations that would traditionally have competed at a technological level now have to drive innovation to attract customers because the open standards that they are built upon mean that there is a level playing field when it comes to the features and functionality that the underlying data formats may adhere to.

Future Trends in Data Sharing: AI, Decentralization, and Ethical Frameworks

As we look ahead to how trends are shaping the future of data sharing, there is the obvious elephant in the room—AI. AI is currently driving an awful lot of innovation in the data sector. Still, it's not the only thing being driven from a technological advancement perspective. AI-driven insights, though, will become necessary. So the data storage formats and how data is accessed to allow LLMs to learn and offer insight will be key.

Blockchain may have passed us by in terms of crypto in some respects, but its use is definitely a unique technology. In the not-too-distant future, we will eventually reach the plateau of productivity, where we can leverage an immutable chain to store, process, and transfer data in an open fashion.

Finally, as we collect more data and understand more about the environment around us, the ethical frameworks and the way that people prioritize responsible data use will shift. They have shifted and will continue to shift as we move further into a data-driven society where so much of what we do is driven by the day-to-day collection, understanding, and dissemination of data across businesses and organizations around the globe.

Subscribe now

Cloud computing history

Today we'll examine how cloud computing enables seamless collaboration and research. But to do so, I want to take you back to the beginning—the beginning of computing, especially cloud computing—and examine how that laid the groundwork for what we consider distributed and seamless collaboration in computing today.

In the 1960s, the concept of 'utility computing' emerged. This was a visionary idea proposed by computer scientist J.C.R. Licklider, who envisioned a global computer network that would function like a utility, allowing users to access data and programs anywhere in the world. This concept, remarkably similar to the modern cloud computing model, laid the foundation for today's cloud computing.

One of the critical developments around this time was ARPANET, which appeared in 1969, the precursor to the Internet. Further on, in the 1970s and 80s, advancements in virtualization, operating systems, storage, and networking occurred. This allowed the World Wide Web, as we know it today, to be launched in 1991, connecting over a million machines to the Internet.

Subscribe now

Over the next few years, quite a lot happened regarding computing speed, how people interacted with computers, and the number of people who had computers. That led to the dawn of modern cloud computing. The late 1990s and the early 2000s saw the emergence and the start of cloud services as we know them today. Salesforce, actually, in 1999, pioneered the concept of delivering enterprise applications via a website, marking the arrival of software as a service; they were the first people out there.

In 2000, Amazon Web Services was conceived, providing the ability to launch some storage and computational tasks. In 2006, Amazon introduced S3, SQS and EC2, the first commercial cloud service, as we know it today, allowing companies and individuals to rent computers to run their applications. Now, it wasn't to say that you couldn't do stuff like this before, but most of it was FTP-based or renting physical machines somewhere else, and EC2 really became the forefront of modern cloud computing.

Following that, 2007 to 2015 saw rapid growth and diversification in cloud adoption and, you know, the entry of more tech players. Google introduced the Google App Engine, offering free plans and low-cost computing services, which, you know, again, drove many people to use cloud computing. 2010, Microsoft entered the cloud market with Azure, supporting web and mobile app development. In 2011, IBM launched its intelligent cloud services. During this time, cloud computing began revolutionizing business operations, offering benefits such as reduced capital cost, improved efficiency, and more manageable IT management.

From 2016 onwards, cloud computing has become the mainstay of most businesses' operations. Yeah, various companies still have on-prem deployments or things that go in more generic data centres, but these days, most companies either leverage cloud services or leverage SaaS services that sit inside of cloud services. There are, though, these days, hybrid and multi-cloud environments that have become more common. Edge computing has emerged as an extension of cloud services where we get the computer as close to the user as humanly possible while still retaining the flexibility of cloud computing. We look at AI and machine learning, which have been integrated into cloud platforms, and they have to be because the sheer scale of the compute that is required to do the processing necessitates a large number of clusters, a large number of servers sat in large clusters inside of a data centre.

COVID-19 and the pandemic accelerated cloud adoption across these industries because no one was in an office anymore or in a data centre to maintain the servers, and so that really sped things up.

Today, cloud computing underpins most technological disruptions and has repeatedly proven its resilience, scalability, and flexibility. While it may experience occasional downtime, it remains a reliable and integral part of the services you interact with daily. According to Gartner, public cloud spending is expected to exceed 45% of all enterprise IT spending by 2026, up from less than 17% in 2021, a testament to its practical benefits and reliability.

As we look to the future, cloud computing continues to evolve, with emerging trends such as serverless computing, edge computing, and quantum computing on the horizon. The cloud's transformative power in enabling digital transformation across industries is set to grow even further, shaping the future of technology and business operations. This evolution inspires optimism and excitement for the future of technology.

Share

Use cases over the years

I want to explore some real-life use cases where we've broken barriers with cloud computing and how it has advanced research. I first want to explore my experience after I started at NASA while working with a large genomics company.

This company had a lot of data, but it was all very much like on either researchers' laptops or shared drives that sat on generic infrastructure under people's desks. The problem with that is that users who want access to that data cannot see it, and when they get access to that data, they work in very siloed environments. They're not collaborating with other people. They're only interacting with other people once they get to a point where they want to be able to share their research so that other people can see it. That creates several problems when driving innovation and the research that underpins it.

So we came up with a plan where we were going to move these terabytes, maybe even petabytes of PDF documentation and Excel spreadsheets and reports, everything, up into AWS. We ordered a Snowball, one of those drives that you can get sent to your business, which you can then use to upload data more seamlessly, load it with data and then send it back to AWS to get it into S3. You have to bear in mind that this was about 10 years ago, and a lot has changed since then, but we got all this data into S3 and then looked at the tooling.

We want to make it all searchable. So, the first thing we had to do was index all of the PDF documentation, including everything that wasn't particularly indexable back then. Standards have changed quite a bit. Getting data out of PDFs in a structured format that would allow for a cohesive search was complex, so we built some tooling around the data extraction and got the research data in there. Then, we had to create user interfaces for the search.

We then had to build tooling to allow researchers to access that data and really start sharing it with each other. Now, it was in the cloud, and anyone in the business could access it.

Of course, that had a knock-on impact because we had to implement role-based security tied into their enterprise authentication mechanism. Cognito didn't exist then, so we had to roll our own. That took weeks. However, it allowed seamless access to all this data that the business otherwise had hidden on servers that were completely inaccessible to people. And it revolutionized how the company did its research going forward.

Next, I'd like to jump forward to almost the present day. For the last four years at NASA, I worked on an open-source tool called Pixelize.

Now, most people don’t know anything about X-ray spectroscopy. I don't know anything about X-ray spectroscopy, but we did write tooling that allowed a very distributed team access to not quite real-time data, but not far off it, as fresh as we could make it from NASA’s latest Mars Rover. They were distributed from various parts of Australia, North America, and Europe, literally spanning the globe.

We would get data coming down from the Mars rover, from the PIXL instrument on Mars 2020, NASA Perseverance, and we would stick that data through many different pipelines based mainly on Lambda functions at the time. So, we would process that data and then turn it into a different data structure that suited our application, which we built from the ground up, with a huge amount of input from our end users, the researchers. This allowed the researchers to access this data and make real-time decisions that would have taken 48 hours by the time you get the data, process it, and then analyze it with colleagues. We'd really turned it into something that allowed users to get that data within about 10 minutes of landing and run some extra processing over the top, which we'd optimized again multiple days down to a few minutes by implementing a bunch of parallel computing techniques.

That would allow people to share data in a Google Docs style but for research, which we'll touch upon later. They could really share that data, discuss it with other people, and then make a go-no-go decision on whether they would do more research in that spot or continue. This explores how cloud infrastructure breaks down geographic and institutional barriers, allowing researchers from different locations to work together more efficiently. It was unique and got a lot of attention internally at NASA.

Subscribe now

Distributed computing

A lot is being said about the return to the office and people working collaboratively now; Amazon is a case in point. I want to look at distributed computing; when I say distributed computing, I don't mean large Spark clusters. That's one aspect of distributing compute, being able to scale up and push the boundaries of the amount of compute you're using because you're leveraging the cloud.

The other aspect of distributed computing is that people are often distributed globally, especially in research. Collaborations between different universities are part of research; they allow people to distribute the cost and the time taken to do the tasks by leveraging different departments at different universities or different researchers at different universities, and these researchers can literally be in Europe and North America and Australasia and so on and so on because that also gives groups access to a range of different research funds and grant angles, which is a collaboration in itself!

This means that you do have to do distributed computing and develop novel ways for people to share their research and insight. One of the things that we make a lot of use of these days is cloud-based communications. Slack, Teams, and even Google Hangouts, if you have to. They all provide a service allowing distributed communication and access to files, people, and whiteboards. You name it, and you can do it. And so that has really been something that has been driven, especially post-COVID, in how people interact with other people online and the features and the functionality that the available software, often for free, now brings.

From a research perspective, the other aspect of distributed computing is collaborating on the data and research output. Collaborative document editing is one of the big things that have come along in the past decade. If you think of Google Docs, Google Sheets, or Office 365, and there are also open-source versions of this stuff that you can host on your own hardware, it really is mainstream these days and not just the purview of the big tech corps. In day-to-day life, people make a lot of use of these and may not even notice it. You may use Word on your laptop but be operating on a document that's sat inside of SharePoint, which someone else also opens and edits in real-time, so you can work on collaborative documentation, perhaps without even realizing it. This is, in essence, what I mean by distributed computing, allowing researchers to access data in a way that facilitates other people's insight, impacting the stuff they do daily.

How you do that often requires novel methods and different ways people think about how they will share that data and how they will interact with it. It's not to say that you can't do it; it's to say that you need to have the opportunity to think about that stuff and be able to build out software that's targeted specifically for that, to allow for people to really understand the power of distributed computing, and take them away from what they do currently on a day-to-day basis, and really propel them into a different way of going about their research, and their learning, and their understanding.

Leave a comment

Tailored Interfaces

This brings me to the crux of this topic, tailored interfaces. Most of what we do today is web-based regarding cloud computing and how we interact with systems. Behind the scenes, there might be a lot of processing going on, but often, the triggers and the stuff that actually gets it going is a web interface. It's a human clicking a button. It's a human writing a query. It's a human doing something. And so, researchers have had a period where they've often done the same thing day in, day out. And invariably, if you're a researcher with some technical background, you will probably use Jupyter Notebooks. You may use them locally. You may use them in Databricks or Azure Lakehouse, there are many different ways of interacting with notebooks. But quite often, that is the limit of the interaction with the data that you have because there's no interface to allow you to really deal with that data in any other way.

If I go back to the Pixlise project that I mentioned earlier, often the researchers would do some processing inside of a Jupyter Notebook, output that information to a CSV file, load it up inside of Excel, do some more processing, create some graphs, and then share that research. That is how they would do their research and collaboration by sharing Excel notebooks by email. One of the things we came across during the project was how we can generate or create user interfaces that push the boundaries of how researchers interact with their data. How can we create interfaces that allow researchers to access the data points they need without having to write a bunch of custom queries, write a whole export chain to Excel and then do some more processing? This is key because researchers, being researchers, have yet to really think about the ways that they could interact with the data other than inside of a notebook or inside of an Excel spreadsheet.

Part of our job as cloud evangilists is to open the eyes of researchers, allow them to see the possibilities out there, and ask what they think is an unachievable goal because that pushes the boundaries of distributed computing. It's what really pushes the limits of how we interact with the data stored inside the cloud and how we really unleash the power and capabilities out there. If everybody continues to interact with data similarly, they open up a notebook, even in the cloud, write some queries and then output a chart at the end; that's not innovation. That's just continuing the same process and pattern that people have done locally but in a remote cloud environment.

And in some cases, especially for smaller projects or more bespoke things, that's a good way of doing it. I'm not talking ill of the notebook environment, but if there's the budget and the scale and the requirement for research to be ongoing for a period of time, talk to those researchers and really understand how they want to interact with their data, what's important to them, and what interfaces they could possibly leverage that allow them to both reduce the complexity of what they're doing but also speed up their access to that data because, at the end of the day, writing queries is often a slow process as you work out what part of the query works and then roll it up and then combine it with another piece of query writing. And so allowing people to think big and think of the possibilities is super essential to allow us to break these silos and enable collaboration in research.

Subscribe now

Conclusion

I want to wrap this up by going through the different phases of how researchers could interact with their data. We've looked at the history of cloud computing. We've looked at some real-life examples of getting data out of local systems and into the cloud, which really fosters collaboration. And that's step one of this challenge, data gets into the cloud so that it can be shared easily.

If step two of that challenge allows people access via notebooks, that is fine. However, consider for a period of time if there is a better way for people to interact with that data and collaborate in real-time on that data. And if the tooling doesn't exist, sit down with researchers and really dig into what's important, what drives them, and how you can make their lives easier. Because we can break those silos.

We can really drive innovation and collaboration in the research environment. But, the researchers need to be able to drive that way of operating. And they must be stakeholders in the process.

If you would like to find out more we also have a case study over on our website.