# Guardian A blog by the Guardian's internal engineering team. We build and run the Guardian website, mobile apps, editorial tools, revenue products, advertising, and data and identity platforms. This blog is where we share our experiences and approaches, including software development tips, code examples, open source software and code stories behind product development This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Cooking up recipe data for the Feast app DevFeed: [Cooking up recipe data for the Feast app]() Original publisher: [Read original article]() Author: Frederick O'Brien Published: 2024-09-23T09:57:02Z Content type: article Language: en Sources: [Guardian]() Topics: [Structured-data](), [Database](), [Data Science](), [Machine learning](), [Content Management System](), [Composer]() Tags: [data](), [data-science](), [database](), [development](), [machine-learning]() ### AI overview The article explains how The Guardian built more than 4,000 structured recipes for its Feast cooking app. It describes the recipe data model, collaborative editorial and engineering work, machine-learning assistance, curation tooling, a database, and integration with the Composer content management system. ### Source excerpt Any dish worth its salt needs the finest ingredients. In the case of the new Feast cooking app, we needed a structured recipe archive. This is how we rustled it up You may have heard the Guardian has a new cooking app called Feast. With thousands of searchable recipes, cook mode, and lots of other lovely features, it's been well received on both iOS and Android. You should definitely get it. No article on its own could possibly do justice to the work that's gone into making Feast a reality (a big book might be able to pull it off) but this is a look at a vital piece of the app - its data. There's no recipe app without recipe data. At least, not one with any functionality worth writing home about. Structure old recipes with a machine-learning model trained in-house by the thoroughly brilliant data science team. Drop the resulting data into a temporary recipe curation tool inside which food journalists could use their expertise to refine and approve recipes. Put the curated data in a database. In parallel the same functionality could be added to the Guardian's bespoke content management system, Composer. When it was ready to assume recipe curation responsibilities, we could switch over. Continue reading... ## Day in the Life: Alex Guild DevFeed: [Day in the Life: Alex Guild]() Original publisher: [Read original article]() Author: Rasha Ardati Published: 2024-09-06T15:12:42Z Content type: article Language: en Sources: [Guardian]() Topics: [Product Management](), [Development](), [Mobile](), [App]() Tags: [app](), [career](), [culture](), [engineering](), [interview](), [mobile](), [product-management]() ### AI overview An interview with Alex Guild about beginning a product management career, developing a mobile app, and the routines, collaboration, and strategic work involved in a typical day at the Guardian. ### Source excerpt Welcome to our 'Day in the Life' blog series where we interview a colleague from the product and engineering department to reflect on their career and experiences at the Guardian How did you begin your product management journey? After university, I wasn't entirely sure what career path to pursue. So I decided to join an exciting startup that allowed me to explore various responsibilities and get exposure to different roles. At that point, I hadn't even heard of product management! However, I did really enjoy speaking directly with users, conducting interviews to understand their needs, and getting feedback on the products we were building. I was also quite curious about how all the different parts of the business came together to create a product. Continue reading... ## Parsing: the merit of strictly typed JSON DevFeed: [Parsing: the merit of strictly typed JSON]() Original publisher: [Read original article]() Author: Max Duval Published: 2024-07-26T16:05:58Z Content type: tutorial Language: en Sources: [Guardian]() Topics: [JSON](), [TypeScript](), [JavaScript](), [Parsing](), [Programming]() Tags: [javascript](), [json](), [parsing](), [rest-apis](), [runtime-errors](), [typescript]() ### AI overview This tutorial explains how loosely typed JSON.parse results can undermine TypeScript's static analysis. It recommends precisely describing and validating the shape of changing JSON responses so API or data-shape errors can be detected and handled more helpfully before they cause runtime failures. ### Source excerpt Dealing with content that keeps changing can lead to bugs on users' devices that are hard to reproduce. By precisely describing the shape of the content using types and ensuring it matches, exceptions can be avoided and users shown helpful messages when errors occur. This post will illustrate how this can be achieved with standard web technologies. The programming language of the web is JavaScript (JS), but its lack of static typing is the top pain point raised from the 2023 State of JS survey. We rely on TypeScript (TS) to get a strong structural type analysis of our code, as mentioned in our article on standardisation. Static typing helps prevent runtime errors on our users' machines. However, to ensure that no TypeError ever appears, we need to avoid the loose 'any' type, which is an escape hatch designed to opt-out of TS' static analysis. JavaScript Object Notation (JSON) is widely used as a data exchange format for REST APIs. Its syntax is compatible with JS, with only a subset of the language's primitives: objects, arrays, strings, numbers and nulls. Simple objects can be turned into a string with JSON.stringify and turned back into an object with JSON.parse, making it an adequate choice for many uses. We use it for powering our comments, new live blog updates, football match scores, weather, most read and other features of the website and Apps. Continue reading... ## Hack Day 2024: Sports and Election Prototypes DevFeed: [Hack Day 2024: Sports and Election Prototypes]() Original publisher: [Read original article]() Author: Rasha Ardati, Mariot Chauvin Published: 2024-06-17T14:42:54Z Content type: article Language: en Sources: [Guardian]() Topics: [Demo](), [Browser Extension](), [Programming](), [Structured-data]() Tags: [demo](), [extension](), [hack](), [programming]() ### AI overview Guardian colleagues held a two-day 2024 hack day themed around sports and upcoming elections. The article highlights selected concepts, including favicon-based liveblog displays, a browser extension that turns paragraphs into crosswords, and structured tennis scores for liveblogs. ### Source excerpt Recently, colleagues across the product and engineering department, as well as other parts of the organisation, came together for another exciting hack day For our first hack day of 2024 we kicked off with a grand theme around all the major events of the world, including sport and upcoming elections around the world. Hackers played around with the idea of using python programming language and real-time mobile telemetry data to influence the outcome of UK elections, they proposed a bold new journalistic geometry, added fun polls, quizzes and newsletters sign ups to the live blog, as well as posed the question: " Can bedrock do what CAPI can do?" And that's only 4 of the 19 amazing hacks produced and presented over the two-day event! A counter in the favicon for unread liveblog entries, which would appear on liveblogs. A more experimental demo that displayed a tiny version of the Guardian article being read entirely within the favicon. This included a Guardian header, real-time scrolling, all the article text, and images. Continue reading... ## When security matters: working with Qubes OS at the Guardian DevFeed: [When security matters: working with Qubes OS at the Guardian]() Original publisher: [Read original article]() Author: Philip McMahon Published: 2024-04-04T09:32:30Z Content type: article Language: en Sources: [Guardian]() Topics: [Security](), [saltstack](), [Debian](), [Infrastructure as code](), [configuration]() Tags: [amazon-s3](), [configuration](), [debian](), [infrastructure-as-code](), [os](), [packages](), [salt](), [secrets](), [security](), [workstation]() ### AI overview The Guardian describes its experience configuring Qubes OS for SecureDrop Workstation. The team used SaltStack to set up multiple workstations for safely handling sensitive documents, including an offline Debian 11 virtual machine, managed packages, an amnesic VM, an Amazon S3-hosted private package, a Nautilus extension, and secret configuration values. ### Source excerpt The latest version of the whistleblowing platform SecureDrop runs on the Qubes operating system. At the Guardian we used the Salt management engine to set up a Qubes environment where journalists could safely interrogate sensitive documents. This post was updated on 9th April 2024 following feedback from the Securedrop Workstation/Qubes community If you've ever scrolled through the Guardian homepage, you may have come across the 'Contact the Guardian securely' banner. This links to a page explaining how to share sensitive information with the Guardian in a number of different ways. Of these, the one that offers the best security is SecureDrop. An offline VM based on Debian 11; Some packages installed from the default repositories and a custom repository; The VM should be 'amnesic' - after a restart it should be reset to a consistent state (so any generated files/downloaded data should be deleted); A custom package installed from a private repository hosted in Amazon S3; A Nautilus extension; A configuration file containing secrets that can't be hard-coded. Continue reading... ## Pinboard: transforming communication across the newsroom (part 3 of 3) DevFeed: [Pinboard: transforming communication across the newsroom (part 3 of 3)]() Original publisher: [Read original article]() Author: Tom Richards Published: 2024-02-10T10:00:38Z Content type: article Language: en Sources: [Guardian]() Topics: [Tool](), [App](), [Tooling]() Tags: [developers](), [tool](), [tooling](), [tools]() ### AI overview The third part of a series examines the development and rollout of Pinboard, a discussion and asset-sharing tool used across the newsroom. It discusses the challenges of defining its MVP, building a feature-rich product to reach critical mass, and encouraging adoption through prototypes, demos, feedback, and production trials. ### Source excerpt Pinboard is a discussion and asset sharing tool (or rather tool within other tools) which is gradually transforming how the news room communicate as news stories move through the various phases of the production process This is part three of three (part one and part two). Continue reading... ## Pinboard: transforming communication across the newsroom (part 2 of 3) DevFeed: [Pinboard: transforming communication across the newsroom (part 2 of 3)]() Original publisher: [Read original article]() Author: Tom Richards Published: 2024-02-09T10:00:38Z Content type: article Language: en Sources: [Guardian]() Topics: [Tool](), [Angular](), [Amazon API Gateway](), [React](), [Development](), [JavaScript]() Tags: [angular](), [api](), [browser](), [development](), [javascript](), [react](), [tool]() ### AI overview This article explains how Pinboard, an internal discussion and asset-sharing tool at The Guardian, is integrated across newsroom tools. It describes why the project was not implemented as a browser extension or shared library, and outlines its script-based loading, authentication, and API Gateway integration. ### Source excerpt Pinboard is a discussion and asset sharing tool (or rather tool within other tools) which is gradually transforming how the news room communicate as news stories move through the various phases of the production process This is part two of three (part one and part three). Continue reading... ## Pinboard: transforming communication across the newsroom (part 1 of 3) DevFeed: [Pinboard: transforming communication across the newsroom (part 1 of 3)]() Original publisher: [Read original article]() Author: Tom Richards (with contributions from Ana Pradas) Published: 2024-02-08T15:55:25Z Content type: article Language: en Sources: [Guardian]() Topics: [Development](), [Tooling](), [Content Management System]() Tags: [cms](), [development](), [interfaces](), [tooling]() ### AI overview The first part of a three-part account of Pinboard, a discussion and asset-sharing tool integrated into newsroom production systems. It describes the project's origins, discovery research across editorial roles and offices, prototyping, and the design constraints identified before beta testing. ### Source excerpt Pinboard is a discussion and asset sharing tool (or rather tool within other tools) which is gradually transforming how the news room communicate as news stories move through the various phases of the production process This is part one of three (part two and part three). Who works on a story? How many people does it take to put a story together? How do they talk to one another? Does a person's role make a difference in their struggles with assets? Do people work differently in different departments? Do people work differently in each office? Not creating yet another thing they need to look at elsewhere. Being integrated with the tools they are already using was important. Each desk is a different world. They work at different paces, with different rota shifts, but at the same time, there is no such thing as an isolated desk. We initially tested with sports liveblogs and the Global Development desk and although the feedback was promising and helpful, we quickly learned that launching to a single desk at a time wasn't going to work. We really wanted to avoid people using 'noted-out article text' at the top of Composer (our content management system) for communicating and exchanging assets. We needed to design something that could work integrated in tools with very different interfaces, purposes and users. We were going to start with Composer (our CMS) and Grid (our picture system) to begin with, but would need to consider potentially becoming available in other tools without requiring a major redesign. It needed to work for people working on small laptops from home and for people with big screens in the office. Screen real estate is already tricky in tools. Continue reading... ## A developer's first year in the Guardian Digital Fellowship DevFeed: [A developer's first year in the Guardian Digital Fellowship]() Original publisher: [Read original article]() Author: Guardian Staff Published: 2024-02-05T16:24:50Z Content type: opinion Language: en Sources: [Guardian]() Topics: [Learning](), [AWS Lambda](), [Code](), [Framework](), [Scala](), [API](), [React]() Tags: [api](), [aws-lambda](), [code](), [continuous-deployment](), [developer](), [framework](), [information](), [learn](), [react](), [scala]() ### AI overview A Guardian Digital Fellowship participant describes his first year learning software development through real editorial technology projects, workshops, team rotations, and production deployments. He discusses working with AWS Lambda, Scala, the Guardian API, data visualisations, and React while receiving support from teammates. ### Source excerpt A developer in the Fellowship Scheme shares his experiences of his first year at the Guardian I clearly remember my early days in the Guardian's Digital Fellowship as a mixture of excitement and being totally overwhelmed. I had joined the editorial tools team, which builds all our digital tools for our journalists. At morning stand-up people were mentioning more acronyms than I thought I could ever remember. Like many of the fellows from previous years I don't have a computer science degree, or even a STEM background, and getting thrown into a department with more than 80 engineers can be intimidating! Fast forward a couple of months I had written my first AWS lambda, contributed to various projects and was regularly deploying production code unsupervised. The value of the fellowship lies in the fact that while you are given the time to develop your skills independently - you might choose to attend workshops run by other members of the department, learn a new framework and give a talk about it or work on a personal project and present it on the amazing Hack Day - from day one you are treated as an integral part of your team. That means that you get your hands dirty on real projects, learning and improving at an incredible pace. Continue reading... ## Large language models and generative AI: a recent hack day DevFeed: [Large language models and generative AI: a recent hack day]() Original publisher: [Read original article]() Author: Rasha Ardati Published: 2023-12-22T11:02:35Z Content type: article Language: en Sources: [Guardian]() Topics: [Large Language Model](), [Generative AI](), [Hackathon](), [Development](), [AI Development](), [ChatGPT](), [Google]() Tags: [ai](), [applications](), [chatgpt](), [data](), [development](), [encryption](), [experiments](), [generative-ai](), [google](), [hackathon](), [large-language-models](), [product]() ### AI overview The Guardian's Product and Engineering department held a two-day hackathon in which developers and data scientists explored applications of large language models and generative AI. The 24 projects included ideas for podcasting, search, image generation, cultural recommendations, documentary showcasing, productivity experiments, and a generative AI screensaver. The article highlights three award-winning hacks, including an iOS multiplayer crossword app, an audio summary tool using OpenAI GPT-3.5, and a method for helping label training data. ### Source excerpt Developers in the Product and Engineering department came together with colleagues from across the Guardian to explore the potential of LLMs and more The discussion of large language models (LLMs) and generative artificial intelligence was everywhere in 2023 - not least in the Guardian's Product and Engineering department. Hack days are a staple part of the software development culture, so it was no surprise that at this year's final hackathon, several developers and data scientists focused their attention in this area - covering potential applications in podcasting, search and image generation. And who wouldn't want a browser extension that assesses the mood of a news article and finds an appropriate music track to enhance your reading experience? In total, the teams, including colleagues from across the Guardian, produced and presented 24 hacks in the course of the two-day event - some of which took an entirely different turn. These included: a product that spits out cultural recommendations; a dedicated platform for showcasing Guardian documentaries; experiments with ChatGPT and Google Bard involving tools like Trello to improve efficiency; and a Guardian-themed generative AI screensaver. Continue reading... ## How the Guardian improved Typerighter, its style-guide checking tool DevFeed: [How the Guardian improved Typerighter, its style-guide checking tool]() Original publisher: [Read original article]() Author: Rhys Mills Published: 2023-12-11T15:12:52Z Content type: article Language: en Sources: [Guardian]() Topics: [Tool](), [Content Management System](), [Composer](), [Scala](), [Web app](), [Library](), [XML]() Tags: [articles](), [content](), [library](), [quality](), [quality-control](), [regex](), [scala](), [tool](), [web-app](), [xml]() ### AI overview The Guardian describes improvements to Typerighter, a style-guide checking tool integrated with its Composer content management system. The system uses curated regular expressions and XML-based LanguageTool rules to identify potential spelling and grammar problems, and the article explains why expanding its use was important for live blogs. ### Source excerpt How we improved Typerighter, the Guardian's style guide checker, to provide more value for the Guardian and its readers Minimising typos is important to a newspaper. Publishing messy prose might undermine trust in our overall quality control - why trust our political reporting if we can't spell Thérèse Coffey's name? We want to provide accurate information, and typos can mutate meaning - a single letter can transform a fiend to a friend and a bowl to a bowel. Worse, a missing word can turn a holy text into a wicked one. The Guardian is in the business of words, and we're keen to avoid "idiosyncratic" spelling decisions. That's why we built Typerighter, a tool that offers recommendations from our style guide into our content management system - Composer - and highlights errors well before the articles get in front of our readers. Continue reading... ## Who said what: using machine learning to correctly attribute quotes DevFeed: [Who said what: using machine learning to correctly attribute quotes]() Original publisher: [Read original article]() Author: Michel Schammel, Alice Morris, Anna Vissens, Paul Nathan, Alicja Polanska and Tara Tahseen Published: 2023-11-21T11:11:31Z Content type: tutorial Language: en Sources: [Guardian]() Topics: [Machine Learning & Artificial Intelligence](), [Deep learning](), [data]() Tags: [data](), [deep-learning](), [developer](), [engineering](), [machine-learning]() ### AI overview The Guardian's data scientists describe a collaboration with PhD students from University College London to train a machine learning model for attributing quotes to speakers. The article explains how coreference resolution links mentions in text to the entities they refer to. ### Source excerpt Today's blog does not come to you from any developer in product and engineering but from our talented colleagues in data and insight Here, the Guardian's data scientists share how they have teamed up with PhD students from University College London to train a machine learning model to accurately attribute quotes. Below the two teams explain how they've been teaching a machine to understand "who said what?" Sarah enjoys a nice cup of tea in the morning. She likes it with milk. Extract a series of mentions which relate to real-world entities For each mention, compute a set of features Based on those features, find the most likely antecedent for each mention Continue reading... ## Standardisation: the merit of consistent patterns DevFeed: [Standardisation: the merit of consistent patterns]() Original publisher: [Read original article]() Author: Max Duval Published: 2023-10-24T10:18:33Z Content type: opinion Language: en Sources: [Guardian]() Topics: [Programming](), [Code](), [CI/CD](), [Pull Request](), [Deployment](), [Documentation]() Tags: [cognitive-load](), [consistency](), [continuous-deployment](), [continuous-integration](), [deployment](), [development](), [documentation](), [engineering](), [programming](), [pull-request]() ### AI overview The article explains how standardised patterns, documentation, code organisation, language learning, testing, code review, and automated deployment help Guardian developers focus on tasks and reduce extraneous cognitive load across complex systems. ### Source excerpt Coming back from a long holiday? Recently joined the department? Collaborating on a complex, cross-team problem that spans different tech stacks and programming languages? At one point, we're all going to find ourselves in one of these situations. So how can we ensure that no one feels overwhelmed or unable to make meaningful contributions? Short answer: by enabling everyone to quickly focus on the task at hand. By "focus" I simply mean deciding what is in scope and what is not. For a company that's been around for more than 200 years, and online for an eighth of that time, a proliferation of systems is bound to happen. No single person is ever going to grasp the complexity in its entirety. Zooming out, we create architectural diagrams to help us understand how the different parts fit together; and zooming in, we write code that requires as little context as necessary. discuss the benefits of standardisation; describe which tools we have created help keep consistency; offer some thoughts on balancing familiarity and innovation. Continue reading... ## How the Guardian's Supporter Revenue engineering team evolved from 2016 to 2023 DevFeed: [How the Guardian's Supporter Revenue engineering team evolved from 2016 to 2023]() Original publisher: [Read original article]() Author: Paul Brown (Head of Engineering) Published: 2023-09-12T16:22:53Z Content type: article Language: en Sources: [Guardian]() Topics: [Development](), [engineering-culture](), [Website](), [stripe]() Tags: [developers](), [development](), [engineering](), [engineering-culture](), [product](), [stripe]() ### AI overview The article recounts how the Guardian's Supporter Revenue engineering team evolved from a small Membership and Subscriptions group in 2016. It describes changes to Guardian support offerings, website subdomains, and projects including Zebra, Giraffe, and Stripe-based payment experiments. ### Source excerpt Since you're here... please find somewhere comfortable to sit and read my story about the journey the Supporter Revenue team has undergone since I joined them as an Engineering Manager back in January 2016 Our readers' support is vital to the Guardian's future, and the Supporter Revenue engineering team is responsible for developing digital experiences that engage our valued supporters. We live our company values by developing in the open, using best practice technologies as much as possible, and upholding our engineering culture while we work hard to ensure digital experiences remain usable, accessible and render quickly. Back at the start of 2016, we were a small team inside a department called Digital Development, and we had a very utilitarian name: Membership and Subscriptions. Working alongside me was an agile coach, a product manager, a graphic designer, and four amazing software developers - two of which are still here but working in different areas. We all had the job of looking after two Guardian website subdomains: membership.theguardian.com and subscribe.theguardian.com. Both of these microsites have changed quite a bit: we've since closed down our membership scheme, subsumed newspaper and digital subscriptions into a "support proposition" - to give you an insight into some Guardian-speak. Continue reading... ## Aurora Serverless - a migration story DevFeed: [Aurora Serverless - a migration story]() Original publisher: [Read original article]() Author: Philip McMahon Published: 2023-08-16T08:46:04Z Content type: article Language: en Sources: [Guardian]() Topics: [migration](), [Serverless](), [Amazon Web Services](), [Databases](), [App]() Tags: [aws](), [database](), [databases](), [migration](), [pitfalls](), [serverless]() ### AI overview The Guardian describes its migration from MongoDB to PostgreSQL and its evaluation of AWS serverless relational databases for Lurch, an internal tool that searches public data sources. The article discusses database selection, read performance, full-text search, indexing, and challenges caused by ingesting large historical datasets. ### Source excerpt On our team we swapped databases 6 times in a year. We've landed on Aurora Serverless V2 - was it worth it? Since our migration from Mongo to Postgres in 2018, at the Guardian the search for the perfect database hasn't stopped. In the last year on the Investigations and Reporting team we've spent a lot of time trying out AWS's Serverless relational databases. We hope that by sharing our learnings we'll help others avoid the pitfalls we hit along the way. Continue reading... ## Day in the Life: Jessica Leivers DevFeed: [Day in the Life: Jessica Leivers]() Original publisher: [Read original article]() Author: Rasha Ardati Published: 2023-07-25T15:41:34Z Content type: article Language: en Sources: [Guardian]() Topics: [Development](), [Software Engineering](), [data-engineering](), [Tech Lead](), [Job]() Tags: [career](), [developer](), [development](), [engineering](), [individual-contributor](), [interview](), [leadership](), [mentoring](), [software-development](), [software-engineering](), [tech-lead]() ### AI overview An interview with Jessica Leivers about her path from early interest in computers to software and data engineering, volunteering and mentoring, and her transition from individual contributor to engineering manager at the Guardian. ### Source excerpt Welcome to our 'Day in the Life' blog series where we interview a colleague from the product and engineering department to reflect on their career and experiences at the Guardian Did you always have an interest in software development from a young age? I definitely had an interest in computers but I didn't know what software development was. I remember being very small when my dad got me a BBC Model B computer and then when I was a bit older we got a family Commodore Amiga A500. I used it to play Superfrog and Lemmings but my dad, who was a mechanical engineer by profession, had become interested in software development. Luckily for me he had built up a small home library of software and web development books that I could dip into as a teenager when I first became interested in making a website. Unfortunately, at university my dissertation supervisor put me off software engineering by warning me "you should be careful, you might end up a developer!" leading me to believe this would be an awful career. Thankfully, in my 20s I ended up in software development anyway and this warning couldn't have been more false. I was careful; I ended up as a developer; and it is great! Continue reading... ## Product and Engineering away day 2023 - as it happened DevFeed: [Product and Engineering away day 2023 - as it happened]() Original publisher: [Read original article]() Author: Calvin Dickson Published: 2023-06-19T15:39:09Z Content type: news Language: en Sources: [Guardian]() Topics: [Development]() Tags: [blog](), [development](), [engineering](), [news](), [technology]() ### AI overview A live blog covering the Guardian's 2023 Product and Engineering away day, including event activities and a recruitment message highlighting the team's work on digital products, journalism tools, distribution services, and financial systems. ### Source excerpt All the news and updates from the 2023 Guardian Product and Engineering away day The second session of the unconference time has come to a close. There will now be some time for feedback on the topics. Who will step up to share their thoughts? It's nearly time on session two. I'm hearing there will be a brief summary of each discussion back to the room. Let's see what the future holds for us all. Continue reading... ## How the Guardian's Ophan Analytics Tool Moved from Elasticsearch Rollups to BigQuery DevFeed: [How the Guardian's Ophan Analytics Tool Moved from Elasticsearch Rollups to BigQuery]() Original publisher: [Read original article]() Author: Sam Hession Published: 2023-06-07T12:44:33Z Content type: article Language: en Sources: [Guardian]() Topics: [Monitoring](), [elasticsearch](), [real-time](), [BigQuery](), [data]() Tags: [analytics](), [bigquery](), [data](), [elasticsearch](), [monitoring](), [pipeline](), [real-time](), [warehouse]() ### AI overview The Guardian describes how Ophan, its real-time analytics tool, expanded from short-term pageview monitoring to historical data analysis. Because Elasticsearch Rollups remained in technical preview and lacked service-level guarantees, the team considered moving the long-term data pipeline to BigQuery. ### Source excerpt How the Guardian's real time analytics tool pivoted from ElasticSearch Rollups to BigQuery and what we learnt along the way Ophan is the Guardian's in-house developed real time analytics tool which allows us to see how our content is performing in real-time, providing our Editorial teams with the insights they need to curate and promote our journalism. Its intuitive ways of monitoring reader engagement with content help the teams to evaluate whether it's performing optimally. Ophan's distinguishing feature has always been to provide journalists with visualised real time pageview information, with features based on the previous two weeks. More recently we have expanded these capabilities to provide insights for wider timescales with years of data available at their fingertips. Continue reading... ## Hack Day: Food Glorious Food DevFeed: [Hack Day: Food Glorious Food]() Original publisher: [Read original article]() Author: Rasha Ardati Published: 2023-05-31T08:45:02Z Content type: article Language: en Sources: [Guardian]() Topics: [Browser Extension](), [App](), [Artificial Intelligence](), [web browser](), [Image](), [stable-diffusion]() Tags: [ai](), [browser](), [extension](), [hack](), [images](), [stable-diffusion]() ### AI overview The Guardian's first hack day of 2023 focused on food-related ideas and produced more than 22 hacks over two days. Projects included browser extensions, recipe interfaces, a restaurant review mapper, microwave-friendly recipes, and an AI tool that generates recipe-step images using stable diffusion or DALL-E. ### Source excerpt This month, colleagues across the product and engineering department, as well as other parts of the organisation, came together for another exciting hack day For our first hack day of 2023 we kicked off with a delicious theme around 'Food', with a side of the usual 'free-for-all' of course. Hackers were invited to think about all things edible, whether as an app, a new search method for recipes, a meal planner or perhaps a choose your own adventure game where your date is coming for dinner in an hour and a half, and you have decided, perhaps too ambitiously, to cook them an elaborate Ottolenghi recipe from the Guardian archives. We produced and presented over 22 amazing hacks over the two-day event. These included: Migrating our tone/recipes page to our latest rendering tiers, A web browser extension to help users understand cooking language and become better cooks, making Guardian recipes microwave-friendly, a restaurant review mapper, implementing a Guardian app section for food content and a web browser extension to appeal to our tofu-eating readers. Continue reading... ## Galaxies of the Guardian: the formation of an irregular org chart DevFeed: [Galaxies of the Guardian: the formation of an irregular org chart]() Original publisher: [Read original article]() Author: Frederick O'Brien Published: 2023-02-03T13:05:39Z Content type: article Language: en Sources: [Guardian]() Topics: [Graphs](), [JavaScript](), [Software Engineering]() Tags: [d3](), [graphs](), [javascript](), [prototype](), [software-engineering](), [structure]() ### AI overview The Guardian's Product & Engineering department built Galaxies, a force-directed visualization of interconnected people, teams, and streams intended to present organizational structure in a beautiful, intuitive, non-hierarchical form. The project grew from learning data visualization with D3 and experimenting with d3-force. ### Source excerpt 'Who does what and where?' is an age-old question. We in Product & Engineering have tried to answer it with Galaxies, a data visualisation of interconnected people, teams, and streams We all feel a little lost at times. Who am I? What is my purpose? Who's the best person to speak to about setting up my AWS credentials? Anyone who says they aren't occasionally haunted by these questions is lying to you. A common way to try to answer these kinds of questions is through organisational charts. Org charts aren't the first things that come to mind when thinking of something inspiring, which is a shame because the information they're trying to convey can be fascinating - it just needs to be presented in the right way. Continue reading...