# findmypast Published articles for findmypast. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## DevOps at Findmypast: How Spanners implement DevOps Practices DevFeed: [DevOps at Findmypast: How Spanners implement DevOps Practices]() Original publisher: [Read original article]() Author: Robbie Hill Published: 2023-08-21T17:30:00Z Content type: tutorial Language: en Sources: [Findmypast]() Topics: [DevOps](), [Development](), [Deployment](), [Testing](), [Monitoring](), [Security]() Tags: [atlassian](), [automation](), [ci](), [ci-cd](), [deployment](), [development](), [devops](), [engineering](), [findmypast](), [monitoring](), [security]() ### AI overview This article explains how Findmypast's Spanners team applies DevOps as an iterative, collaborative approach to developing, testing, deploying, and monitoring software services. It describes team goals focused on deployment speed, collaboration, quality, reliability, and security. ### Source excerpt Introduction There are many ways to define DevOps and even more ways to implement its core principles within an organisation. To the Spanners team, DevOps is an iterative and collaborative approach to the development, testing, deployment and monitoring of software services. Overall, there is a great deal of alignment with the thoughts outlined by Mueller and Willis regarding what DevOps is, and how it is best implemented. Spanners see our role within the wider engineering cohort at Findmypast as advocates for DevOps principles and practices, and the vehicle through which those practices can be implemented day-to-day. Spanners operate as an autonomous team, but our efforts centre around providing feature teams with the processes and tools they need to reap the rewards of following a DevOps approach when developing and maintaining services. Source: Atlassian Team Goals Our goals always trace back to wider organisation goals that in turn align with the expected company and department behaviours. For Spanners our quarterly team goals generally focus on how we improve five key areas in terms of software delivery across the engineering teams: Speed of Deployment - The rate of code deployments and the speed with which code can be built, tested and deployed to our environments. Collaboration - Enable teams to investigate issues and monitor services and systems using a well-defined and trusted series of tools. Providing a framework for teams to take Ownership of the service pipelines (sometimes described as "You build it, you run it"). Quality - Enable teams to continually improve upon their service/s through automation in the form of CI/CD pipelines, and quickly identify issues and shortcomings in those implementations. Reliability - Reduce variables when it comes to the building, testing and deployment of code changes, and provide near real-time feedback (measurement) of the effectiveness of those code changes. Security - We strive to manage everything as code and provide ## DevOps at Findmypast: Postgres Database Upgrades DevFeed: [DevOps at Findmypast: Postgres Database Upgrades]() Original publisher: [Read original article]() Author: Robbie Hill Published: 2022-11-17T11:30:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [Databases](), [DevOps](), [upgrade](), [Microservices](), [Kubernetes](), [Puppet](), [Ubuntu]() Tags: [back-end](), [clusters](), [configuration](), [databases](), [deployment](), [devops](), [engineering](), [findmypast](), [kubernetes](), [kubernetes-clusters](), [linux](), [microservices](), [mongodb](), [postgres](), [puppet](), [relational-databases](), [ubuntu](), [updates](), [upgrade](), [upgrades]() ### AI overview Findmypast describes its process for upgrading production and non-production Postgres databases. The article covers the database infrastructure, backup arrangements, Puppet-managed installations, version drift between environments, and the reasons for standardizing on the latest supported Postgres version. ### Source excerpt Introduction Findmypast and its sister site; the British Newspaper Archive (BNA), are served by a web of microservices operating across multiple Kubernetes clusters. A number of our microservices rely upon a back-end database, and although we do operate other database technologies (MSSQL, MongoDB etc.), Postgres is our de-facto standard for relational databases. When the DevOps team first realised the need to upgrade all of the database nodes to the latest Postgres version, we operated a total of twelve production Postgres databases, each running on their own dedicated Ubuntu Linux virtual machine (VM). Running on standalone VMs allows for separation of concerns, and limits the 'blast radius' of issues and failures should they occur. All VMs are managed by our Hyper-V cluster, and backed up regularly by Veeam. In addition to this backup method we operate Barman, which takes a full backup of all production Postgres database hosts nightly, and continuously receives WAL (Write Ahead Log) updates. We operate a a similar number of staging / integration / utility Postgres databases. Unlike the production databases these are (mostly) co-located on a single host, given the reduced storage and performance requirements of these databases. Existing Processes We manage Postgres installations and configuration through Puppet, pinning the installed version of Postgres. After the initial deployment we were not actively upgrading the Postgres version running on the hosts. Upgrades of the production databases were instead only undertaken by the feature teams (those teams that manage our micro-services), or by us when requested by the feature teams. This meant that we had a variety of Postgres databases deployed to production, between versions 9 and 14 (the latest supported version at the time). It also meant that the Production Postgres versions differed from their respective staging / integration database. We knew this situation, as well as the approach to upgrades was bad practice ## Scaling Findmypast for 1921 Census release - Part 3 DevFeed: [Scaling Findmypast for 1921 Census release - Part 3]() Original publisher: [Read original article]() Author: Mike Thomas Published: 2022-04-26T00:00:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [Deployment](), [Hardware](), [Kubernetes](), [Network](), [virtual machines](), [Amazon Web Services](), [Linux](), [sql-server](), [Windows](), [ASP.NET](), [C#](), [.NET]() Tags: [aws](), [blog-post](), [c-sharp](), [chaos](), [deployment](), [findmypast](), [hardware](), [infrastructure](), [k8s](), [kubernetes](), [load](), [net](), [network](), [release](), [sql-server](), [stress](), [testing](), [virtual-machines](), [windows]() ### AI overview Findmypast describes the infrastructure challenges involved in scaling its service for the 1921 Census of England and Wales release. The third post in the series examines network timeouts and throughput problems during stress testing, leading the team to investigate hardware, Hyper-V, virtualized networking, Kubernetes, AWS, and related service infrastructure. ### Source excerpt Findmypast (FMP) released the 1921 Census of England & Wales at midnight on Jan 6th 2022 to an eager community of genealogists. The preparation of the census - preservation, digitisation and transcription took three years of hard work. Aside from the data preparation, we also had technical challenges to address. Specifically, could our services deal with the projected increase in users for the first few days of 1921 launch period? This post is the third in a series that details how we approached scaling our service to deal with a projected day one 12x increase of users. Digging deeper into the infrastructure We left the second blog post in a poor position, our scaling efforts were not going well; network timeouts and throughput issues dogged each and every one of our stress test game days. At this point, we started to suspect the issue was related to hardware rather than the services. We started to dig deeper into all layers of the network stack, along with looking at the new Hyper-V hardware. Before we dig too deeply into the things, lets recap on our existing hardware setup. For the search backend, we use SOLR with a cluster of bare metal machines to index and retrieve search results. This was one area where we knew in advance that we needed more resources, so we scaled the cluster into AWS for extra compute. The majority of the Findmypast service runs on K8s, but not on bare metal. The K8s cluster is composed of Linux virtual machines running on Hyper-V. Our Hyper-V setup consisted of two clusters of bare metal machines, however both clusters were running different versions of Hyper-V We also have some areas of the site still running Windows servers with a Microsoft .Net/C# stack with SQL Server backend. These are also virtual machines running on the same Hyper-V hardware. The internal services, staging environments, etc also run as virtual machines under Hyper-V. So, SOLR aside, Hyper-V managed pretty much everything, from our internal deployment process to the ## Team Diversity DevFeed: [Team Diversity]() Original publisher: [Read original article]() Author: Lorna Tyrtania Published: 2022-03-24T00:00:00Z Content type: opinion Language: en Sources: [Findmypast]() Topics: [User experience (UX)](), [Usability](), [Testing]() Tags: [diversity](), [findmypast](), [inclusion](), [language](), [languages](), [real-time](), [support](), [team](), [teams](), [testing](), [usability](), [work]() ### AI overview A Product and Design team member reflects on how diversity in age, gender, ethnicity, culture, languages, knowledge, and thinking styles supports collaboration and problem-solving. The article also describes using user research, customer feedback, forums, and usability testing to represent users in product discussions. ### Source excerpt Team Diversity On-line meetings have their advantages and disadvantages; I think most of us are feeling a little "zoomed out" through the Covid-19 lock downs. But, one useful feature is the ability to see all the faces of our Product and Design team, side-by-side on my screen each week for on our weekly catch-up. It led me to reflect on how fortunate we are to work with such a diverse team of colleagues and how important that is for team dynamics. We're a diverse bunch in terms of age, gender, ethnicity & cultures but also in terms of generational insight, platform and domain knowledge and different lived experiences. Between us we speak 19 different languages from Arabic to Welsh via Kiswahili (a tally I personally contribute nothing to, although a couple of us did make a plea for Brummie to count as a language!) Add on top of that very different cognitive thinking styles and the result is a team capable of taking problem solving in very different directions and allowing us to really stretch the art of the possible. I'm always keen to try and think about those not in the room, when I'm having meetings and usually as a Product Manager this is the user. It's the job of a Product person to always champion them and their perspective, but I do find that the user is always pretty well represented in our conversations, through feedback from UX research panels, insight from Customer Support teams and user forums and real-time feedback and insight from our usability testing tools. The fact that so many people in the business are personally so interested in family history also really helps, alongside regular "dog fooding" sessions where the whole organisation comes together to use our product and to think about family history ensures no one is working in an abstract way. Another diversity we celebrate and where I've observed real parity of esteem is for the types of work different teams are delivering. The contributions to building or rebuilding the foundations of our site o ## Scaling Findmypast for 1921 Census release - Part 2 DevFeed: [Scaling Findmypast for 1921 Census release - Part 2]() Original publisher: [Read original article]() Author: Mike Thomas Published: 2022-03-02T00:00:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [systems](), [Testing](), [GraphQL](), [Kubernetes](), [Deployment]() Tags: [automated](), [chaos](), [engineering](), [findmypast](), [graphql](), [infrastructure](), [k8s](), [kubernetes](), [load](), [performance](), [stress](), [testing]() ### AI overview Findmypast describes the second phase of scaling its service for the 1921 Census of England and Wales release. The engineering teams ran coordinated load-test game days against production, targeting up to four times normal January load. The first test day kept the site generally responsive but exposed failures in a GraphQL service and other performance and Kubernetes configuration issues. ### Source excerpt Findmypast (FMP) released the 1921 Census of England & Wales at midnight on Jan 6th 2022 to an eager community of genealogists. The preparation of the census - preservation, digitisation and transcription took three years of hard work. Aside from the data preparation, we also had technical challenges to address. Specifically, could our services deal with the projected increase in users for the first few days of 1921 launch period? This post is the second in a series that details how we approached scaling our service to deal with a projected day one 12x increase of users. If you haven't already, I'd suggest you read the first post which details the initial scaling steps taken by our engineering teams. Stress testing "game" days We ended the first post detailing that the load tests created by the teams were effective at testing the service in isolation. The next step was to get all the teams together and run their load tests at the same time against our production service. While still not a realistic example of a user journey, it would still test how our systems as a whole responded to the increased load. Game day # 1 Initially, we set a simple schedule of a load test "game day" each month. Scheduled for Thursday 8th April and spread over two hours we planned to run 3 load tests. Each was 20 mins in duration and we aimed for the first test to add 2x our normal load, the next test was 3x and the final test was 4x the load. Note that "normal load" means the load we would normally expect in a typical January. (1921 Census was launched in January). We have seasonal visitor patterns with more visitors during (the northern hemisphere) winter months than summer. When we set our targets for the load test, we aimed for 4x the normal January load, not the current April load. Overall, the first stress test game day was a success - the site stayed alive and generally responsive. But we did learn a few things from the first day: One of our key services that deals with GraphQL quer ## Scaling Findmypast for 1921 Census release - Part 1 DevFeed: [Scaling Findmypast for 1921 Census release - Part 1]() Original publisher: [Read original article]() Author: Mike Thomas Published: 2022-01-31T00:00:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [data](), [Kubernetes](), [Testing](), [Back end](), [telemetry]() Tags: [chaos](), [data](), [findmypast](), [infrastructure](), [k8s](), [kubernetes](), [load](), [metrics](), [performance](), [release](), [stress](), [testing]() ### AI overview Findmypast describes how its engineering team prepared for the 1921 Census of England & Wales release, anticipating a 12-fold increase in users. The article covers a scaling working group, performance improvements, service metrics, chaos and stress testing, and mapping dependencies across nearly 140 Kubernetes-based microservices. ### Source excerpt Findmypast (FMP) released the 1921 Census of England & Wales at midnight on Jan 6th 2022 to an eager community of genealogists. The preparation of the census - preservation, digitisation and transcription took three years of hard work. Aside from the data preparation, we also had technical challenges to address. Specifically, could our services deal with the projected increase in users for the first few days of 1921 launch period? This post is the first in a series that details how we approached scaling our service to deal with a projected day one 12x increase of users. In the beginning... ..way back in Nov 2020 the release of the 1921 census was just over 13 months away. Within engineering, our thoughts started to turn to ensuring that Findmypast was fit to meeting the anticipated increase in load. We knew from previous incidents that the reliability and performance of some of the services can cause issues when under load, so, within engineering, we started a scaling working group to address not only these specific issues, but to prove to management and our external stakeholders that the system as a whole was resilient and could perform well under load. The expectations for the working group was to: Champion performance improvements within the team. That is, evangelise scaling to the team, make sure that backend improvement work was added to backlogs, scheduled into sprints, etc. Ensure that the key metrics for each service owned by the team were correctly instrumented, visualised and alerted upon. Schedule time for chaos and stress testing team services. The initial meeting set the goals and expectations for the team and also discussed a few incidents that caused a site outages due to cascading failures and the like. Essentially, we highlighted the things that have gone wrong in the past and the discussed areas that would have limited the blast impact of the outage. We also re-visited our documentation, we have almost 140 micro services running in Kubernetes (K8s) a