# Findmypast Technology and techniques from the Findmypast Tech teams This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## DevOps at Findmypast: How Spanners implement DevOps Practices DevFeed: [DevOps at Findmypast: How Spanners implement DevOps Practices]() Original publisher: [Read original article]() Author: Robbie Hill Published: 2023-08-21T17:30:00Z Content type: tutorial Language: en Sources: [Findmypast]() Topics: [DevOps](), [Development](), [Deployment](), [Testing](), [Monitoring](), [Security]() Tags: [atlassian](), [automation](), [ci](), [ci-cd](), [deployment](), [development](), [devops](), [engineering](), [findmypast](), [monitoring](), [security]() ### AI overview This article explains how Findmypast's Spanners team applies DevOps as an iterative, collaborative approach to developing, testing, deploying, and monitoring software services. It describes team goals focused on deployment speed, collaboration, quality, reliability, and security. ### Source excerpt Introduction There are many ways to define DevOps and even more ways to implement its core principles within an organisation. To the Spanners team, DevOps is an iterative and collaborative approach to the development, testing, deployment and monitoring of software services. Overall, there is a great deal of alignment with the thoughts outlined by Mueller and Willis regarding what DevOps is, and how it is best implemented. Spanners see our role within the wider engineering cohort at Findmypast as advocates for DevOps principles and practices, and the vehicle through which those practices can be implemented day-to-day. Spanners operate as an autonomous team, but our efforts centre around providing feature teams with the processes and tools they need to reap the rewards of following a DevOps approach when developing and maintaining services. Source: Atlassian Team Goals Our goals always trace back to wider organisation goals that in turn align with the expected company and department behaviours. For Spanners our quarterly team goals generally focus on how we improve five key areas in terms of software delivery across the engineering teams: Speed of Deployment - The rate of code deployments and the speed with which code can be built, tested and deployed to our environments. Collaboration - Enable teams to investigate issues and monitor services and systems using a well-defined and trusted series of tools. Providing a framework for teams to take Ownership of the service pipelines (sometimes described as "You build it, you run it"). Quality - Enable teams to continually improve upon their service/s through automation in the form of CI/CD pipelines, and quickly identify issues and shortcomings in those implementations. Reliability - Reduce variables when it comes to the building, testing and deployment of code changes, and provide near real-time feedback (measurement) of the effectiveness of those code changes. Security - We strive to manage everything as code and provide ## DevOps at Findmypast: Postgres Database Upgrades DevFeed: [DevOps at Findmypast: Postgres Database Upgrades]() Original publisher: [Read original article]() Author: Robbie Hill Published: 2022-11-17T11:30:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [Databases](), [DevOps](), [upgrade](), [Microservices](), [Kubernetes](), [Puppet](), [Ubuntu]() Tags: [back-end](), [clusters](), [configuration](), [databases](), [deployment](), [devops](), [engineering](), [findmypast](), [kubernetes](), [kubernetes-clusters](), [linux](), [microservices](), [mongodb](), [postgres](), [puppet](), [relational-databases](), [ubuntu](), [updates](), [upgrade](), [upgrades]() ### AI overview Findmypast describes its process for upgrading production and non-production Postgres databases. The article covers the database infrastructure, backup arrangements, Puppet-managed installations, version drift between environments, and the reasons for standardizing on the latest supported Postgres version. ### Source excerpt Introduction Findmypast and its sister site; the British Newspaper Archive (BNA), are served by a web of microservices operating across multiple Kubernetes clusters. A number of our microservices rely upon a back-end database, and although we do operate other database technologies (MSSQL, MongoDB etc.), Postgres is our de-facto standard for relational databases. When the DevOps team first realised the need to upgrade all of the database nodes to the latest Postgres version, we operated a total of twelve production Postgres databases, each running on their own dedicated Ubuntu Linux virtual machine (VM). Running on standalone VMs allows for separation of concerns, and limits the 'blast radius' of issues and failures should they occur. All VMs are managed by our Hyper-V cluster, and backed up regularly by Veeam. In addition to this backup method we operate Barman, which takes a full backup of all production Postgres database hosts nightly, and continuously receives WAL (Write Ahead Log) updates. We operate a a similar number of staging / integration / utility Postgres databases. Unlike the production databases these are (mostly) co-located on a single host, given the reduced storage and performance requirements of these databases. Existing Processes We manage Postgres installations and configuration through Puppet, pinning the installed version of Postgres. After the initial deployment we were not actively upgrading the Postgres version running on the hosts. Upgrades of the production databases were instead only undertaken by the feature teams (those teams that manage our micro-services), or by us when requested by the feature teams. This meant that we had a variety of Postgres databases deployed to production, between versions 9 and 14 (the latest supported version at the time). It also meant that the Production Postgres versions differed from their respective staging / integration database. We knew this situation, as well as the approach to upgrades was bad practice ## Data Science at FMP: Tree Structure Analysis DevFeed: [Data Science at FMP: Tree Structure Analysis]() Original publisher: [Read original article]() Author: David Lack Published: 2022-08-09T00:00:00Z Content type: tutorial Language: en Sources: [Findmypast]() Topics: [Graphs](), [Algorithms](), [data](), [Data Science]() Tags: [algorithm](), [algorithms](), [analysis](), [data](), [data-science](), [graph](), [product](), [structure]() ### AI overview The article explains how Findmypast analyzes the structure of customer family trees using directional graphs. It models people as nodes and relationships as directed edges, then uses weighted shortest-path algorithms to calculate generations and the people in each generation. ### Source excerpt Introduction A key part of our product is the ability to build family trees on our site. To better understand how our customers are building and using the tree features we wanted to work out what sort of structure the trees generally have and answer questions like how many generations do users have? Do users follow their pedigree line, their parents, grandparents and so on or build out their other relations? Do users focus on a paternal or maternal line or both? Trees can be quite complex structures with a lot of information contained within them. For example, a small tree with just a mother, father and daughter has 3 people but multiple relations depending on how you think about it. It could have 3 simple relations: Mother and Father are linked by a married relation Mother and Daughter are linked by a child relation Father and Daughter are also linked by a child relation These relations can be expanded on to give direction information as well, in which case you end up with 6 relationships: Mother is married to Father Father is married to Mother Mother is parent of Daughter Daughter is child of Mother Father is parent of Daughter Daughter is child of Father Adding in direction information has doubled the amount of relationships in the tree and this is before including other members of the tree and adding other relationships such as sibling relationships. Analysing this complex data is a challenge, how do we work out how many generations there are? How many people are in each generation? To solve these problems we decided to use directional graphs, as this seemed the most natural way to deal with this data. In a directional graph, nodes are connected by edges with a direction. The example of the simple tree matches a graph well, each person is a node and each relationship linking people is an edge. Calculating Generation Information In order to use graphs to calculate the number of generations and the people within each generation, we need to perform a weighted short ## Scaling Findmypast for 1921 Census release - Part 3 DevFeed: [Scaling Findmypast for 1921 Census release - Part 3]() Original publisher: [Read original article]() Author: Mike Thomas Published: 2022-04-26T00:00:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [Deployment](), [Hardware](), [Kubernetes](), [Network](), [virtual machines](), [Amazon Web Services](), [Linux](), [sql-server](), [Windows](), [ASP.NET](), [C#](), [.NET]() Tags: [aws](), [blog-post](), [c-sharp](), [chaos](), [deployment](), [findmypast](), [hardware](), [infrastructure](), [k8s](), [kubernetes](), [load](), [net](), [network](), [release](), [sql-server](), [stress](), [testing](), [virtual-machines](), [windows]() ### AI overview Findmypast describes the infrastructure challenges involved in scaling its service for the 1921 Census of England and Wales release. The third post in the series examines network timeouts and throughput problems during stress testing, leading the team to investigate hardware, Hyper-V, virtualized networking, Kubernetes, AWS, and related service infrastructure. ### Source excerpt Findmypast (FMP) released the 1921 Census of England & Wales at midnight on Jan 6th 2022 to an eager community of genealogists. The preparation of the census - preservation, digitisation and transcription took three years of hard work. Aside from the data preparation, we also had technical challenges to address. Specifically, could our services deal with the projected increase in users for the first few days of 1921 launch period? This post is the third in a series that details how we approached scaling our service to deal with a projected day one 12x increase of users. Digging deeper into the infrastructure We left the second blog post in a poor position, our scaling efforts were not going well; network timeouts and throughput issues dogged each and every one of our stress test game days. At this point, we started to suspect the issue was related to hardware rather than the services. We started to dig deeper into all layers of the network stack, along with looking at the new Hyper-V hardware. Before we dig too deeply into the things, lets recap on our existing hardware setup. For the search backend, we use SOLR with a cluster of bare metal machines to index and retrieve search results. This was one area where we knew in advance that we needed more resources, so we scaled the cluster into AWS for extra compute. The majority of the Findmypast service runs on K8s, but not on bare metal. The K8s cluster is composed of Linux virtual machines running on Hyper-V. Our Hyper-V setup consisted of two clusters of bare metal machines, however both clusters were running different versions of Hyper-V We also have some areas of the site still running Windows servers with a Microsoft .Net/C# stack with SQL Server backend. These are also virtual machines running on the same Hyper-V hardware. The internal services, staging environments, etc also run as virtual machines under Hyper-V. So, SOLR aside, Hyper-V managed pretty much everything, from our internal deployment process to the ## Testing Library ByRole Performance Improvements DevFeed: [Testing Library ByRole Performance Improvements]() Original publisher: [Read original article]() Author: Jonathan Darrer Published: 2022-04-13T00:00:00Z Content type: tutorial Language: en Sources: [Findmypast]() Topics: [Testing](), [React](), [ui]() Tags: [improvements](), [performance](), [react](), [testing](), [testing-library](), [tests]() ### AI overview This article explains how Testing Library queries affect test performance in React UI tests. It compares role-based queries with the more limited ByText and ByLabelText queries, notes that they are not interchangeable for every assertion, and describes contextualizing queries when necessary. In the tested instances, the recommended changes improved performance from 320 ms to 16 ms, a 95% improvement. ### Source excerpt We use Testing Library to test our React UI pages and components. For example, we may have the following HelpFaqsTile component in a page: const HelpFaqsTile = () => (

Need some help?

Our FAQs are a lifesaver

 ); export default HelpFaqsTile; Let suppose we want to test whether "the 'Help Faqs' tile provides a link with the text Search FAQs": import { act, render } from '@testing-library/react'; await render( ); const buttonToFaq = screen.getByRole('link', { name: 'Search FAQs', }); expect(buttonToFaq).toBeInTheDocument(); *ByRole methods like screen.getAllByRole('link', {name: 'Search FAQs'}) can be slow due to the way in which they search through the entire DOM tree looking for nodes matching the role type, and optionally a variety of attributes (label, aria-label, etc. depending on additional criteria, here name). In contrast, the more performant *ByText and *ByLabelText methods look for text nodes and nodes with more limited attributes (label and aria-label only) respectively. So, we could use screen.getByText('Search FAQs') instead: import { act, render } from '@testing-library/react'; await render( ); const buttonToFaq = screen.getByText('Search FAQs'); expect(buttonToFaq).toBeInTheDocument(); They are not like for like replacements however. Let's suppose we want to test "the 'Help Faqs' tile provides a link which directs the user to our FAQ's" import { act, render } from '@testing-library/react'; await render( ); const buttonToFaq = screen.getByRole('link', { name: 'Search FAQs', }); expect(buttonToFaq).toBeInTheDocument(); expect(buttonToFaq).toHaveAttribute('href', '/ ## Team Diversity DevFeed: [Team Diversity]() Original publisher: [Read original article]() Author: Lorna Tyrtania Published: 2022-03-24T00:00:00Z Content type: opinion Language: en Sources: [Findmypast]() Topics: [User experience (UX)](), [Usability](), [Testing]() Tags: [diversity](), [findmypast](), [inclusion](), [language](), [languages](), [real-time](), [support](), [team](), [teams](), [testing](), [usability](), [work]() ### AI overview A Product and Design team member reflects on how diversity in age, gender, ethnicity, culture, languages, knowledge, and thinking styles supports collaboration and problem-solving. The article also describes using user research, customer feedback, forums, and usability testing to represent users in product discussions. ### Source excerpt Team Diversity On-line meetings have their advantages and disadvantages; I think most of us are feeling a little "zoomed out" through the Covid-19 lock downs. But, one useful feature is the ability to see all the faces of our Product and Design team, side-by-side on my screen each week for on our weekly catch-up. It led me to reflect on how fortunate we are to work with such a diverse team of colleagues and how important that is for team dynamics. We're a diverse bunch in terms of age, gender, ethnicity & cultures but also in terms of generational insight, platform and domain knowledge and different lived experiences. Between us we speak 19 different languages from Arabic to Welsh via Kiswahili (a tally I personally contribute nothing to, although a couple of us did make a plea for Brummie to count as a language!) Add on top of that very different cognitive thinking styles and the result is a team capable of taking problem solving in very different directions and allowing us to really stretch the art of the possible. I'm always keen to try and think about those not in the room, when I'm having meetings and usually as a Product Manager this is the user. It's the job of a Product person to always champion them and their perspective, but I do find that the user is always pretty well represented in our conversations, through feedback from UX research panels, insight from Customer Support teams and user forums and real-time feedback and insight from our usability testing tools. The fact that so many people in the business are personally so interested in family history also really helps, alongside regular "dog fooding" sessions where the whole organisation comes together to use our product and to think about family history ensures no one is working in an abstract way. Another diversity we celebrate and where I've observed real parity of esteem is for the types of work different teams are delivering. The contributions to building or rebuilding the foundations of our site o ## Scaling Findmypast for 1921 Census release - Part 2 DevFeed: [Scaling Findmypast for 1921 Census release - Part 2]() Original publisher: [Read original article]() Author: Mike Thomas Published: 2022-03-02T00:00:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [systems](), [Testing](), [GraphQL](), [Kubernetes](), [Deployment]() Tags: [automated](), [chaos](), [engineering](), [findmypast](), [graphql](), [infrastructure](), [k8s](), [kubernetes](), [load](), [performance](), [stress](), [testing]() ### AI overview Findmypast describes the second phase of scaling its service for the 1921 Census of England and Wales release. The engineering teams ran coordinated load-test game days against production, targeting up to four times normal January load. The first test day kept the site generally responsive but exposed failures in a GraphQL service and other performance and Kubernetes configuration issues. ### Source excerpt Findmypast (FMP) released the 1921 Census of England & Wales at midnight on Jan 6th 2022 to an eager community of genealogists. The preparation of the census - preservation, digitisation and transcription took three years of hard work. Aside from the data preparation, we also had technical challenges to address. Specifically, could our services deal with the projected increase in users for the first few days of 1921 launch period? This post is the second in a series that details how we approached scaling our service to deal with a projected day one 12x increase of users. If you haven't already, I'd suggest you read the first post which details the initial scaling steps taken by our engineering teams. Stress testing "game" days We ended the first post detailing that the load tests created by the teams were effective at testing the service in isolation. The next step was to get all the teams together and run their load tests at the same time against our production service. While still not a realistic example of a user journey, it would still test how our systems as a whole responded to the increased load. Game day # 1 Initially, we set a simple schedule of a load test "game day" each month. Scheduled for Thursday 8th April and spread over two hours we planned to run 3 load tests. Each was 20 mins in duration and we aimed for the first test to add 2x our normal load, the next test was 3x and the final test was 4x the load. Note that "normal load" means the load we would normally expect in a typical January. (1921 Census was launched in January). We have seasonal visitor patterns with more visitors during (the northern hemisphere) winter months than summer. When we set our targets for the load test, we aimed for 4x the normal January load, not the current April load. Overall, the first stress test game day was a success - the site stayed alive and generally responsive. But we did learn a few things from the first day: One of our key services that deals with GraphQL quer ## Scaling Findmypast for 1921 Census release - Part 1 DevFeed: [Scaling Findmypast for 1921 Census release - Part 1]() Original publisher: [Read original article]() Author: Mike Thomas Published: 2022-01-31T00:00:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [data](), [Kubernetes](), [Testing](), [Back end](), [telemetry]() Tags: [chaos](), [data](), [findmypast](), [infrastructure](), [k8s](), [kubernetes](), [load](), [metrics](), [performance](), [release](), [stress](), [testing]() ### AI overview Findmypast describes how its engineering team prepared for the 1921 Census of England & Wales release, anticipating a 12-fold increase in users. The article covers a scaling working group, performance improvements, service metrics, chaos and stress testing, and mapping dependencies across nearly 140 Kubernetes-based microservices. ### Source excerpt Findmypast (FMP) released the 1921 Census of England & Wales at midnight on Jan 6th 2022 to an eager community of genealogists. The preparation of the census - preservation, digitisation and transcription took three years of hard work. Aside from the data preparation, we also had technical challenges to address. Specifically, could our services deal with the projected increase in users for the first few days of 1921 launch period? This post is the first in a series that details how we approached scaling our service to deal with a projected day one 12x increase of users. In the beginning... ..way back in Nov 2020 the release of the 1921 census was just over 13 months away. Within engineering, our thoughts started to turn to ensuring that Findmypast was fit to meeting the anticipated increase in load. We knew from previous incidents that the reliability and performance of some of the services can cause issues when under load, so, within engineering, we started a scaling working group to address not only these specific issues, but to prove to management and our external stakeholders that the system as a whole was resilient and could perform well under load. The expectations for the working group was to: Champion performance improvements within the team. That is, evangelise scaling to the team, make sure that backend improvement work was added to backlogs, scheduled into sprints, etc. Ensure that the key metrics for each service owned by the team were correctly instrumented, visualised and alerted upon. Schedule time for chaos and stress testing team services. The initial meeting set the goals and expectations for the team and also discussed a few incidents that caused a site outages due to cascading failures and the like. Essentially, we highlighted the things that have gone wrong in the past and the discussed areas that would have limited the blast impact of the outage. We also re-visited our documentation, we have almost 140 micro services running in Kubernetes (K8s) a ## Data Science at FMP: Dataset Recommendation DevFeed: [Data Science at FMP: Dataset Recommendation]() Original publisher: [Read original article]() Author: Tom Thurstan Published: 2021-12-23T00:00:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [recommendation systems](), [Data Science](), [datasets](), [data]() Tags: [data](), [data-science](), [datasets](), [product](), [recommendation-systems](), [research]() ### AI overview Findmypast's data science team explores recommendation systems for helping users discover datasets relevant to their research. The article compares collaborative filtering and content-based filtering, explains why content-based filtering is unsuitable for this domain, and describes limitations caused by the lack of user ratings before introducing another approach. ### Source excerpt Problem Outline One of the problems we recently tackled in the data science team was how we could introduce our users to new datasets they may not be familiar with. We wanted these datasets to be personal to each user and relevant to their research areas. A recommendation system of some sort was the obvious way to solve this problem. Recommendation Systems Recommendation systems come with several possible methodologies. Two of the most widely known methodologies are Collaborative Filtering (which can be split into two branches, user-based or item-based) and Content-Based Filtering. User-based collaborative filtering recommendations are based on finding users with similar preferences and recommending items that similar users rated highly but have not been viewed by the target user. E.g., "Users who liked similar items to you also liked..." For item-based collaborative filtering, instead of finding similar users we find items that have been rated similarly by different users and recommended based on a user's previous preferences. E.g., "Users that liked this item also liked..." An important part of the data required for these techniques is ratings, so you can tell which items users liked. Another popular technique is Content-Based Filtering. This technique requires less information about each individual user and more about the items themselves. The recommendations are created by finding items that have similar characteristics, e.g., movies of the same genre, with the same actors, or directors. Then given a user's previous preferences you can recommend items similar to items they previously enjoyed. A broad and consistent range of metadata is needed for each of the items and again ratings are useful for this methodology. Within our domain at Findmypast a Content-Based Filtering system would not be appropriate. It is easy to see that some datasets which would share a lot of metadata would not necessarily make good recommendations. E.g., Kentucky Birth Records would most lik ## Data Science at FMP: Network Science DevFeed: [Data Science at FMP: Network Science]() Original publisher: [Read original article]() Author: Anup Purewal Published: 2021-08-23T00:00:00Z Content type: article Language: en Sources: [Findmypast]() Topics: [Network](), [Graphs](), [Data Science](), [Statistics](), [Mathematics](), [Machine learning]() Tags: [data-science](), [graph](), [machine-learning](), [network](), [product](), [statistics]() ### AI overview Findmypast introduces its data science team and the first in a series of posts about its work. This article explains network science through nodes, edges, paths, directionality and measures, with examples illustrating how network analysis can represent real-world relationships and support genealogical research. ### Source excerpt Data Science at FMP Findmypast is at the forefront of facilitating historical, and in particular, genealogical research. This has been achieved by embracing advances in technology that have allowed us to digitise vast quantities of records and make these easily available online. The result is a diverse user-base that covers everyone from advanced professional researchers through to those just starting out on their family history journey. Recently, we have taken another stride forward by establishing a data science team who utilise advanced mathematics, statistics and machine learning to improve a user's research experience. Can we help users who have a hit a brick wall in their research? Can we suggest the next record set that they should explore or the next search they should conduct? Can we help users glean information more easily from less structured texts such as newspapers? This is the first in a series of blogs that will give an insight into what the Data Science team is working on and more importantly how that helps our users. Network Science In simple terms, a network (or graph) is made up of nodes (vertices) connected in some fashion by a set of edges. The nodes can represent a variety of different real-world entities depending on the subject matter, for example people, places, animals, or neurons. The edges that link these nodes can be customised to represent different types of connectivity by introducing weighting or directionality. A path (or walk) through a network is a sequence of nodes and edges generated by traversing the network where the sequenced nodes and edges are incident to one another. In the very simple predator-prey (food chain) example below, the animals (and plant) are the nodes and the linking lines between them are the edges. The arrows on the edges indicate directionality, which in this example indicates who is the predator and who is the prey. For example, the rabbit consumes the plant as the arrow extends from the plant to the rabbit