# Remind Code with purpose with our engineering team. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Transitive Closure in PostgreSQL DevFeed: [Transitive Closure in PostgreSQL]() Original publisher: [Read original article]() Author: Remind Published: 2023-09-29T00:00:00Z Content type: tutorial Language: en Sources: [Remind]() Topics: [PostgreSQL](), [Database](), [Graphs]() Tags: [database](), [graphs](), [postgresql](), [query](), [scale]() ### AI overview This tutorial explains how Remind models hierarchical organization data as graphs in PostgreSQL. It compares recursive queries, nested sets, and materialized transitive closure, selecting transitive closure to support efficient queries while accommodating frequent leaf-node updates. ### Source excerpt At Remind we operate one of the largest communication tools for education in the United States and Canada. We have millions of parents, students, teachers and administrators use our application each day to improve learning outcomes by sending 10s of millions of messages per day. With this large scale usage we've had many opportunities to innovate and some of our most recent work has been around using PostgreSQL to capture graph-based structure and efficiently query them. Graphs in a relational database Our particular dataset consists of multiple organization trees. In education, most classes belong to a school and most schools belong to a district. But some classes may belong to a department, which may belong to a school, which may belong to a college, which may belong to a campus, which may belong to a university. We wanted to build a system that could not only represent these hierarchies but do so in an efficient manner. As a concrete example, here's how a middle school mach class might be represented: In our database, we store these entities as an adjacency list. Note: We store these with UUIDs, but here we'll use the names to make it easier to follow. organization parent Algebra I Maple Middle School Maple Middle School Springfield District Springfield District NULL Common query patterns we have require finding all classes in a district, or finding the district a class belongs to. Naively, we need to traverse through Maple Middle School to determine that Algebra 1 is a descendent of Springfield District. Our goal was to make such queries fast and performant while maintaining accuracy. There are a few techniques for doing this inside a PostgreSQL database: We can traverse through the adjacency list using Recursive Queries. We can use Nested Sets We can materialize the Transitive Closure. We started with option 1 but performance wasn't quite where we wanted it to be. Option 2 was a viable path, but concerns around maintenance and our update pattern (where leaf nod ## Reducing scaling costs with Betterscaling DevFeed: [Reducing scaling costs with Betterscaling]() Original publisher: [Read original article]() Author: Remind Published: 2022-06-28T00:00:00Z Content type: tutorial Language: en Sources: [Remind]() Topics: [autoscaling](), [Amazon Web Services](), [API](), [servers]() Tags: [article](), [autoscaling](), [aws](), [cost](), [cpu](), [infrastructure](), [optimize]() ### AI overview Remind developed Betterscaling, a lightweight autoscaling tool for its largest cluster, after finding that AWS autoscaling left 20-30% of CPUs provisioned but unreserved. Betterscaling queries the ECS API based on running tasks and adjusts ASG capacity; Remind reports that average unreserved CPU fell from 30% to around 15%. ### Source excerpt Betterscaling was designed and implemented by Phil Frost. Phil was an exceptional engineer and a brilliant person, and he is dearly missed by all of us at Remind who knew him. I had the privilege of interviewing Phil extensively about Betterscaling over the course of writing this article -- much of it is in his own words. I hope that it will serve in some small way to reflect his expertise and dedication to his craft, and to stand as a tribute to his memory. More students, teachers, and parents are relying on Remind's free messaging features than ever, and although Remind is free for the majority of its users, serving those users still incurs costs. To keep Remind accessible to everyone, we're constantly working to optimize our infrastructure in order to reduce unnecessary overhead. As we dove into the data on our operating costs, we found that overprovisioned resources were a significant cost driver that was ripe for optimization. Specifically, in summer and fall of 2020, on average 20-30 percent of the CPUs in our cluster were provisioned but unreserved -- processing power we were paying for, but not using. Percentage of unreserved CPU before and after Betterscaling (implemented in mid-October 2020) Remind relies on AWS for our server infrastructure, and we found that this overprovisioning was mostly a result of inefficiencies in AWS autoscaling. This by no means an indictment of AWS autoscaling in general -- we simply found that it wasn't ideal for our use case, for reasons I'll discuss in more detail below. Instead, we developed a simple, lightweight autoscaling tool we're calling Betterscaling. In a nutshell, Betterscaling evaluates a control loop which queries the ECS API to determine how big the cluster needs to be based on the tasks running in it, then manipulates the DesiredCapacity on an ASG to make the cluster that big. This article discusses how and why we decided to implement Betterscaling, and then dives into the mechanics of the system itself. We now rel ## Inclusive Language Updates DevFeed: [Inclusive Language Updates]() Original publisher: [Read original article]() Author: Remind Published: 2022-01-21T00:00:00Z Content type: article Language: en Sources: [Remind]() Topics: [systems](), [GitHub](), [Databases]() Tags: [databases](), [engineering](), [github](), [inclusive](), [postgresql](), [repositories]() ### AI overview Remind's Sustaining Engineering team describes updates to inclusive language across its codebases, including changing GitHub's default branch from master to main and replacing master/slave and whitelist/blacklist terminology with writer/reader and allowlist/blocklist. ### Source excerpt Inclusive Language Overview The Sustaining Engineering team at Remind recently implemented changes that focus on using inclusive language within our codebases. We encourage other engineering teams to do the same, and hope these updates create a more inclusive space for everyone at Remind. Terms like master and slave have no place in a modern codebase. They don't belong inside of our repositories, and they certainly do not belong on a repository's default branch on GitHub. At Remind, we have 59 distinct services that power our application. Each of these services has language that needs to be updated thoughtfully and intentionally to ensure that our systems remain available to the parents, teachers, and students that depend on them. We followed the lead of Indeed's VP of Engineering Jack Humphrey to identify and replace problematic language with the help of this blog post. We also had a number of internal conversations between team members that helped us iterate on the language that we intended to change, which worked specifically for our teams at Remind. Hopefully, outlining the changes that we made can serve to help other organizations working to make their workplaces more inclusive. Actions We're Taking We've committed ourselves to the following updates: Github default branch master " main Database Terminology Master/Slave -> Writer/Reader Permission Lists Whitelist/Blacklist -> Allowlist/Blocklist I began our updates within three of the services that my team - Sustaining Engineering - owns and operates within regularly. The steps that I took to update each of these repositories are as follows: Checked the repo for hard coded references to master branch and changed to main locally Checked across all repos at Remind for hard coded references to the master branch of the repo I was currently working to update Checked the repo for master and slave terminology, generally related to databases and updated with writer and reader This was updated from source and replica to ## A Letter to Remind's Team About Keeping the Service Running During the COVID-19 Surge DevFeed: [A Letter to Remind's Team About Keeping the Service Running During the COVID-19 Surge]() Original publisher: [Read original article]() Author: Remind Published: 2021-01-08T00:00:00Z Content type: opinion Language: en Sources: [Remind]() Topics: [systems](), [Slack]() Tags: [canada](), [communication](), [education](), [slack](), [systems](), [team](), [us]() ### AI overview A letter thanks Remind's team for responding to service crashes and rapidly increasing load after COVID-19-related school closures drove unprecedented demand. It recognizes their sustained teamwork and effort to keep messages flowing for teachers, parents, and students. ### Source excerpt You sent me a Word doc on Wed 18 Mar 2020. Inside the doc was a copy/paste of a very large slack conversation. Inside the conversation was the complete contents of our production incidents channel on the morning of Mon 16 Mar. I glanced at the document, but was busy - our systems were straining and crashing because COVID-19 had erupted and US schools had closed, and teachers turned to Remind at scales we had never seen. Teachers have relied on Remind for ten years, and now they needed Remind more than ever. On Monday 16 Mar 2020, you were all up at 0500 PT, in the dark. Services crashed, and you fixed the error, and moved on to the next crash. You partnered to tackle really tough fixes. Load kept climbing, you felt the sense of panic and fear as it continued to climb all morning. It did not abate until 1500 PT that day. And you did it again Tuesday. And on Wednesday, when I received that Word doc. And Thursday, and Friday of that week. You sustained this pace, for weeks. While helping your own children try to learn online. While your partners were battling on the front lines of the pandemic. While your friends and family were isolated from you. While the scourge of racism burned in our streets. While people you knew and loved became sick. While people you knew passed on, without a good bye. And you kept Remind running for our teachers. You delivered every single message that our teachers, parents, and students sent. Even as teachers experienced delays and outages with other platforms, they did not with Remind, and they wrote in to let you know. I've never seen a finer example of teamwork. Thank you, team. Now, as you were working on a 2020 retro presentation to share with the company, I paused to read this document you sent me ten months ago. I am struck by the elegant choreography of your response on that Monday morning. I am struck by your selflessness. I am struck by how you made it look effortless, when it was anything but. I am struck by your courage in the fac ## Remind Secures Shared Content with Signed CDN URLs and a Private S3 Bucket DevFeed: [Remind Secures Shared Content with Signed CDN URLs and a Private S3 Bucket]() Original publisher: [Read original article]() Author: Remind Published: 2020-07-07T00:00:00Z Content type: opinion Language: en Sources: [Remind]() Topics: [Security](), [Bug Bounty](), [Amazon S3]() Tags: [amazon-s3](), [bug-bounty](), [cloudfront](), [files](), [security]() ### AI overview Remind describes how a bug bounty researcher's report led to changes in its content-sharing security. Content delivered through its CDN is now signed and expires after seven days, while the Amazon S3 bucket is private to prevent random access and protect content UUIDs. ### Source excerpt At Remind, we help educators communicate with millions of students and families. These connections are critical for supporting student learning, especially with school closures this year, and it's equally important for all of the communications on our platform to be secure, protected, and reliably delivered. As part of our security stance, we engage the security researcher community via a bug bounty program--which is how a recent collaboration with a researcher helped us identify improvements to user safety while maintaining the essential content-sharing function of our platform. First, some background: Along with messages, the Remind platform allows users to share content--like assignments, flyers, newsletters and other materials and resources--in their classes, schools, and districts. Files are uploaded to Remind and stored in Amazon S3 and served via our Amazon Cloudfront CDN, as is practice for many other companies. But unlike other platforms, a significant percentage of Remind users only receive content via SMS text notifications. To make sure that students and families could access this content directly from their text messages, especially if they had limited data or couldn't download our app, we made the decision many years ago to make files publicly accessible in an approach generally similar to the one used by a number of social networking applications: Files were obscured behind a randomly-generated UUID but could otherwise be accessed by anyone with the relevant UUID. Recently, we received a researcher submission challenging the security of our content-sharing implementation. As a result of the researcher's report and the improvements we made, content distributed via Remind's content delivery network (CDN) is now signed and automatically expires after 7 days; our S3 bucket is also now private, preventing random access. These enhancements prevent brute-force attacks on our S3 bucket aimed at discovering content UUIDs and further protect these UUIDs against le ## How we reached 99.9% uptime during our busiest season DevFeed: [How we reached 99.9% uptime during our busiest season]() Original publisher: [Read original article]() Author: Remind Published: 2020-02-20T00:00:00Z Content type: article Language: en Sources: [Remind]() Topics: [Availability](), [autoscaling](), [App](), [systems](), [cpu]() Tags: [app](), [autoscaling](), [availability](), [cpu](), [latency](), [systems]() ### AI overview Remind describes how it improved availability during the back-to-school season by measuring downtime, establishing and tightening uptime SLAs, and adopting autoscaled systems. Reported downtime fell from more than 1,000 minutes in August and September 2018 to 80 minutes in 2019. ### Source excerpt Availability is the lifeblood of any app. Downtime blocks new users from signing up, it damages the company reputation, and so on. Additionally for Remind, serving education administrators as they work to increase engagement of entire communities further drives the need for our systems to be available. While administrators share many of the same expectations that teachers have of Remind, that ultimate responsibility to their communities make it crucial for administrators to know that their messages were delivered and that delivery was timely. The impact of downtime is most apparent during the back-to-school season. Because Remind usage is so closely tied to the school year, we see our traffic increase 5 times and our sign-ups increase 15 times between July 1st and September 30th each year. As a result, we are always looking for ways to improve our availability. First, we need to be able to measure it. Internally, we define downtime at Remind as an error rate over 5%. When we first set out to measure our availability 2 years ago, the first step was to establish an internal uptime Service Level Agreement, or SLA. Once we established a baseline, we prioritized work to help us conform to that SLA. And once we met that SLA, we optimized them to be even tighter. We have integrated the cycle of honing our targets and tuning our systems to meet those targets into our product delivery cadence. This investment has paid off; during back to school 2018 we had over 1000 minutes of downtime across August and September. In 2019, we reduced that to 80 minutes. Looking back, here are some tactics that enabled us to conform to progressively more aggressive SLAs at Remind. Elastic systems are more forgiving (Our original viewpoint on autoscaling systems was financially motivated. If we scale systems down at night, then we save money. This is still true, but improvements to our cost model have made the financial savings less of a motivator.) As we focused on conforming to our availabil ## Hypothesis Testing in Production DevFeed: [Hypothesis Testing in Production]() Original publisher: [Read original article]() Author: Remind Published: 2019-12-18T00:00:00Z Content type: tutorial Language: en Sources: [Remind]() Topics: [Testing](), [debugging](), [incident](), [Microservice](), [Amazon Web Services](), [Databases](), [Cache]() Tags: [architecture](), [availability](), [aws](), [cache](), [databases](), [debugging](), [incident](), [load-balancer](), [testing]() ### AI overview This tutorial explains how to use hypothesis testing to investigate and resolve production incidents. It presents a debugging process based on monitoring, notifications, measurements, data, and verification, using an example involving a routing service in a microservice architecture with a legacy monolith. ### Source excerpt At Remind we have built a robust communication platform that makes it easy for educators, students, and parents to work together. Our users use Remind to keep classes organized, to help each other with school assignments, and to send out announcements for school events. Given these important use cases, users depend on Remind to deliver out their messages on time and reliably. As we scale, grow and change our systems to support all our users and meet their expectations, issues are bound to happen. When issues do happen, it is critical to identify and resolve them quickly to reduce the impact on our users. Throughout the years, we have developed techniques and common patterns that are useful when fixing issues. In this post, we'll walk through a process of using hypothesis testing to guide us in fixing an issue. An overview of our architecture Our architecture is a pretty standard microservice architecture (mixed with a legacy monolith). We have a routing layer that takes each user request and routes it to the appropriate service. Each service consumes other systems like databases or cache and lives behind a load balancer so that we can scale independently to handle our load. All of this is running on AWS spread out across multiple availability zones for robustness. While this serves us well, it adds an additional layer of complexity when we are debugging issues and incidents. We'll be going through an example which touches multiple services as well as our legacy monolith, which is called r101-api. Debugging an incident The first step for debugging an incident is to have monitors in place. With a monitor in place, notifications can be put in place to alert the appropriate people when an issue arises. Creating a good monitor is a topic in and of itself, but a good place to start is to alert when a threshold of failing requests is reached. Once we have an initial measurement of a potential issue, we can start to diagnose it and look at the data to form a hypothesis of w ## Practical iOS Layout Patterns with MVVM and UIView Subclasses DevFeed: [Practical iOS Layout Patterns with MVVM and UIView Subclasses]() Original publisher: [Read original article]() Author: Remind Published: 2019-05-28T00:00:00Z Content type: tutorial Language: en Sources: [Remind]() Topics: [iOS](), [ui](), [MVVM](), [Code review]() Tags: [code-review](), [ios](), [mvvm](), [patterns](), [ui]() ### AI overview This article describes iOS layout practices developed at Remind, including programmatic layout with SnapKit, code-reviewable layouts, and UIView subclasses that encapsulate reusable UI components. ### Source excerpt For the past couple years we've been implementing new features with an MVVM-driven approach. Over that time, we've settled into a couple of good (I think) patterns for implementing UIView subclasses, composing them in various view hierarchies, and handling the interactions between them. Ultimately, the goal of any iOS application is to get data presented onto the screen, so I thought I'd share what's been working well for us in that department. Layout In Code It's a debate where every team eventually has to pick a side: Do we use storyboards and xibs, or do we write all of our layout in code? To make a long story short, our team started with Interface Builder, but we abandoned xibs a few years ago and have gone to fully programmatic layout code using SnapKit as our DSL of choice. Every team's needs will be different, but we think it's important that we can effectively code-review each other's layout implementations and potentially have multiple people working on the same screen, neither of which are really practical with xibs. I'll leave it at that -- you do you. Use UIView Subclasses Liberally Nothing clutters the actual logic of a UIViewController like big blocks of view construction code. Whenever you have a logical grouping UI primitives (UILabel, UIImageView, UIButton, etc), make a UIView subclass that encapsulates all of the relative layout inside that view container. This allows you to build the view hierarchy for your view controller in simpler, larger, more logical chunks of UI. One of the more complex UI layouts we have in our application is the settings screen for a class the user owns. Let's look at the layout and focus on the CLASS SETTINGS section, which contains two items, each consisting of two labels and a switch. On our first iteration, our view controller maintained property accessors for each of those primitives: class ClassSettingsViewController: UIViewController { ... private let isPublicTitleLabel = UILabel() private let isPublicMessageLabel = ## How I gained commit access to Homebrew in 30 minutes DevFeed: [How I gained commit access to Homebrew in 30 minutes]() Original publisher: [Read original article]() Author: Remind Published: 2018-08-07T00:00:00Z Content type: article Language: en Sources: [Remind]() Topics: [Homebrew](), [Security](), [supply chain attacks](), [Jenkins](), [GitHub](), [Environment Variables](), [OAuth](), [GitHub API](), [incident]() Tags: [backdoor](), [credential-theft](), [curl](), [environment-variables](), [github](), [homebrew](), [incident](), [jenkins](), [npm](), [permissions](), [rubygems](), [security](), [supply-chain-attacks]() ### AI overview A security researcher describes gaining commit access to Homebrew repositories through a GitHub API token exposed in a publicly accessible Jenkins environment. The access was tested and then reported to Homebrew maintainers, highlighting risks to package-manager infrastructure and software supply chains. ### Source excerpt This issue was publicly disclosed on the Homebrew blog at https://brew.sh/2018/08/05/security-incident-disclosure/ Since the recent NPM, RubyGems, and Gentoo incidents, I've become increasingly interested, and concerned, with the potential for package managers to be used in supply chain attacks to distribute malicious software. Specifically with how the maintainers and infrastructure of these projects can be targeted as an attack vector. On Jun 31st, I went in with the intention of seeing if I could gain access to Homebrew's GitHub repositories. About 30 minutes later, I made my first commit to Homebrew/homebrew-core. Let's get leaky My initial strategy going in was based on credential theft; find if there were any credentials leaked by members of the Homebrew GitHub org. An OSSINT tool from Michael Henriksen called gitrob makes automating this search really easy. I ran it across the Homebrew organization, but ultimately didn't come up with anything interesting. Next, I took a look at previously disclosed issues on https://hackerone.com/Homebrew. From there, I found that Homebrew runs a Jenkins instance that's (intentionally) publicly exposed at https://jenkins.brew.sh. After some digging, I noticed something interesting; builds in the "Homebrew Bottles" project were making authenticated pushes to the BrewTestBot/homebrew-core repo: This got me thinking, "where are the credentials stored?". I noticed the "Environment Variables" link on the left, which led to an exposed GitHub API token: I tested it locally to see what scopes the token had: $ curl https://api.github.com/user/repos -u $GITHUB_API_TOKEN:x-oauth-basic | jq '.[] | {repo: .full_name, permissions: .permissions}' { "repo": "BrewTestBot/homebrew-core", "permissions": { "admin": true, "push": true, "pull": true } } { "repo": "Homebrew/brew", "permissions": { "admin": false, "push": true, "pull": true } } { "repo": "Homebrew/formulae.brew.sh", "permissions": { "admin": false, "push": true, "pull": true } } { " ## Building Infrastructure in Parallel DevFeed: [Building Infrastructure in Parallel]() Original publisher: [Read original article]() Author: Remind Published: 2018-03-05T00:00:00Z Content type: tutorial Language: en Sources: [Remind]() Topics: [Reverse Dependencies](), [cloud-infrastructure](), [AWS CloudFormation](), [Amazon Web Services](), [Algorithms](), [Computer science](), [make](), [systemd](), [Terraform](), [coding]() Tags: [algorithms](), [building](), [cloudformation](), [computer-science](), [dependency](), [graphs](), [infrastructure](), [make](), [parallel](), [parallelism](), [postgresql](), [refactor](), [systemd](), [terraform](), [vpc]() ### AI overview This tutorial explains how Remind improved stacker, its infrastructure deployment tool, to update CloudFormation stacks in parallel. It models stack dependencies as a directed acyclic graph and reports reducing execution time from about 10 minutes to under 1.5 minutes; the change was included in stacker 1.2. ### Source excerpt In the early days of stacker at Remind, the number of stacks that we were managing was just a handful. CloudFormation stacks were updated sequentially, without any parallelism, and everyone was happy. Today, stacker manages 153 CloudFormation stacks that build out all of the AWS infrastructure that makes remind.com work. As you can imagine, attempting to update 153 stacks without any parallelism takes a long time, about 10 minutes to be precise. When you consider that most stacks don't ever change, 10 minutes feels like an eternity, and prevents a quick feedback loop. Over the last couple of weeks, I took it upon myself to refactor the core issues in stacker that made implementing parallelism difficult, and dropped our total execution time from 10 minutes, to under 1.5 minutes. This work is now included in stacker 1.2 so everyone can benefit. Thinking in Graphs Before I dive into how we implemented parallelism in stacker, I think it's important to take a step back, and look at what stacker is, at it's core; a tool for linking infrastructure together as a dependency graph. Let's take a look at a hypothetical stacker config to build out some infrastructure. stacks: - name: vpc class_path: blueprints.VPC - name: cluster class_path: blueprints.Cluster variables: VpcId: ${output vpc::Id} - name: db class_path: blueprints.DB variables: VpcId: ${output vpc::Id} - name: app class_path: blueprints.App variables: DBUrl: ${output db::URL} Cluster: ${output cluster::Id} When described in this way, the output lookup creates an explicit dependency relationship between stacks. The "app" stack depends on a "db" stack and a "cluster" stack, the "db" stack depends on a "vpc" stack, etc. We can visualize the resulting dependency graph using dot. digraph { cluster -> vpc; db -> vpc; app -> db; app -> cluster; } $ cat graph.dot | dot -Tpng > graph.png In fact, creating dependency graphs like this isn't really a new concept in computing; make does it, systemd does it, and programing lang