# data migrations

Published articles for data migrations.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## TimescaleDB to ClickHouse replication: Use cases, features, and how we built it

DevFeed: [TimescaleDB to ClickHouse replication: Use cases, features, and how we built it](<https://devfeed.tech/articles/timescaledb-to-clickhouse-replication-use-cases-features-and-how-we-built-it-5604.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/timescale-to-clickhouse-clickpipe-cdc>)

Author: The ClickPipes Team

Published: 2025-09-09T00:00:00Z

Content type: article

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [Replication](<https://devfeed.tech/topics/replication.md>), [migration](<https://devfeed.tech/topics/migration.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>), [Database](<https://devfeed.tech/topics/database.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [data](<https://devfeed.tech/tags/data.md>), [data-migrations](<https://devfeed.tech/tags/data-migrations.md>), [database](<https://devfeed.tech/tags/database.md>), [migration](<https://devfeed.tech/tags/migration.md>), [migrations](<https://devfeed.tech/tags/migrations.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [replication](<https://devfeed.tech/tags/replication.md>), [sync](<https://devfeed.tech/tags/sync.md>), [time-series](<https://devfeed.tech/tags/time-series.md>)

### AI overview

This article explains how ClickHouse's Postgres CDC connector in ClickPipes supports one-time migrations and continuous replication from TimescaleDB to ClickHouse Cloud. It describes online and iterative migration scenarios, synchronization with low lag, and co-existence of TimescaleDB and ClickHouse for real-time workloads and analytics.

### Source excerpt

The Postgres CDC connector in ClickPipes now supports one-time migrations and continuous replication from TimescaleDB.

## How we automatically handle ClickHouse® schema migrations

DevFeed: [How we automatically handle ClickHouse® schema migrations](<https://devfeed.tech/articles/how-we-automatically-handle-clickhouse-schema-migrations-18767.md>)

Original publisher: [Read original article](<https://www.tinybird.co/blog/when-not-to-migrate-your-data>)

Author: Raquel Barbadillo

Published: 2025-06-27T00:00:00Z

Content type: article

Language: en

Sources: [Tinybird](<https://devfeed.tech/sources/tinybird.md>)

Topics: [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lineage](<https://devfeed.tech/tags/data-lineage.md>), [data-migrations](<https://devfeed.tech/tags/data-migrations.md>), [migrations](<https://devfeed.tech/tags/migrations.md>), [scalable-analytics-architecture](<https://devfeed.tech/tags/scalable-analytics-architecture.md>), [schema](<https://devfeed.tech/tags/schema.md>)

### AI overview

The article explains how data lineage is used to optimize ClickHouse table deployments and avoid unnecessary, expensive data migrations.

### Source excerpt

How we use data lineage to optimize ClickHouse® table deployments and avoid unnecessary and expensive data migrations.

## A Pre-Deployment Checklist for Risky Data Migrations

DevFeed: [A Pre-Deployment Checklist for Risky Data Migrations](<https://devfeed.tech/articles/the-checklist-for-deploying-a-scary-change-19165.md>)

Original publisher: [Read original article](<https://artsy.github.io/blog/2023/09/13/deploying-a-scary-change/>)

Published: 2023-09-13T00:00:00Z

Content type: tutorial

Language: en

Sources: [Artsy](<https://devfeed.tech/sources/artsy.md>)

Topics: [Command-line interface](<https://devfeed.tech/topics/cli.md>), [Docker](<https://devfeed.tech/topics/docker.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [Sidekiq](<https://devfeed.tech/topics/sidekiq.md>), [Latency](<https://devfeed.tech/topics/latency.md>)

Tags: [command-line](<https://devfeed.tech/tags/command-line.md>), [data-migrations](<https://devfeed.tech/tags/data-migrations.md>), [deploy-process](<https://devfeed.tech/tags/deploy-process.md>), [docker](<https://devfeed.tech/tags/docker.md>), [error-reporting](<https://devfeed.tech/tags/error-reporting.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [latency](<https://devfeed.tech/tags/latency.md>), [logs](<https://devfeed.tech/tags/logs.md>), [migrations](<https://devfeed.tech/tags/migrations.md>), [retry](<https://devfeed.tech/tags/retry.md>), [ruby-on-rails](<https://devfeed.tech/tags/ruby-on-rails.md>), [sidekiq](<https://devfeed.tech/tags/sidekiq.md>)

### AI overview

This article presents a pre-deployment checklist for risky data migrations and other changes that could affect production services. It recommends planning rollback steps and monitoring error rates, reported errors, latency, logs, functionality, and failing Sidekiq retries after deployment.

### Source excerpt

Lately, I've been getting involved with some sketchy stuff. You know what I'm talking about-data migrations. I've been rolling out changes that have a significant risk of breaking our production environment for mission-critical services. It's been exciting work (keep your eyes out for more posts on the exact project, coming soon™), but I've definitely caused a couple incidents along the way. After accidentally taking down a key service for a couple hours, I decided I needed to have a better pre-deploy process for these changes. I did some thinking and came up with a short checklist to run through before I press the shiny green button. Here's the checklist I came up with: What is your plan if something goes wrong? Run through ramifications of rolling back. If there's a reason you're worried about rolling back, then you're not ready to deploy the change yet! Figure out exactly what command(s) you will need to run to roll back. At Artsy, this is usually a one-liner using Hokusai, our command-line Docker/Kubernetes CLI How will you tell if something is going wrong after you deploy? Error rate (DataDog) Specific error reporting (Sentry) Latency (DataDog) Logs (Papertrail) Functionality (does it still work? Are people using it successfully? Important for things where errors may not be bubbled up correctly or reported immediately) Sidekiq (are there lots of jobs queued to retry that are failing?) With this checklist in hand, I'm deploying more confidently and causing fewer incidents along the way. Do you have something similar? Are there things you think this checklist should include? Let me know in the comments!

## SQL Tips for Application DBAs

DevFeed: [SQL Tips for Application DBAs](<https://devfeed.tech/articles/some-sql-tricks-of-an-application-dba-33940.md>)

Original publisher: [Read original article](<https://hakibenita.com/sql-tricks-application-dba>)

Author: Haki Benita

Published: 2020-07-26T21:00:00Z

Content type: tutorial

Language: en

Sources: [Haki Benita](<https://devfeed.tech/sources/haki-benita.md>)

Topics: [SQL](<https://devfeed.tech/topics/sql.md>), [Development](<https://devfeed.tech/topics/development.md>), [Databases](<https://devfeed.tech/topics/databases.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>), [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [etl](<https://devfeed.tech/topics/etl.md>)

Tags: [articles](<https://devfeed.tech/tags/articles.md>), [data-migrations](<https://devfeed.tech/tags/data-migrations.md>), [databases](<https://devfeed.tech/tags/databases.md>), [development](<https://devfeed.tech/tags/development.md>), [etl](<https://devfeed.tech/tags/etl.md>), [performance](<https://devfeed.tech/tags/performance.md>), [schema-design](<https://devfeed.tech/tags/schema-design.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

An application DBA shares practical SQL and database-development advice, including limiting updates to rows that need changes and considering constraint costs when loading or updating many rows.

### Source excerpt

Some tips and misconceptions about database development I gathered along the way.

## How to Process Large Data Sets with Ruby

DevFeed: [How to Process Large Data Sets with Ruby](<https://devfeed.tech/articles/how-to-process-large-data-sets-with-ruby-21036.md>)

Original publisher: [Read original article](<https://jakeyesbeck.com/2015/08/09/how-to-process-large-data-sets-with-ruby/>)

Published: 2015-08-09T12:00:00Z

Content type: tutorial

Language: en

Sources: [Jake Yesbeck](<https://devfeed.tech/sources/jake-yesbeck.md>)

Topics: [Ruby](<https://devfeed.tech/topics/ruby.md>), [Rails](<https://devfeed.tech/topics/rails.md>), [migration](<https://devfeed.tech/topics/migration.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [systems](<https://devfeed.tech/topics/systems.md>), [jobs](<https://devfeed.tech/topics/jobs.md>)

Tags: [activerecord](<https://devfeed.tech/tags/activerecord.md>), [data-migrations](<https://devfeed.tech/tags/data-migrations.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [rails](<https://devfeed.tech/tags/rails.md>), [redis](<https://devfeed.tech/tags/redis.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [ruby-on-rails](<https://devfeed.tech/tags/ruby-on-rails.md>), [scale](<https://devfeed.tech/tags/scale.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

This tutorial explains how to process a large Ruby on Rails data migration more efficiently. It first shows why a serial script for updating two million users is too slow, then introduces Resque and Redis to distribute the work across background jobs and multiple workers.

### Source excerpt

The need for data migrations in mature systems is real. At times, requests for these migrations can appear at random. One minute, a system is behaving as specified, happily fulfilling requests, and then bam! All the user objects suddenly need an extremely crucial attribute. Well that seems relatively simple, right? All that is needed is a simple ruby script to iterate over all users and update every user with this essential piece of data. To demonstrate such a problem, we can assume the following: A Ruby on Rails application exists with a User class, each with a phone_number attribute. class User < ActiveRecord::Base validates_presence_of :phone_number end The application is relatively popular, which results in 2,000,000 users: User.count #=> 2000000 Finally, the requested task: Add a "+1" to the beginning of all user phone numbers (we are assuming that all phone numbers belong to users in the USA or Canada). Disclaimer: Ruby is probably not the best tool for this kind of data migration but for argument's sake we can assume it is the only one available. Serial scripts at scale are slow Without giving it too much thought, an approach to solve this problem might look something like: User.find_each do |user| user.phone_number = "+1#{ user.phone_number }" user.save! end This approach will work. The find_each method will make sure the memory footprint of the script stays low (it will not load every user into memory at once) and the phone numbers will be updated. However, this will be painfully slow. Even if the system is able to update 20 users per second, it will take approximately 27 hours to complete. 2,000,000 users / 20 users per second = 100,000 seconds to process all users 100,000 seconds / 60 seconds ~= 1,666 minutes 1,666 minutes / 60 minutes ~= 27 hours Resque to the Rescue Resque is a very useful Ruby library for creating background jobs. Redis is utilized as the storage for these jobs and individual Resque workers pick one job off each queue at a time. With a

## New release: pgloader 3.2

DevFeed: [New release: pgloader 3.2](<https://devfeed.tech/articles/new-release-pgloader-3-2-34548.md>)

Original publisher: [Read original article](<https://tapoueh.org/blog/2015/01/new-release-pgloader-3.2/>)

Author: Dimitri Fontaine PostgreSQL Major Contributor; Author

Published: 2015-01-16T08:35:00Z

Content type: release

Language: en

Sources: [Dimitri Fontaine](<https://devfeed.tech/sources/dimitri-fontaine.md>)

Topics: [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>), [migration](<https://devfeed.tech/topics/migration.md>), [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [Database](<https://devfeed.tech/topics/database.md>), [MySQL](<https://devfeed.tech/topics/mysql.md>), [Command-line interface](<https://devfeed.tech/topics/cli.md>), [SQLite](<https://devfeed.tech/topics/sqlite.md>), [sql-server](<https://devfeed.tech/topics/sql-server.md>)

Tags: [command-line](<https://devfeed.tech/tags/command-line.md>), [csv](<https://devfeed.tech/tags/csv.md>), [data-migrations](<https://devfeed.tech/tags/data-migrations.md>), [error-handling](<https://devfeed.tech/tags/error-handling.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [release](<https://devfeed.tech/tags/release.md>), [rollback](<https://devfeed.tech/tags/rollback.md>), [sql-server](<https://devfeed.tech/tags/sql-server.md>), [sqlite](<https://devfeed.tech/tags/sqlite.md>)

### AI overview

This release announces pgloader 3.2, which simplifies common data-loading jobs from the command line. It supports migrations such as MySQL to PostgreSQL with schema discovery, type casting, cleanup, and support for indexes, keys, and comments.

### Source excerpt

PostgreSQL comes with an awesome bulk copy protocol and tooling best known as the COPY and \copy commands. Being a transactional system, PostgreSQL COPY implementation will ROLLBACK any work done if a single error is found in the data set you're importing. That's the reason why pgloader got started: it provides with error handling for the COPY protocol.