# pg\_basebackup Adds Parallel File Fetching for PostgreSQL Base Backups

DevFeed: [pg\_basebackup Adds Parallel File Fetching for PostgreSQL Base Backups](<https://devfeed.tech/articles/pg-basebackup-34401.md>)

Original publisher: [Read original article](<https://tapoueh.org/blog/2010/11/pg_basebackup/>)

Author: Dimitri Fontaine PostgreSQL Major Contributor; Author

Published: 2010-11-07T12:45:00Z

Content type: article

Language: en

Sources: [Dimitri Fontaine](<https://devfeed.tech/sources/dimitri-fontaine.md>)

Topics: [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>), [Script](<https://devfeed.tech/topics/script.md>), [parallel](<https://devfeed.tech/topics/parallel.md>), [Concurrent Programming](<https://devfeed.tech/topics/concurrent-programming.md>)

Tags: [jobs](<https://devfeed.tech/tags/jobs.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [processes](<https://devfeed.tech/tags/processes.md>), [script](<https://devfeed.tech/tags/script.md>)

## AI overview

The article describes an update to pg_basebackup that repeatedly fetches changed xlog files during a base backup. It adds a --jobs option to launch multiple helper processes, each using its own PostgreSQL connection to fetch backup files in compressed chunks.

## Source excerpt

Hannu just gave me a good idea in this email on -hackers, proposing that pg_basebackup should get the xlog files again and again in a loop for the whole duration of the base backup. That's now done in the aforementioned tool, whose options got a little more useful now: Usage: pg_basebackup.py [-v] [-f] [-j jobs] "dsn" dest Options: -h, --help show this help message and exit --version show version and quit -x, --pg_xlog backup the pg_xlog files -v, --verbose be verbose and about processing progress -d, --debug show debug information, including SQL queries -f, --force remove destination directory if it exists -j JOBS, --jobs=JOBS how many helper jobs to launch -D DELAY, --delay=DELAY pg_xlog subprocess loop delay, see -x -S, --slave auxilliary process --stdin get list of files to backup from stdin Yeah, as implementing the xlog idea required having some kind of parallelism, I built on it and the script now has a --jobs option for you to setup how many processes to launch in parallel, all fetching some base backup files in its own standard ( libpq) PostgreSQL connection, in compressed chunks of 8 MB (so that's not 8 MB chunks sent over).