# Extracting 100% of Data From a Stubborn, Dying ZFS Pool

DevFeed: [Extracting 100% of Data From a Stubborn, Dying ZFS Pool](<https://devfeed.tech/articles/extracting-100-of-data-from-a-stubborn-dying-zfs-pool-20935.md>)

Original publisher: [Read original article](<https://jakewharton.com/extracting-100-percent-of-data-from-a-stubborn-dying-zfs-zpool/>)

Published: 2020-02-12T00:00:00Z

Content type: article

Language: en

Sources: [Jake Wharton](<https://devfeed.tech/sources/jake-wharton.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [Homelab](<https://devfeed.tech/topics/homelab.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>), [IO](<https://devfeed.tech/topics/io.md>), [Kernel](<https://devfeed.tech/topics/kernel.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [Software](<https://devfeed.tech/topics/software.md>), [ssh](<https://devfeed.tech/topics/ssh.md>), [Ubuntu](<https://devfeed.tech/topics/ubuntu.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [home-server](<https://devfeed.tech/tags/home-server.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [process](<https://devfeed.tech/tags/process.md>), [rsync](<https://devfeed.tech/tags/rsync.md>), [server](<https://devfeed.tech/tags/server.md>), [software](<https://devfeed.tech/tags/software.md>), [ssh](<https://devfeed.tech/tags/ssh.md>), [ubuntu](<https://devfeed.tech/tags/ubuntu.md>)

## AI overview

The article describes recovering all data from a deteriorating ZFS pool built from ten-year-old hard drives. During migration to a new server, drive read failures and pool lockups caused rsync transfers to enter uninterruptible sleep in the I/O stack, requiring repeated hard reboots and extending the estimated transfer time from four days to approximately twelve.

## Source excerpt

In 2010 I built a home server with five 2TB drives. It ran Solaris and ZFS for the redundancy and data checksumming to ensure no data could be lost or corrupted. Just 16 months later five 3TB drives were added to the pool. This computer took the 2600-mile trip to live in San Francisco with me. It then endured the 2600-mile return trip when I left. Having sat unplugged for five years, I recently powered the server back on for new workloads. But relying on 10 ten-year-old hard drives in 2020 is asking for cascading failure. And not only were the drives old, they've experienced physical trauma. So instead I built a new server and endeavored to migrate the data. During the transfer the drives exhibited consistent read failures as expected, but ZFS was able to transparently mitigate them. Occasionally, though, the pool would lock up in a way that could only be fixed with a hard reboot. These lock ups sent me on a weird journey of software and hardware orchestration to complete the data transfer. Symptoms During transfer of the data, progress would stall randomly in a way that seemingly could not be killed. CTRL+C had no effect. No kill signal had an effect. Even last-resort shutdown -r nows did nothing. The system was oddly otherwise responsive. You could SSH in from another tab and poke around. ps showed that the transfer process was in the "D+" state which was uninterruptible sleep in the foreground. jake 21749 1.1 0.0 8400 2124 pts/0 D+ 23:42 0:00 rsync ... That explained why the process wouldn't die. The dmesg output also confirmed the problem happened deep in the I/O stack. [ 3626.101527] INFO: task rsync:30680 blocked for more than 120 seconds. [ 3626.101547] Tainted: P O 5.3.0-26-generic #28-Ubuntu [ 3626.101563] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 3626.101580] rsync D 0 30680 1 0x00000000 [ 3626.101584] Call Trace: [ 3626.101590] __schedule+0x2b9/0x6c0 [ 3626.101596] schedule+0x42/0xb0 [ 3626.101601] schedule_timeout+0x152/