can computer shutdown during transmission download corrupt disk drive

Ask for help and report issues not specific to either the Mac OS X or GTK+ versions of Transmission
Post Reply
inorganic
Posts: 2
Joined: Tue May 23, 2023 10:12 pm

can computer shutdown during transmission download corrupt disk drive

Post by inorganic »

This is probably a question and request for someone who intimately understands how the internals of transmission torrent client functions.

I'm usually overly careful about how I operate my computers, but about a week ago I was in a hurry and did not close transmission while it was busy downloading 5 large torrent/magnet "files" from (wrong spelling) biratepay (about 150GB each, each containing about 32 actual individual files) before I shutdown Linux Mint v21.10 OS the usual way (via the LM button at the bottom-left corner of the display).

The next day when I booted-up the computer, the 02TB SSD M.2 NVME default boot-up drive (and destination for downloading Transmission files) did not appear as an optional drive or OS to boot-up. Usually it was the top entry and the default entry (meaning after a wait of several seconds it would boot-up on that top entry).

The default boot-up drive/OS was a Samsung 980 PRO M.2 NVME SSD "stick" on the ASUS ROG x670e extreme motherboard.

When I went into BIOS that SSD did appear under the NVME section.

However, when I looked in the list of SATA drives and the drives that could be booted-up onto ... it did not appear.

So I booted-up on a conventional 16TB HDD that I had also installed Linux Mint v21.10 onto the same day as the 02TB SSD that seemed to vanish.

Once I was in Linux Mint on the 16TB HDD, I ran the "disks" application --- and it showed NO partitions, just a completely empty SSD. Then I performed a hex-dump of the first few sectors of the 02TB SSD drive and ... it showed the 4-entry (64-byte) partition table as containing all 0x00 bytes, followed by the expected 0xAA55 marker (or was it 0x55AA, not sure).

When I ran the [c]testdisk[/c] application, it showed it could recover 4 or 5 partitions (that looked like the partitions I expected to find).

-----

So, those are the basic facts. Here are my questions.

#01: I would expect an application that does what Transmission does to not be "confused" by the OS being shutdown while it was running, and that presumably the Transmission application would not even lose track of what chunks of the files had been downloaded and saved to disk, and what chunks of the files had not yet been saved to the allocated disk files. Is this true?

#02: I would expect that any modern, decent OS ... certainly including Linux Mint v21.10 (the latest version) ... would be written in a way that no application could corrupt the OS, the disks, and certainly not the partition tables on the mounted disk drives, including the default/boot drive. Is this true?

#03: I'm guessing the problem that happened has nothing to do with the Transmission application. But ... is this necessarily true?

#04: Is it considered "bad practice" to not shutdown the Transmission application before shutdown of the OS / computer?

#05: As a mostly unrelated question, I was running the Transmission application because I could not find where to download Vuze ... the application that I've been running on my Linux Mint v19.10 computer for the past several years. When you look at the display of files being downloaded by Vuze, you see that tiny "chunks" or "slivers" of files are downloaded from all over the place in the file ... most definitely not in order from first-to-last "chunk" or "sliver". However, the Transmission display makes it appear that the files are being downloaded from first-byte to last-byte in order. Just from the obvious fact that the files are being downloaded from several different places simultaneously, I infer that the Transmission application is functioning the same way as Vuze does ... more-or-less random order (perhaps decided by the computers the data is being download from). Is this true? Is there a way to see where the Transmission application is saving "chunks" or "slivers" of data (like Vuze shows)? PS: I note that when a new file is selected, the size of the file reported by the nemo file browser application is the final total size of the file. This implies to me that room for the entire file is allocated immediately, which tends to imply the same behavior as Vuze. But ... I have no way to be sure.

Thanks for information anyone can supply.
darmok
Posts: 119
Joined: Tue Oct 16, 2007 9:14 pm
Location: New Jersey, USA

Re: can computer shutdown during transmission download corrupt disk drive

Post by darmok »

#01: Transmission does normal file i/o. It does not corrupt anything. If you exit Transmission normally then it will have time to write out the last buffers and tell the torrent's trackers that you're exiting normally. If you just kill Transmission then those last buffers are lost - which may later smell like the file on disk is corrupted, but it's not - it just needs (will ask for) a repeat of the last few chunks. And the trackers are left hanging, which are seen by some as an offensive thing, so they might ban you for n hours.

#02: Yes. In fact, OS developers spend quite a bit of time with their state diagrams to ensure that interruptions, such as power hits, don't play havoc with their file systems. *However* SSDs are especially susceptible to bit loss/flipping and block death, so keep good backups.

#03: Yea, that's true. Transmission just does normal C++ file i/o to its own files. Nothing out-of-band, as it were.

#04: I like to shutdown Transmission manually just to ensure it gets all the time it needs to write those last buffers and notify the trackers.

#05: ew Java. heh. Transmission downloads whatever chunks are offered by the seeding peers. That's rarely in numerical/ascending order. As to showing the order, um, I'm not sure. I run the macOS version which has several ways to show what's being received. Not sure about the other builds.

hth,
- Dan.
ThePowerTool
Posts: 14
Joined: Wed May 29, 2013 4:00 pm

Re: can computer shutdown during transmission download corrupt disk drive

Post by ThePowerTool »

Hi Inorganic,

I'll add one point specific to hardware:

You didn't mention what type of system or the manufacturer of your 2TB M.2.

You also didn't mention critical information regarding what happened immediately after you pressed your LM button--did you a) twiddle thumbs while watching system go through normal shutdown processes (or walk away leaving this process to complete) 8-) ; or b) power-off system, terminate with extreme prejudice, BRS! ? :shock:

Most M.2 SSDs use SLC caching. This means if the drive isn't powered-down properly (flushing the cache) you may experience corruption.

You may have properly shut-down. I'm primarily adding this for anyone that likes to skip the normal power-down which, with faster systems I suspect may be less of an issue.

Additional note: A properly configured laptop can be shut down, closed, and put in a case with the shutdown continuing normally until it completes--but you need to know your laptop is properly configured to do this.
inorganic
Posts: 2
Joined: Tue May 23, 2023 10:12 pm

Re: can computer shutdown during transmission download corrupt disk drive

Post by inorganic »

ThePowerTool wrote: Thu May 25, 2023 6:08 pm Hi Inorganic,

I'll add one point specific to hardware:

You didn't mention what type of system or the manufacturer of your 2TB M.2.

You also didn't mention critical information regarding what happened immediately after you pressed your LM button--did you a) twiddle thumbs while watching system go through normal shutdown processes (or walk away leaving this process to complete) 8-) ; or b) power-off system, terminate with extreme prejudice, BRS! ? :shock:

Most M.2 SSDs use SLC caching. This means if the drive isn't powered-down properly (flushing the cache) you may experience corruption.

You may have properly shut-down. I'm primarily adding this for anyone that likes to skip the normal power-down which, with faster systems I suspect may be less of an issue.

Additional note: A properly configured laptop can be shut down, closed, and put in a case with the shutdown continuing normally until it completes--but you need to know your laptop is properly configured to do this.

#01: The 02TB SSD is a Samsung 980 PRO M.2 NVME installed in my motherboard. I know about the firmware problem Samsung had with this product, but I installed the new firmware into the Samsung SSD before installing the SSD into my motherboard.

#02: Yes, I was a bad boy ... well, a slightly bad boy. I am usually conservative about such things, but I was in a hurry that time and did not shutdown Transmission before I shutdown Linux Mint v21.10 the usual correct way ... with the LM button at the bottom left of the screen followed by the red shutdown button that appears just above the LM button when you click the LM button.

It is my understanding that apps ... and especially the OS ... are designed so shutting down the OS doesn't cause problems when apps are still running ... especially disastrous problems like this one. I would think an application like Transmission ... that commonly runs for many hours or even many days without stop ... would be designed to tolerate the OS shutting down without the app being told to shutdown first. Truth is, that should not be difficult given the fact Transmission allocates the files and space on disk for the entire size of the files as soon as the files are created. So it shouldn't even be possible to write outside the space already allocated for the files ... even with cache-flush delays and such.

So it seems to me like only an error in how the OS is written could cause this problem. I assume Transmission is not running as a normal users and not as root or superuser, and therefore should not even be able to write directly to /dev/sda or /dev/nvme0n1 where the MBR and partition tables are.

So ... I just don't get it. I don't see how Transmission could cause the error ... and I don't see how the OS could cause the error without some radically stupid code. I mean ... when would an OS write to the partition table AT ALL while running normal apps (and I was not running any oddball apps like "disks" or "gparted" or anything).

#03: I'm not familiar with SLC caching. Can you explain? And more important, does the nature of SLC caching permit very sensitive places like the partition table to be written to?

#04: Yeah, I absolutely NEVER just press the power button on the computer to shutdown!!! Maybe I'm just an old fart who is used to power buttons actually physically shutting off power instantly. I dunno. But I would never turn off power to shutdown a computer! Yikes! :-o

#05: My computer is a new self-built computer with a full tower case, full-size EATX motherboard, AMD 7950x CPU, ASUS ROG Crosshair x670e Extreme motherboard, 64GB DDR5 5600MHz memory sticks, liquid CPU cooler, two 16TB SATA3 HDD, two 08TB SATA3 HDD, 1000W power supply, and 6 x 140mm fans plus 4 x 120mm fans and a 43" 3840x2160 display. So ... not a laptop! Hahaha. OTOH, it is probably true that laptops have more stringent protection against being turned off "rudely" by their owners. :-)

#06: As for Java ... definitely yuck! Hahaha. I'm mostly a C and assembly-language programmer myself.
ThePowerTool
Posts: 14
Joined: Wed May 29, 2013 4:00 pm

Re: can computer shutdown during transmission download corrupt disk drive

Post by ThePowerTool »

If you allowed your Linux Mint to shut down properly while applications were running this is typically not an issue. I do this occasionally with my systems--execute a shutdown and just let the shutdown process handle apps (and specifically Transmission). That means, as I think you described (but just to be clear), you used the Linux Mint shutdown by clicking (as you described) and then watched and waited until your system powered down. If that's what you did then you minimized the potential for problems. That doesn't eliminate problems but it does eliminate the egregious issues from mishandling your computer.

Here's some extra food-for-thought to help with why waiting is so critical (while also addressing some of your questions):

The way HD caching works is discretely isolated from application output. The application executes a write but that isn't a directive to the HD. You can take a look at the OSI Model and spend hours diving into that if you want details. I'll give you the executive summary.

If the OS is halted before the normal shutdown process completes, I can summarize what happened--Murphy's law says I'm correct in this :-)

1. Your app was chugging away (doesn't matter what app and could have been more that one app--not necessarily in your case but follow along with me)
2. Your app executed a "write" statement and other apps may have executed "write" statements.
3. These combined write statements (1 or more from your app and other apps) requested execution of data delivery to the HD which was propagated through the layers of the OSI Model.
4. To keep this simple I'm skipping to the HD which receives a write command at the Physical layer (7 layers down from the Application layer). The 1st thing that happens is the TurboWrite SLC caching in your drive fills up and actual writes to the drive slow--this means the cache remains full and there's a backlog of data.
5. The system was unexpectedly powered off. No OS or app can prevent what happens next in this scenario. The state of the last/most-current write (unless you are very lucky) is unknown. All cache data is lost. All data at the above layers waiting to make it to the cache is (likely) lost. The "likely" is because the exceptions, while they exist, aren't applicable in your specific case.

Once data is lost in that manner, subsequent access to the data that is actually on your hard drive but was impacted by the loss (i.e. "truncated files" and/or possible bad pointers to data) is subject to Murphy's 2nd law: "When Murphy's law is active Murphy gets 200% bonuses on all activities." An extra "gotcha" when Murphy hits a bonus round is if your system launches a process just before you hard-stop. This means multiple processes can suddenly start taking-up CPU cycles (slowing your system) and executing writes (filling the pipeline with write requests unrelated to your primary needs) which dilutes the data you're interested-in making the loss more significant (because instead of your write requests being 100% of the requests and data they can become 50% of the requests and data, or less).

Things that commonly happen given your scenario:
Corrupted boot sector - your system says "can't find the operating system" and requires rewriting the boot strap
Corrupted applications - strange things happen starting one or more applications and typically this is resolved by reinstalling the app or completely removing the app, removing all entries related to the app, and reinstalling from scratch vs. the simple reinstall.
Corrupted data - spreadsheets, documents, etc open and show strange characters and your doctorate thesis is gone or 90% gibberish (but you can still read the section heading (unless they were equally important)), databases kick funny errors and make you think they are working when, in fact, the are not, applications start but then crash or act "funny" or display strange errors. The fix for this ranges from you can recover from back-ups and/or re-enter some information to you can pay someone to recover data for you (lots of $$$$$$$ and no guarantees).

I hope that helps explain things a little and why it's so important to properly shut down any OS and unmount any type of drive before unplugging it and powering it down.
darmok
Posts: 119
Joined: Tue Oct 16, 2007 9:14 pm
Location: New Jersey, USA

Re: can computer shutdown during transmission download corrupt disk drive

Post by darmok »

inorganic wrote: Sun May 28, 2023 12:47 pm #01: The 02TB SSD is a Samsung 980 PRO M.2 NVME installed in my motherboard. I know about the firmware problem Samsung had with this product, but I installed the new firmware into the Samsung SSD before installing the SSD into my motherboard.
Keep good backups!
It is my understanding that apps ... and especially the OS ... are designed so shutting down the OS doesn't cause problems when apps are still running
As part of the OS shutdown process, apps are told to exit. And, normally, the OS then waits for all the apps to finish before continuing the shutdown.

- Dan.
Post Reply