Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #82553 > unrolled thread

Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop Bud Heal <budheal508@gmail.com> - 2024-05-23 13:10 +0200
    Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop Bud Heal <budheal508@gmail.com> - 2024-08-07 06:10 +0200
      Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop Ben Hutchings <ben@decadent.org.uk> - 2024-10-09 19:10 +0200

#82553 — Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop

[Multipart message — attachments visible in raw view] — view raw

I apologize, I misinterpreted "report back here" but I did not look at the
distribution list or grab "reply all".

I prepared to follow the suggestions, although I was not going to risk that
particular microSD unless it was clearly needed. As I prepared to dd to new
a new SD, I realized that asking a desktop computer to copy many files was
already taking us out the reasonable possibility of replicating the error,
because that required (1) an external filesystem (2) thumbnails on and (3)
letting Linux copy the files while the thumbnails are being updated. As I
was writing the bug report, I tried turning thumbnails off and noted that
was a workaround. As far as the bug itself, it looks like a race condition
or maybe a buffer collision. One can literally watch the corruption in
action. It is likely to succeed if there are only a few files to copy.

In order to validate the SD's size, I used dd to copy the raw image to
/dev/null, reporting: 331914983424 bytes (32 GB, 30 GiB) copied, 815 s,
39.1 MB/s.This and the other SD I bought with it are on a well-known brand
and I have used them extensively. I delete the offending files, rescan,
give it to Windows or Linux without thumbnails on, and the files are
intact. I can insert the SD into the laptop's slot and the same problem
occurs. If I move the files (in a folder) through the network, or any
manner other than on the SD's FAT32 filesystem, all is well.

As I wrote, the compressed raw image (starting on a new microSD) was still
almost 500MB and I was unable to upload it.

On Wed, May 15, 2024 at 4:18 PM Ben Hutchings <ben@decadent.org.uk> wrote:

> Control: tag -1 moreinfo
>
> On Mon, 12 Feb 2024 03:47:14 -0500 Bud Heal <budheal508@gmail.com>
> wrote:
> [...]
> > I scanned a few pages, then plugged into the USB cable, at which
> point
> > the wand shows "USB" and Bullseye mounts the SD as a USB disk. The
> first
> > time on Bullseye, it worked okay. The second and afterwards, not so
> > okay.
> >
> > The key before Bullseye was the thumbnails would show all black,
> usually
> > at the bottom of a folder. The images before the all black ones,
> usually
> > were apparently unaffected. By memory, there were screwy problems but
> > nothing like now.
> >
> > Under Bullseye, just inserting or mounting the SD leads to
> corruption.
> > The files are are "readable" but not as proper images. When I insert
> the
> > SD into a Windows (10) laptop, it asks to check the filesystem and
> will
> > toss the unreadable images and leave the ones that are all black.
> [...]
>
> If I experienced these problems, I would assume that my SD card was
> faulty (or even a counterfeit that has less storage than it claims to).
>
> Since the card appeared to work under Windows, please could you test it
> more thoroughly under Windows:
>
> 1. Copy files onto the card to fill it up.
> 2. Remove and reinsert the card.
> 3. Compare the files on the card with the files you copied from.
>
> and report back here.
>
> Ben.
>
> --
> Ben Hutchings
> Anthony's Law of Force: Don't force it, get a larger hammer.
>
>

[toc] | [next] | [standalone]

#83331

Ben Hutchings wrote:
> On Tue, 2024-06-04 at 06:51 -0400, Bud Heal wrote:
>> I filled up the (micro)SDs, including the current one I have been scanning
>> into, the one that I set aside as probably defective and the pair I bought
>> for testing. I will append the bash script.
> I appreciate that you've scripted this, but you missed step 2: remove
> and reinsert the card.  Without that, the script may just be reading
> files out of the cache if your system has enough RAM.
Do I read this clearly enough - that you have not tried this either? I 
gave bug reports mentioning three devices - a scanner saving to a 
microSD, a builtin SD read/writer, and transfer to Windows 10 to 
confirmation and comparison, innumerable times.

Simply removing and reinserting the microSC card can lead to 
irrecoverable corruption requiring a reformat. At best it will lead to 
loss of the last set of scans, prior to inserting the card into the 
Debian system.
>   In the script
> you could alternately flush the cache:
>
>      echo 1 > /proc/sys/vm/drop_caches
>
> if you're running it as root.
You already have a bug description showing that moving the SD (FAT32) to 
Windows confirm the loss of data. Should I invoke /bin/sync too? 
Instead, I let the desktop inform me that it is safe to remove the device.

>> As I look again, I read that
>> you wanted this done on Windows but I did not use cygwin because putting
>> Windows into the mix could affect the steps that invoke the data
>> corruption.
> I wanted you to do this on Windows so that we could see if the same
> corruption could occur.
As I read, I was to copy files over and over to see if the device could 
contain 32GB files, because there is a claimed horde of forged devices. 
I supplied a bash script which would create a test bed without putting 
thumbnails anywhere. Then, when the image files are mounted on the 
Debian system, the bug is primed. I wrote the script to replicate your 
request - not to fill the device completely. For that, I refer to the dd 
steps I reported, months ago.
>
>> Instead, I installed Debian bookworm without a graphical
>> desktop (unchecked anything except system and ssh). One also activates the
>> network interface and DHCP. That way, the SD and its folders will not be
>> affected by adding thumbnails, which I have identified as a way to avoid
>> the problem.
> So disabling automatic thumbnail generation mitigates the problem, but
> we already knew that, didn't we?
>
>> After satisfying that step, I tried a SD without using the wand scanner.
>> This newest laptop does not have a SD slot so I used a USB device. I
>> enabled showing thumbnails. Bookworm's behavior is different from observed
>> on buster and (probably) bullseye: when one copies a folder to the
>> computer, and opens on the desktop during copying: (1) the folder on the SD
>> quickly displays the thumbnails (serially from the top) but (2) the
>> destination folder opens, begins to show thumbnails but is prone to
>> continue copying but stopping placing thumbnail. If the SD's folder is
>> opened, no more thumbnails are added on the computer until the copying is
>> finished. If the SD's folder is not opened, the thumbnails may stop or may
>> continue, but there is an extended pause without any indication that the
>> last two files have been copied, until the progress dialog is closed and
>> its window is refreshed. If neither folder is opened during copying, the
>> copying progresses without such pauses and one can then open the folders
>> and watch the thumbnails be created.
> [...]
>
> So one or both of:
>
> - A change in the way this newer version of GNOME (or whichever desktop
> environment you're using) does thumbnail generation
> - Using USB storage instead of an SD controller
>
> mitigates the problem, but we don't know which.
Using USB storage does not mitigate the problem. You have a bug 
corrupting a file system and losing data for an unwary user. There may 
be a patch ready to be backported in the latest version, but that is not 
mine to know. I have given you all that is needed to replicate the 
problem. Fill a device full enough to let you open a folder while it is 
generating thumbnails, and test, or not.
>
> Ben.
>

[toc] | [prev] | [next] | [standalone]

#84252

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2024-08-07 at 00:07 -0400, Bud Heal wrote:
> Ben Hutchings wrote:
> > On Tue, 2024-06-04 at 06:51 -0400, Bud Heal wrote:
> > > I filled up the (micro)SDs, including the current one I have been scanning
> > > into, the one that I set aside as probably defective and the pair I bought
> > > for testing. I will append the bash script.
> > I appreciate that you've scripted this, but you missed step 2: remove
> > and reinsert the card.  Without that, the script may just be reading
> > files out of the cache if your system has enough RAM.
> Do I read this clearly enough - that you have not tried this either? I 
> gave bug reports mentioning three devices - a scanner saving to a 
> microSD, a builtin SD read/writer, and transfer to Windows 10 to 
> confirmation and comparison, innumerable times.

But no-one else has reported similar issues, and I don't have the same
devices, which is why I kept asking you to run tests.

I have now tried reproducing this with my own microSD card and reader,
and didn't see this problem.

> Simply removing and reinserting the microSC card can lead to 
> irrecoverable corruption requiring a reformat. At best it will lead to 
> loss of the last set of scans, prior to inserting the card into the 
> Debian system.
[...]
> > So one or both of:
> > 
> > - A change in the way this newer version of GNOME (or whichever desktop
> > environment you're using) does thumbnail generation
> > - Using USB storage instead of an SD controller
> > 
> > mitigates the problem, but we don't know which.
> Using USB storage does not mitigate the problem. You have a bug 
> corrupting a file system and losing data for an unwary user.

I was not suggesting that this was a practical mitigation.  I am trying
to understand what the exact conditions are for this issue to occur.

> There may 
> be a patch ready to be backported in the latest version, but that is not 
> mine to know. I have given you all that is needed to replicate the 
> problem. Fill a device full enough to let you open a folder while it is 
> generating thumbnails, and test, or not.
> 

But I couldn't replicate the problem.

At this point I think it would be helpful to get disk images of a card
that shows this issue:

1. Erase a card completely, to avoid private data being included in the
images.
2. Use the scanner to format and add some non-sensitive images to the
card.
3. Insert the card into a Linux system with automounting disabled (i.e.
no desktop running).
4. Use dd and gzip to create an image of the card before corruption.
5. Start the desktop and verify that the card does gets corrupted from
the state that was imaged.  If not, go back to step 2.
6. Use dd and gzip to create an image of the card with corruption.

I'm guessing the disk images are going to be rather large, in which
case they should not be attached to this bug report.  I can send you a
link privately to upload them.

Ben.

-- 
Ben Hutchings
Klipstein's 4th Law of Prototyping and Production:
                               A fail-safe circuit will destroy others.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web