Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.rc > #364646 > unrolled thread

Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop

Back to article view | Back to linux.debian.bugs.rc

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop Bud Heal <budheal508@gmail.com> - 2024-06-04 13:00 +0200
    Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop Ben Hutchings <ben@decadent.org.uk> - 2024-07-31 20:50 +0200

#364646 — Bug#1063754: fat-modules: SD corruption upon opening file on Linux desktop

[Multipart message — attachments visible in raw view] — view raw

I filled up the (micro)SDs, including the current one I have been scanning
into, the one that I set aside as probably defective and the pair I bought
for testing. I will append the bash script. As I look again, I read that
you wanted this done on Windows but I did not use cygwin because putting
Windows into the mix could affect the steps that invoke the data
corruption. Instead, I installed Debian bookworm without a graphical
desktop (unchecked anything except system and ssh). One also activates the
network interface and DHCP. That way, the SD and its folders will not be
affected by adding thumbnails, which I have identified as a way to avoid
the problem.

After satisfying that step, I tried a SD without using the wand scanner.
This newest laptop does not have a SD slot so I used a USB device. I
enabled showing thumbnails. Bookworm's behavior is different from observed
on buster and (probably) bullseye: when one copies a folder to the
computer, and opens on the desktop during copying: (1) the folder on the SD
quickly displays the thumbnails (serially from the top) but (2) the
destination folder opens, begins to show thumbnails but is prone to
continue copying but stopping placing thumbnail. If the SD's folder is
opened, no more thumbnails are added on the computer until the copying is
finished. If the SD's folder is not opened, the thumbnails may stop or may
continue, but there is an extended pause without any indication that the
last two files have been copied, until the progress dialog is closed and
its window is refreshed. If neither folder is opened during copying, the
copying progresses without such pauses and one can then open the folders
and watch the thumbnails be created.

Since these jpeg files are scanned from text, those thumbnails are not
informative, unless they reflect that the image contents have been
disturbed.

The next step would be check this effect on earlier releases but I will not
do this immediately.

You can refer to the script I used to verify these SDs, or it may be
helpful. I used it to fill a SD with folders, each with the same 200
images, for testing, one folder at a time. Otherwise, it will exercise its
targets on the off chance that their capacities are not as advertised.

#!/bin/bash
folderpa="/mnt/DCIM/100MEDIA"
foldernm=`echo $folderpa | sed -e 's/^\(\/[^/]*\)*\///g'`
folderdu=`du $folderpa | awk '$1 ~ /^[0-9]+$/ {print $1}'`
deviceso=`df /mnt --output=source | awk '$1 ~ /\/dev\// {print $1}'`
devicesp=`df --output=avail $deviceso| awk '$1 ~ /^[0-9]+$/ {print $1}'`
fldrcnt=$((0))
filecnt=$((1))
index=$((1))
for file in `find $folderpa -type f|sed -e 's/^\(\/[^/]*\)*\///g'`
do
#echo $file
srcsha[$index]=`shasum $folderpa/$file | sed -e 's/ *[^ ]*$//'`
#echo "${srcsha[$index]} index $index"
index=$(($index+1))
done
count=$(($index))
while [[ $devicesp -gt $folderdu ]]
do
# mock up the new folder name
((fldrcnt++))
printf -v fldrnew "%s%s%d" "/mnt/" $foldernm $fldrcnt
mkdir $fldrnew
index=$((1))
for file in `find $folderpa -type f | sed -e 's/^\(\/[^/]*\)*\///g'`
do
#echo "find file $file"
cp $folderpa"/"$file $fldrnew"/"$file
cmp "$folderpa""/""$file" "$fldrnew""/""$file"
cmperror=$?
if [[ $cmperror -ne 0 ]]
then
echo "comparison mismatch $fldrnew/$file"
fi
savsha=`shasum $fldrnew/$file | sed -e 's/ *[^ ]*$//'`
#echo "$index $fldrnew/$file $savsha"
#echo "${srcsha[$index]}"
if [[ $savsha != ${srcsha[$index]} ]]
then
echo "sha sum $fldrnew/$file mismatches"
fi
((index++))
done
#read -p "Folder $fldrcnt ?" okay
#if [[ $okay != "" ]]
#then
#exit
#fi
devicesp=`df --output=avail $deviceso | awk '$1 ~ /[0-9]+/ {print $1}'`
done
echo "device space $devicesp"
echo "folder space $folderdu"
echo "folders created $fldrcnt"
# Imagine that the target device is posing at a higher space than the
hardware
# Perhaps the newest files have overwritten the earlier so the unsuspecting
# user can access the last-written files but the earliest files are lost.
# In order to detect that, go through the SHA sums again from first to last
fldridx=$((1))
echo "Now comparing SHA sums again, reading each copied files again"
while [[ $fldridx -le $fldrcnt ]]
do
printf -v fldr "%s%s%d" "/mnt/" $foldernm $fldridx
#echo Comparing SHA sums for $fldr
((fldridx++))
index=$((1))
for file in `find $folderpa -type f | sed -e 's/^\(\/[^/]*\)*\///g'`
do
savsha=`shasum $fldr/$file | sed -e 's/ *[^ ]*$//'`
if [[ $savsha != ${srcsha[$index]} ]]
then
echo "sha sum $fldr/$file mismatches"
fi
((index++))
done
done
echo "Done comparing SHA sums starting from earliest to latest files
created"
#

On Wed, May 29, 2024 at 3:15 PM Ben Hutchings <ben@decadent.org.uk> wrote:

> Control: tag -1 unreproducible moreinfo
>
> Hi Bud,
>
> Please understand that when I aksed you to carry out a specific test, I
> had a good reason for doing exactly that.  The tests that you have
> carried out so far, while useful, don't provide the same information.
>
> Ben.
>
> --
> Ben Hutchings
> Always try to do things in chronological order;
> it's less confusing that way.
>
>

[toc] | [next] | [standalone]

#368458

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2024-06-04 at 06:51 -0400, Bud Heal wrote:
> I filled up the (micro)SDs, including the current one I have been scanning
> into, the one that I set aside as probably defective and the pair I bought
> for testing. I will append the bash script.

I appreciate that you've scripted this, but you missed step 2: remove
and reinsert the card.  Without that, the script may just be reading
files out of the cache if your system has enough RAM.  In the script
you could alternately flush the cache:

    echo 1 > /proc/sys/vm/drop_caches

if you're running it as root.

> As I look again, I read that
> you wanted this done on Windows but I did not use cygwin because putting
> Windows into the mix could affect the steps that invoke the data
> corruption.

I wanted you to do this on Windows so that we could see if the same
corruption could occur.

> Instead, I installed Debian bookworm without a graphical
> desktop (unchecked anything except system and ssh). One also activates the
> network interface and DHCP. That way, the SD and its folders will not be
> affected by adding thumbnails, which I have identified as a way to avoid
> the problem.

So disabling automatic thumbnail generation mitigates the problem, but
we already knew that, didn't we?

> After satisfying that step, I tried a SD without using the wand scanner.
> This newest laptop does not have a SD slot so I used a USB device. I
> enabled showing thumbnails. Bookworm's behavior is different from observed
> on buster and (probably) bullseye: when one copies a folder to the
> computer, and opens on the desktop during copying: (1) the folder on the SD
> quickly displays the thumbnails (serially from the top) but (2) the
> destination folder opens, begins to show thumbnails but is prone to
> continue copying but stopping placing thumbnail. If the SD's folder is
> opened, no more thumbnails are added on the computer until the copying is
> finished. If the SD's folder is not opened, the thumbnails may stop or may
> continue, but there is an extended pause without any indication that the
> last two files have been copied, until the progress dialog is closed and
> its window is refreshed. If neither folder is opened during copying, the
> copying progresses without such pauses and one can then open the folders
> and watch the thumbnails be created.
[...]

So one or both of:

- A change in the way this newer version of GNOME (or whichever desktop
environment you're using) does thumbnail generation 
- Using USB storage instead of an SD controller

mitigates the problem, but we don't know which.

Ben.

-- 
Ben Hutchings
Any sufficiently advanced bug is indistinguishable from a feature.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.rc

csiph-web