Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #4828 > unrolled thread

c# .net 4 unable to move file - file in use by another process (by itself)

Back to article view | Back to comp.programming

Contents

  c# .net 4 unable to move file - file in use by another process (by itself) lachlann@gmail.com - 2014-10-16 08:08 -0700
    Re: c# .net 4 unable to move file - file in use by another process (by itself) Kaz Kylheku <kaz@kylheku.com> - 2014-10-16 15:41 +0000
      Re: c# .net 4 unable to move file - file in use by another process (by itself) lachlann@gmail.com - 2014-10-16 12:59 -0700
    Re: c# .net 4 unable to move file - file in use by another process (by itself) Justin Marshall <lachlann@gmail.com> - 2015-03-24 12:12 -0700
      Re: c# .net 4 unable to move file - file in use by another process (by itself) CarolHudson <CarolHudson0@yahoo.com> - 2015-04-22 21:05 +0200
    Re: c# .net 4 unable to move file - file in use by another process (by itself) Queego <tonyhecht@gmail.com> - 2015-08-14 19:14 -0500

#4828 — c# .net 4 unable to move file - file in use by another process (by itself)

I am having a very strange issue with the System.IO.File.Move operation. On subsequent moves i get an error "file is in use by another process" using process explorer it is my application that is holding the lock. 

This is a .net service that is supposed to process files, at the start of the program it loads the list of files and then grabs the first X files and renames them to .pending. it is then supposed to transform these files and when finished moves them to an 'archive' folder. however if there is a problem with the transform or transforming and an exception is raised its supposed to rename the file to '.failed'. it is this piece that is failing.

When i attempt to rename the file from .pending -> .failed i get an error "file is in use by another process". originally this service was multi-threaded so i assumed it was a collision with other threads however upon further investigation i've made it single-threaded and it is still failing. it even happens when stepping through the code using the debugger (VS2010). 

I've moved all "file" logic into 3 methods:
readfromfile
writetofile
movefile

in each of these 3 methods i've added 3 debugging lines:
1) Enter method
2) finish method (success) or exception
3) exit method
i record the timestamp, thread and details of the move.

If i deliberately point it at a bad transform the read/write methods are bypassed so the only method involving the filesystem is the "movefile" method.

the move method checks for file in use error and waits 1 second and retries 3x. interestingly when it gives up on the 3rd attempt i'm able to rename the file without issues using windows explorer however i am unable to delete the file unless the service is turned off.

i'm testing on a windows 7 (64-bit) machine with an NTFS filesystem.


here is the full code for the movefile method

 private bool AttemptMoveFile(FileInfo OldFile, string NewName, int iRetries, string FileTypeName)
        {
            try
            {
                IOException FinalException = null;
                FileInfo NewFile = new FileInfo(NewName);
                alertError(System.Threading.Thread.CurrentThread.ManagedThreadId + "," + AppDomain.GetCurrentThreadId().ToString() + " " + DateTime.Now.ToString() + " Enter Move: " + OldFile.FullName + " --> " + NewName);
                try
                {
                    // attempt to recover 3 times
                    for (int i = 0; i < iRetries; i++)
                    {
                        try
                        {
                            if (OldFile.Exists)
                            {
                                if (!NewFile.Directory.Exists)
                                {
                                    NewFile.Directory.Create();
                                }

                                try
                                {
                                    if (NewFile.Exists)
                                    {
                                        alertError("WARNING! Overwriting new " + FileTypeName + " file: " + NewName + " with " + OldFile.FullName);
                                        NewFile.Delete();
                                    }
                                    alertError("DEBUG! Moving File " + OldFile.FullName + "-->" + NewName);
                                    //NewFile = null;
                                    File.Move(OldFile.FullName, NewName);
                                    //OldFile.MoveTo(NewName);
                                }
                                // if file doesn't exist, return failed to moved
                                catch (FileNotFoundException)
                                {
                                    return false;
                                }
                            }
                            else
                            {
                                return false;
                            }
                            return true;
                        }
                        catch (PathTooLongException e)
                        {
                            PathTooLongException ex = new PathTooLongException("Source/Destination file path  is too long, OldFile: " + OldFile + " (len=" + OldFile.Length.ToString() + "), NewFile: " + NewFile + " (len=" + NewFile.Length.ToString() + "). The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.", e);
                            ex.Data.Add("Action", "Move");
                            ex.Data.Add("NewFile", NewFile);
                            ex.Data.Add("OldFile", OldFile);
                            throw ex;
                        }
                        catch (IOException e)
                        {
                            if (IsFileInUseException(e))
                            {
                                e.Data.Add("Action", "Move");
                                e.Data.Add("NewFile", NewFile);
                                e.Data.Add("OldFile", OldFile);
                                FinalException = e;
                                // wait 1 second and try again
                                Pause(1000);
                            }
                            else
                            {
                                e.Data.Add("Action", "Move");
                                e.Data.Add("NewFile", NewFile);
                                e.Data.Add("OldFile", OldFile);
                                FinalException = e;
                                throw;
                            }
                        }
                    }


                }
                // ignore file not found errors
                catch (FileNotFoundException)
                { }

                if (FinalException != null)
                {
                    if (IsFileInUseException(FinalException))
                    {
                        // do something
                        throw new IOException(sMessage, FinalException);
                    }
                    else
                    {
                        throw FinalException;
                    }
                }

                return false;
            }
            finally
            {
                alertError(System.Threading.Thread.CurrentThread.ManagedThreadId + "," + AppDomain.GetCurrentThreadId().ToString() +" "+DateTime.Now.ToString()+ " Exit Move: " + OldFile.FullName + " --> " + NewName);
            }
        }

here is the log excerpt, as you will see there are a few calls to the "write" method however they are the 'output' of the transform and to a completely different file. i've added comments prefixed by ** to help 

2014_10_15_LNOSPUPDebug.log is the debug filename, the next # is the filename.

** first move xml -> pending
2014_10_15_LNOSPUPDebug.log	14491	* 21:19:01.5862608 [5-Timer] * 5,13328 10/15/2014 9:19:01 PM Enter Move: C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.xml --> C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending	Move
2014_10_15_LNOSPUPDebug.log	14493	* 21:19:01.5882608 [5-Timer] * DEBUG! Moving File C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.xml-->C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending	File
2014_10_15_LNOSPUPDebug.log	14495	* 21:19:01.5912608 [5-Timer] * 5,13328 10/15/2014 9:19:01 PM Exit Move: C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.xml --> C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending	Move

** giving up due to bad config
2014_10_15_LNOSPUPDebug.log	14497	* 21:19:01.5982608 [5-Timer] * Failed to Load File before failure limit: C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending	File

** writing a failure response to a DIFFERENT file
2014_10_15_LNOSPUPDebug.log	14499	* 21:19:01.5992608 [5-Timer] * 5,13328 10/15/2014 9:19:01 PM Enter Write: C:\LNOSDataFiles\BOLRouteImport\ARCH\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Response	File
2014_10_15_LNOSPUPDebug.log	14501	* 21:19:01.6032608 [5-Timer] * 5,13328 10/15/2014 9:19:01 PM Finished Write: C:\LNOSDataFiles\BOLRouteImport\ARCH\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Response	File
2014_10_15_LNOSPUPDebug.log	14503	* 21:19:01.6042608 [5-Timer] * 5,13328 10/15/2014 9:19:01 PM Exit Write: C:\LNOSDataFiles\BOLRouteImport\ARCH\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Response	File

** recognition of the issue
2014_10_15_LNOSPUPDebug.log	14505	* 21:19:01.6062608 [5-Timer] * File failed to be processed: C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending, Response: Transform is configured but is invalid: C:\LNOSDataFiles\Transforms\WRtoRP_20140925_bad.xsl	File

** attempt to "fail" the file, this is what fails
2014_10_15_LNOSPUPDebug.log	14507	* 21:19:01.6702608 [5-Timer] * 5,13328 10/15/2014 9:19:01 PM Enter Move: C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending --> C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.Failed	Move

** try 3 times to rename
2014_10_15_LNOSPUPDebug.log	14509	* 21:19:01.6712608 [5-Timer] * DEBUG! Moving File C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending-->C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.Failed	File
2014_10_15_LNOSPUPDebug.log	14511	* 21:19:02.6872608 [5-Timer] * DEBUG! Moving File C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending-->C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.Failed	File
2014_10_15_LNOSPUPDebug.log	14513	* 21:19:03.7092608 [5-Timer] * DEBUG! Moving File C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending-->C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.Failed	File
2014_10_15_LNOSPUPDebug.log	14515	* 21:19:23.5912608 [5-Timer] * 5,13328 

** give up - (finally) logging
10/15/2014 9:19:23 PM Exit Move: C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending --> C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.Failed	Move

** logging of the file in use exception from the 3 attempts
2014_10_15_LNOSPUPDebug.log	14517	* 21:19:23.5922608 [5-Timer] * Error! A final file could not be failed: C:\LNOSDataFiles\BOLRouteImport\IN\RP\ExportCaseTestAuditFailure-bad.BAT-L00704-141015211901585-5-PreProcess.Pending, System.IO.IOException ---> System.IO.IOException: The process cannot access the file because it is being used by another process.	file




I've been fighting this for several weeks and I am completely stumped as to what could be happening.

[toc] | [next] | [standalone]

#4829

On 2014-10-16, lachlann@gmail.com <lachlann@gmail.com> wrote:
> When i attempt to rename the file from .pending -> .failed i get an error
> "file is in use by another process". originally this service was
> multi-threaded so i assumed it was a collision with other threads however
> upon further investigation i've made it single-threaded and it is still
> failing. it even happens when stepping through the code using the debugger
> (VS2010). 
>
> I've moved all "file" logic into 3 methods:
> readfromfile
> writetofile
> movefile

Who closes the handle to release the file? Is it possible that you might be
leaving that responsibility to a finalization performed by the garbage
collector?

[toc] | [prev] | [next] | [standalone]

#4830

On Thursday, October 16, 2014 11:41:48 AM UTC-4, Kaz Kylheku wrote:
> On 2014-10-16, lachlann@gmail.com <lachlann@gmail.com> wrote:
> 
> > When i attempt to rename the file from .pending -> .failed i get an error
> 
> > "file is in use by another process". originally this service was
> 
> > multi-threaded so i assumed it was a collision with other threads however
> 
> > upon further investigation i've made it single-threaded and it is still
> 
> > failing. it even happens when stepping through the code using the debugger
> 
> > (VS2010). 
> 
> >
> 
> > I've moved all "file" logic into 3 methods:
> 
> > readfromfile
> 
> > writetofile
> 
> > movefile
> 
> 
> 
> Who closes the handle to release the file? Is it possible that you might be
> 
> leaving that responsibility to a finalization performed by the garbage
> 
> collector?

that should all be wrapped into the .NET 4 File.Move logic, i am not at any point getting the handle i'm simply providing a "from" and "to" filename. the Core method handles all the handles. No pun intended :)

[toc] | [prev] | [next] | [standalone]

#5354

Well we eventually identified root cause, it was in a completely unrelated section of code (error notification). it appears the SMTP class requires that you explicitly close the attachments once you are done, if you don't it will hang onto them indefinitely even after the class that instantiated them terminates.

[toc] | [prev] | [next] | [standalone]

#5410

On 25.03.2015 01:09, Julio Di Egidio wrote:
> On Tuesday, March 24, 2015 at 7:12:17 PM UTC, Justin Marshall wrote:
>
>> Well we eventually identified root cause, it was in a completely unrelated section of code (error notification). it appears the SMTP class requires that you explicitly close the attachments once you are done, if you don't it will hang onto them indefinitely even after the class that instantiated them terminates.
>
> In .NET 4, SmtpClient, AttachmentBase, MailMessage, and other classes from the System.Net.Mail namespace implement the IDisposable interface (check the docs).  To correctly use classes that implement IDisposable, you should use the "disposable patter" (the using statement), and, if appropriate, you can encapsulate the whole thing in a custom class of yours that itself implements IDisposable.
>
> HTH,
>
> Julio
>
Julio, you are blind, you are a moron, that's not how it works!

[toc] | [prev] | [next] | [standalone]

#5626

Actually, this "disposable patter" solution worked great for me.  So thanks Julio! (et al).  We'd spent weeks trying to figure out why we were getting locked files.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.programming

csiph-web