Copying large quantity of files taking forever

  • Thread starter Thread starter SwitchKat
  • Start date Start date
S

SwitchKat

Guest
Hi, I wanted to throw this question out because after losing over a week of
productivity, I'm not sure where else to turn.

Here's my issue. We're trying to restore a file repository for a project
that has just under 2 million files. We've tried restoring from tape,
copying from another Windows server and even moving from another Windows
server. The problem we run into in every case is that the file copy to the
new system takes off fast and we can see file transfer rates start out
copying 1000's of files per second, but over time it always slows down to a
crawl and we end up where we are now. We are currently transferring one file
every few minutes.

All the files are fairly small (<10MB), gigabit network (although we see the
same performance if we attach the drive locally and just try a copy), and all
are running Windows Server 2000 or 2003 server. The target is always running
2003. We've been monitoring resources, and at this point, both servers
involved show flatlines for their resources being used. My hunch is that it
has something to do with a data block size issue or the file table (maybe
fragmentation?), but I'm not sure. Any ideas?
 
Re: Copying large quantity of files taking forever

SwitchKat wrote:
> Hi, I wanted to throw this question out because after losing over a week of
> productivity, I'm not sure where else to turn.
>
> Here's my issue. We're trying to restore a file repository for a project
> that has just under 2 million files. We've tried restoring from tape,
> copying from another Windows server and even moving from another Windows
> server. The problem we run into in every case is that the file copy to the
> new system takes off fast and we can see file transfer rates start out
> copying 1000's of files per second, but over time it always slows down to a
> crawl and we end up where we are now. We are currently transferring one file
> every few minutes.
>
> All the files are fairly small (<10MB), gigabit network (although we see the
> same performance if we attach the drive locally and just try a copy), and all
> are running Windows Server 2000 or 2003 server. The target is always running
> 2003. We've been monitoring resources, and at this point, both servers
> involved show flatlines for their resources being used. My hunch is that it
> has something to do with a data block size issue or the file table (maybe
> fragmentation?), but I'm not sure. Any ideas?

Give RoboCopy a try

--

/ ) Regards,
/ /_________
_|__|__) Paul Weterings
/ (O_) http://www.servercare.nl
__/ (O_)
____(O_)
 
Re: Copying large quantity of files taking forever

Ya, Robocopy was the first tool we tried. It works fine up until we cross
the 1.25 million files, and it too starts to slow down to a crawl.

"Paul Weterings" wrote:

> SwitchKat wrote:
> > Hi, I wanted to throw this question out because after losing over a week of
> > productivity, I'm not sure where else to turn.
> >
> > Here's my issue. We're trying to restore a file repository for a project
> > that has just under 2 million files. We've tried restoring from tape,
> > copying from another Windows server and even moving from another Windows
> > server. The problem we run into in every case is that the file copy to the
> > new system takes off fast and we can see file transfer rates start out
> > copying 1000's of files per second, but over time it always slows down to a
> > crawl and we end up where we are now. We are currently transferring one file
> > every few minutes.
> >
> > All the files are fairly small (<10MB), gigabit network (although we see the
> > same performance if we attach the drive locally and just try a copy), and all
> > are running Windows Server 2000 or 2003 server. The target is always running
> > 2003. We've been monitoring resources, and at this point, both servers
> > involved show flatlines for their resources being used. My hunch is that it
> > has something to do with a data block size issue or the file table (maybe
> > fragmentation?), but I'm not sure. Any ideas?
>
> Give RoboCopy a try
>
> --
>
> / ) Regards,
> / /_________
> _|__|__) Paul Weterings
> / (O_) http://www.servercare.nl
> __/ (O_)
> ____(O_)
>
 
Re: Copying large quantity of files taking forever


"SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message
news:EFEF754F-402C-438E-9654-0ABC30409346@microsoft.com...
> Hi, I wanted to throw this question out because after losing over a week
> of
> productivity, I'm not sure where else to turn.
>
> Here's my issue. We're trying to restore a file repository for a project
> that has just under 2 million files. We've tried restoring from tape,
> copying from another Windows server and even moving from another Windows
> server. The problem we run into in every case is that the file copy to
> the
> new system takes off fast and we can see file transfer rates start out
> copying 1000's of files per second, but over time it always slows down to
> a
> crawl and we end up where we are now. We are currently transferring one
> file
> every few minutes.
>
> All the files are fairly small (<10MB), gigabit network (although we see
> the
> same performance if we attach the drive locally and just try a copy), and
> all
> are running Windows Server 2000 or 2003 server. The target is always
> running
> 2003. We've been monitoring resources, and at this point, both servers
> involved show flatlines for their resources being used. My hunch is that
> it
> has something to do with a data block size issue or the file table (maybe
> fragmentation?), but I'm not sure. Any ideas?

You don't say where these files reside but if they reside in a single
directory then this is your answer. If you want to keep your machine running
at top speed, don't keep more than 10,000 entries in a single folder. 2
million is two hundred times too many.
 
Re: Copying large quantity of files taking forever

The files are spread out through many directories. I originally thought this
might be an issue too since it also takes a long time to open any of the
folders.

"Pegasus (MVP)" wrote:

>
> "SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message
> news:EFEF754F-402C-438E-9654-0ABC30409346@microsoft.com...
> > Hi, I wanted to throw this question out because after losing over a week
> > of
> > productivity, I'm not sure where else to turn.
> >
> > Here's my issue. We're trying to restore a file repository for a project
> > that has just under 2 million files. We've tried restoring from tape,
> > copying from another Windows server and even moving from another Windows
> > server. The problem we run into in every case is that the file copy to
> > the
> > new system takes off fast and we can see file transfer rates start out
> > copying 1000's of files per second, but over time it always slows down to
> > a
> > crawl and we end up where we are now. We are currently transferring one
> > file
> > every few minutes.
> >
> > All the files are fairly small (<10MB), gigabit network (although we see
> > the
> > same performance if we attach the drive locally and just try a copy), and
> > all
> > are running Windows Server 2000 or 2003 server. The target is always
> > running
> > 2003. We've been monitoring resources, and at this point, both servers
> > involved show flatlines for their resources being used. My hunch is that
> > it
> > has something to do with a data block size issue or the file table (maybe
> > fragmentation?), but I'm not sure. Any ideas?
>
> You don't say where these files reside but if they reside in a single
> directory then this is your answer. If you want to keep your machine running
> at top speed, don't keep more than 10,000 entries in a single folder. 2
> million is two hundred times too many.
>
>
>
 
Re: Copying large quantity of files taking forever

The utility xxcopy.exe is known for baulking at large copy jobs. Perhaps
there is a similar but much higher limit for robocopy. Which version do you
use? It should be at least Version XP010.

If this is a limitation of robocopy.exe then you may be able to walk around
it by splitting up the job into many smaller jobs, using a batch file. I
would try this:
@echo off
for %%a in (0 1 .. 9 a b .. z) do robocopy /s /.. "D:\SourceFolder"
"E:\DestFolder" %%a*.*
robocopy /s /XO "D:\SourceFolder" "E:\DestFolder" *.*

The second line will copy all files starting with 0 .. 9 and a .. z. You
must expand the dots to include the full alphabet - the batch file won't do
it for you! The third line will scoop up any file that was not already
copied by the first line. If the third line slows down to a crawl after a
while then you have to expand the collection of characters in Line 2 to
include all valid ASCII characters. Use charmap.exe to identify them!



"SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message
news:561323C4-1B08-4034-9BB6-63C4089824C6@microsoft.com...
> The files are spread out through many directories. I originally thought
> this
> might be an issue too since it also takes a long time to open any of the
> folders.
>
> "Pegasus (MVP)" wrote:
>
>>
>> "SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message
>> news:EFEF754F-402C-438E-9654-0ABC30409346@microsoft.com...
>> > Hi, I wanted to throw this question out because after losing over a
>> > week
>> > of
>> > productivity, I'm not sure where else to turn.
>> >
>> > Here's my issue. We're trying to restore a file repository for a
>> > project
>> > that has just under 2 million files. We've tried restoring from tape,
>> > copying from another Windows server and even moving from another
>> > Windows
>> > server. The problem we run into in every case is that the file copy to
>> > the
>> > new system takes off fast and we can see file transfer rates start out
>> > copying 1000's of files per second, but over time it always slows down
>> > to
>> > a
>> > crawl and we end up where we are now. We are currently transferring
>> > one
>> > file
>> > every few minutes.
>> >
>> > All the files are fairly small (<10MB), gigabit network (although we
>> > see
>> > the
>> > same performance if we attach the drive locally and just try a copy),
>> > and
>> > all
>> > are running Windows Server 2000 or 2003 server. The target is always
>> > running
>> > 2003. We've been monitoring resources, and at this point, both servers
>> > involved show flatlines for their resources being used. My hunch is
>> > that
>> > it
>> > has something to do with a data block size issue or the file table
>> > (maybe
>> > fragmentation?), but I'm not sure. Any ideas?
>>
>> You don't say where these files reside but if they reside in a single
>> directory then this is your answer. If you want to keep your machine
>> running
>> at top speed, don't keep more than 10,000 entries in a single folder. 2
>> million is two hundred times too many.
>>
>>
>>
 
Re: Copying large quantity of files taking forever

Thanks Pegasus! I didn't even think to script it out and break it up. I'll
give it a try.

"Pegasus (MVP)" wrote:

> The utility xxcopy.exe is known for baulking at large copy jobs. Perhaps
> there is a similar but much higher limit for robocopy. Which version do you
> use? It should be at least Version XP010.
>
> If this is a limitation of robocopy.exe then you may be able to walk around
> it by splitting up the job into many smaller jobs, using a batch file. I
> would try this:
> @echo off
> for %%a in (0 1 .. 9 a b .. z) do robocopy /s /.. "D:\SourceFolder"
> "E:\DestFolder" %%a*.*
> robocopy /s /XO "D:\SourceFolder" "E:\DestFolder" *.*
>
> The second line will copy all files starting with 0 .. 9 and a .. z. You
> must expand the dots to include the full alphabet - the batch file won't do
> it for you! The third line will scoop up any file that was not already
> copied by the first line. If the third line slows down to a crawl after a
> while then you have to expand the collection of characters in Line 2 to
> include all valid ASCII characters. Use charmap.exe to identify them!
>
>
>
> "SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message
> news:561323C4-1B08-4034-9BB6-63C4089824C6@microsoft.com...
> > The files are spread out through many directories. I originally thought
> > this
> > might be an issue too since it also takes a long time to open any of the
> > folders.
> >
> > "Pegasus (MVP)" wrote:
> >
> >>
> >> "SwitchKat" <SwitchKat@discussions.microsoft.com> wrote in message
> >> news:EFEF754F-402C-438E-9654-0ABC30409346@microsoft.com...
> >> > Hi, I wanted to throw this question out because after losing over a
> >> > week
> >> > of
> >> > productivity, I'm not sure where else to turn.
> >> >
> >> > Here's my issue. We're trying to restore a file repository for a
> >> > project
> >> > that has just under 2 million files. We've tried restoring from tape,
> >> > copying from another Windows server and even moving from another
> >> > Windows
> >> > server. The problem we run into in every case is that the file copy to
> >> > the
> >> > new system takes off fast and we can see file transfer rates start out
> >> > copying 1000's of files per second, but over time it always slows down
> >> > to
> >> > a
> >> > crawl and we end up where we are now. We are currently transferring
> >> > one
> >> > file
> >> > every few minutes.
> >> >
> >> > All the files are fairly small (<10MB), gigabit network (although we
> >> > see
> >> > the
> >> > same performance if we attach the drive locally and just try a copy),
> >> > and
> >> > all
> >> > are running Windows Server 2000 or 2003 server. The target is always
> >> > running
> >> > 2003. We've been monitoring resources, and at this point, both servers
> >> > involved show flatlines for their resources being used. My hunch is
> >> > that
> >> > it
> >> > has something to do with a data block size issue or the file table
> >> > (maybe
> >> > fragmentation?), but I'm not sure. Any ideas?
> >>
> >> You don't say where these files reside but if they reside in a single
> >> directory then this is your answer. If you want to keep your machine
> >> running
> >> at top speed, don't keep more than 10,000 entries in a single folder. 2
> >> million is two hundred times too many.
> >>
> >>
> >>
>
>
>
 
Back
Top