Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.java.programmer > #11788
| From | Robert Klemme <shortcutter@googlemail.com> |
|---|---|
| Newsgroups | comp.lang.java.programmer |
| Subject | Re: How do I scp extremely large files |
| Date | 2012-02-05 23:54 -0800 |
| Organization | http://groups.google.com |
| Message-ID | <7f3940d0-cc97-466d-87bf-7a420e53df5c@gi10g2000vbb.googlegroups.com> (permalink) |
| References | <a7abf79b-a373-4f52-9c1c-6c24d509fbcb@o13g2000vbf.googlegroups.com> |
On 5 Feb., 03:03, Mike <mikesmith...@gmail.com> wrote: > I need to copy extremely large files (30-100G) from a remove server to > a machine where my code will be running. I have code in place now > that uses an sftp connection to scan the file directory watching for > files to show up for me to copy. My question is how do I read in > chunks of these large files and write them out in chunks? I cannot > hold the bytes in memory obviously. Any help or pseudo code is > greatly appreaciated! I am not sure I understand your question properly. Any Java library which implements sftp or scp's protocols will have a means to copy remote files or at least open remote files and obtain an InputStream or Channel, from which you can read in chunks and store data locally. If you need to do the transfer on a regularly basis maybe rsync is better than cooking your own version of it. Kind regards robert
Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
How do I scp extremely large files Mike <mikesmith813@gmail.com> - 2012-02-04 18:03 -0800
Re: How do I scp extremely large files Robert Klemme <shortcutter@googlemail.com> - 2012-02-05 23:54 -0800
Re: How do I scp extremely large files Mike <mikesmith813@gmail.com> - 2012-02-06 12:54 -0800
Re: How do I scp extremely large files Lew <lewbloch@gmail.com> - 2012-02-06 17:21 -0800
Re: How do I scp extremely large files Mike <mikesmith813@gmail.com> - 2012-02-08 16:29 -0800
Re: How do I scp extremely large files Daniel Pitts <newsgroup.nospam@virtualinfinity.net> - 2012-02-08 21:19 -0800
Re: How do I scp extremely large files Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-02-07 08:12 -0500
Re: How do I scp extremely large files Arne Vajhøj <arne@vajhoej.dk> - 2012-02-07 17:02 -0500
Re: How do I scp extremely large files Eric Sosman <esosman@ieee-dot-org.invalid> - 2012-02-07 20:47 -0500
Re: How do I scp extremely large files Arne Vajhøj <arne@vajhoej.dk> - 2012-02-07 20:56 -0500
Re: How do I scp extremely large files Robert Klemme <shortcutter@googlemail.com> - 2012-02-07 22:50 +0100
Re: How do I scp extremely large files Arne Vajhøj <arne@vajhoej.dk> - 2012-02-07 17:12 -0500
Re: How do I scp extremely large files Arne Vajhøj <arne@vajhoej.dk> - 2012-02-07 17:09 -0500
Re: How do I scp extremely large files Arne Vajhøj <arne@vajhoej.dk> - 2012-02-07 17:10 -0500
Re: How do I scp extremely large files Roedy Green <see_website@mindprod.com.invalid> - 2012-02-08 22:33 -0800
Re: How do I scp extremely large files Roedy Green <see_website@mindprod.com.invalid> - 2012-02-08 22:37 -0800
csiph-web