Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.basic.visual.misc > #3578

Resume Download Curl

Show all headers | View raw

cURL and Wget are popular web interface command line utilities that among their many features can resume interrupted or canceled downloads.cURL is a library with a command-line frontend, while Wget is a command line tool.


After that, the download is resumed and completes - but the file is corrupt. This has happened multiple times on different devices, which have in common that they are in China and have poor connection (the curl download takes longer).



resume download curl

Download https://0hydque-0stagre.blogspot.com/?r=2x0VdX 






Just for fun, I had looked into getting my CV into the terminal, such that a quick curl would make it available to anyone. Like a (optionally self-hosted) terminal-LinkedIn profile if you will. I was unable to find anything ready-made (only a couple repos of people rolling their own for their own domains and use cases), so off I went to implement this critical app millions had been aching for for decades:


In summary, you (re-)create your CV as a JSON file according to the JSON Resume schema, put that into a gist and off you go! curl -L ancv.io/$USERNAME using your own GitHub $USERNAME now works. The README contains more details on this, including self-hosting it instead of having it GitHub-hosted in a public gist. See this README section.


This behavior is 100% consistent. I can call the 3rd party site and return any number of times within the 120 second window after authentication and the authenticated session is always resumed. But as soon as I return after 120 seconds, a new session is started.


With the Unix curl command, you can perform basically every web request that might come to your mind. It can even download files to your disk. But what can you do if you started a download in the size of a 3-storey wedding cake, and it cancelled at 87%? curl has got you covered.


Changes: fetch_blob function uses a temporary file for its first connection. It then retrieves 30x HTTP redirect from this. It attempts a header retrieval on the final url and checks whether the local copy has the full file. Otherwise, it will begin a resume curl operation. The calling function which passes it a valid token has a loop surrounding retrieving a token, and fetch_blob which ensures the full file is obtained.


Hi,

We are seeing this problem at the moment. I thought it should be fixed already. Can anyone confirm that resume should now work when pulling from a private account on dockerhub? Any suggestions on why it may not be working for us?

Thanks

Ed






When using an asynchronous request, the system stores the products for a period ( 7 days) before deletion. This allows the final user to be able to resume a download process from an already existing request. This can be a useful functionality if the whole requested data to be downloaded has a large size and the internet connection is not very stable. Currently this functionality is only available by using Firefox browser or curl.


I have recently been trying to download an ISO of a certain Linux distro, but Ifind myself at the wrong end of a crappy connection. I tried downloading it withboth Firefox and Chrome but both choked very early in the process, the currentspeed would go down to 0, and after a while the connection would just cutoffwith no way to resume the download from where I left off.


Fear not because cURL comes to the rescue. After typing Up-Arrow andEnter more time than I care to admit, I decided to let the computer do thehard work and automate this whole thing. So I created a small Bash script calledacurl.sh (for Automated cURL) that looks like this:


The code above expects you to pass the file to download as a parameter, which wehave in the $1 variable. Then we tell curl to download it and write it toour local disk with the same name as the remote file (this is option -O). Wealso tell curl to resume a previous file download at a certain offset (this isdone via option -C and giving it the value - will allow curl toautomatically figure out how to resume the transfer). The next step is tocapture curl error codes on exit. This is done by assigning the value of $?(which has the value of the most recent exit code) to our variable $ec. If$ec is greater than 0 then we try the download with curl again, otherwise wehave a successful download.


Alternatively one could install Chromium (since auto-resume add-ons exist). However I guess this would also make fingerprinting easier and add a whole level of possible attack vectors (flatpak, Chromium, the add-on itself, etc.)


I am not aware into any research of this. curl is far less complex than Tor Browser. I would be surprised if fingerprinting curl is easier than Firefox / Tor Browser which has far more fingerprinting surface.


Below is an example of creating a PAT in the UI:

 



Token authentication is best used for any programmatic use of Ansible Tower's API, such as Python scripts or tools like curl. See the example for a personal access token (PAT) below:



Curl Example



First, create an OAuth 2 token without an associated Application; in other words, a personal access token. In this example, we will do so through the API with curl. 




Web browsers, curl, and wget all support resuming downloads. If your download stops before it is complete, your web browser can generally pick up where the download left off. Some of the CLI web tools support this too through special command line switches. But, Invoke-WebRequest and Invoke-RestMethod have not supported this. If you try to use a partial file as the -OutFile, the web cmdlets will start over downloading the file and overwrite the existing file.


The resume feature only works on the size of the file. It does not do any validation to ensure that the file supplied in -OutFile is the same as the remote file you are trying to download. It is up to you to ensure you are trying to resume the correct file.


If your connection drops while downloading a file, you can resume the download with the -C- option. This is an especially useful feature when downloading large sized files, ex DVD ISO files, or MP4 video files. This way, if your connection drops when downloading a file, you can resume the download instead of starting from scratch, using, for example:

 curl -C- -O


When attempting to download a file or gather other information using curl, you may discover that the target site may be designed to block curl. In this case, it is useful to emulate a browser, such as Firefox, to return the information you are looking for. To do this, use the following command:


Another important feature of curl is its ability to transfer files. This is useful when interacting with servers through the command line, particularly if you are trying to take advantage of potential vulnerabilities. To access a protected FTP server, use the -u option to specify the username and password:


I recently had to copy about 20 GB of data split between about 20 files from my laptop to an external hard drive. Since this operation takes quite a while (at 560kb/s), I was wondering if there was any way to pause the transfer and resume it later (in case, I need to interrupt the transfer). As a side question, is there any performance difference between copying from the terminal vs copying from Nautilus?


I can answer only the part about the difference between terminal and nautilus. I have checked several times this. It appears copying from terminal is faster than any graphical way like nautilus. At least in my case, when i copy about 32GB of info, it takes about 3 to 5 minutes less from terminal than nautilus from a 25 minute copy. that is about 10%-20% of the time. About a way to resume a copy to your external unit i think curl or rsync would resume a failed copy. At least with cp command. I have not tried this myself (I will try tomorrow) but this two might work.

 35fe9a5643

Back to comp.lang.basic.visual.misc | Previous | Next | Find similar

Thread

Resume Download Curl Giselle Korsak <korsakgiselle@gmail.com> - 2024-01-01 19:57 -0800

csiph-web