fundamentals.download.multiobject_download

*Download resources from a list of URLs.

There are options to rename all the downloaded resource, index the files, set differing download locations and pass basic authentication credentials.*

Author

David Young

Module Contents

Functions

fetch_url

multiobject_download

get multiple url documents and place them in specified download directory/directories

Data

gtimeout

llog

API

fundamentals.download.multiobject_download.fetch_url(entry)[source][source]
fundamentals.download.multiobject_download.gtimeout[source]

10.0

fundamentals.download.multiobject_download.llog = <Multiline-String>[source]
fundamentals.download.multiobject_download.multiobject_download(urlList, downloadDirectory, log, timeStamp=True, timeout=180, concurrentDownloads=10, resetFilename=False, credentials=False, longTime=False, indexFilenames=False)[source][source]

get multiple url documents and place them in specified download directory/directories

Key Arguments

  • urlList – list of document urls

  • downloadDirectory – directory(ies) to download the documents to - can be one directory path or a list of paths the same length as urlList

  • log – the logger

  • timestamp – append a timestamp the name of the URL (ensure unique filenames)

  • longTime – use a longer timestamp when appending to the filename (greater uniqueness)

  • timeout – the timeout limit for downloads (secs)

  • concurrentDownloads – the number of concurrent downloads allowed at any one time

  • resetFilename – a string to reset all filenames to

  • credentials – basic http credentials { ‘username’ : “…”, “password”, “…” }

  • indexFilenames – prepend filenames with index (where url appears in urllist)

Return

  • list of timestamped documents (same order as the input urlList)

Usage

# download the pages linked from the main list page
from fundamentals.download import multiobject_download
localUrls = multiobject_download(
    urlList=["https://www.python.org/dev/peps/pep-0257/","https://en.wikipedia.org/wiki/Docstring"],
    downloadDirectory="/tmp",
    log="log",
    timeStamp=True,
    timeout=180,
    concurrentDownloads=2,
    resetFilename=False,
    credentials=False,  # { 'username' : "...", "password", "..." }
    longTime=True
)

print localUrls
# OUT: ['/tmp/untitled_20160316t160650610780.html',
# '/tmp/Docstring_20160316t160650611136.html']

… image:: https://i.imgur.com/QYoMm24.png width=600px