10 recipes for gradle-download-task

I re­ceived a couple of re­quests lately con­cern­ing com­mon prob­lems when down­load­ing files in Gradle build scripts us­ing gradle-down­load-task. Al­most all of these is­sues could be solved by just us­ing gradle-down­load-task the right way in­stead of ex­tend­ing the plug­in. I have col­lec­ted a num­ber of ex­ample build scripts for these is­sues over the last months and I thought it would be good idea to sum­mar­ize them here.

For those of you who don’t know what gradle-down­load-task is: the plug­in provides a Download task that dis­plays pro­gress in­form­a­tion while down­load­ing files, just like Gradle does when it fetches an ar­ti­fact from a re­pos­it­ory.

Note: the fol­low­ing ex­amples have been tested with Gradle 5.0 or later and gradle-down­load-task 5.0.0 or later.

Recipe #1: download a single file

This is the simplest use case for gradle-down­load-task. The fol­low­ing snip­pet down­loads a single file to the pro­ject’s build dir­ect­ory.

task downloadFile(type: Download) {
    src 'https://repo.maven.apache.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
    dest buildDir
    overwrite true
}

Al­tern­at­ively, use the download ex­ten­sion to down­load a file any­where in your build script:

task downloadFile {
    doLast {
        download.run {
            src 'https://repo.maven.apache.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
            dest buildDir
            overwrite true
        }
    }
}

Com­plete ex­ample build scripts: simple/​build.gradle and simple-ex­ten­sion/​build.gradle

Recipe #2: download multiple files

If you want to down­load mul­tiple files at once to the pro­ject’s build dir­ect­ory use the fol­low­ing snip­pet.

task downloadFiles(type: Download) {
    src([
        'https://repo.maven.apache.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
        'https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-server/9.1.3.v20140225/jetty-server-9.1.3.v20140225-javadoc.jar'
    ])
    dest buildDir
    overwrite true
}

Com­plete ex­ample build script: mul­tiple-files/​build.gradle

Recipe #3: download and rename multiple files

Down­load mul­tiple files into a tar­get dir­ect­ory and spe­cify a des­tin­a­tion file name for each of them as fol­lows:

/**
 * Define files to download and destination file names
 */
ext.urls = [
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD': 'config.guess',
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD': 'config.sub'
]

task downloadFiles(type: Download) {
    src urls.keySet()
    dest buildDir
    eachFile { f ->
        f.name = urls[f.sourceURL.toString()]
    }
}

Com­plete ex­ample build script: mul­tiple-files-re­name/​build.gradle

Recipe #4: custom HTTP headers

Down­load a single file and spe­cify a cus­tom HTTP header with the fol­low­ing snip­pet.

task downloadFile(type: Download) {
    src 'https://repo.maven.apache.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
    dest buildDir
    header 'User-Agent', 'gradle-download-task'
}

Com­plete ex­ample build script: cus­tom-header/​build.gradle

Recipe #5: download all files from a directory

The fol­low­ing snip­pet down­loads all files from a dir­ect­ory on a server. It makes use of Ivy’s URL lister to read the server’s dir­ect­ory list­ing.

task downloadDirectory {
    doLast {
        def dir = 'https://repo.maven.apache.org/maven2/de/undercouch/gradle-download-task/4.1.2/'
        def urlLister = new org.apache.ivy.util.url.ApacheURLLister()
        def files = urlLister.listFiles(new URL(dir))
        download.run {
           src files
           dest buildDir
        }
    }
}

You also need to in­clude a build script de­pend­ency to Apache Ivy in or­der to make this re­cipe work. Put the fol­low­ing right at the be­gin­ning of your build script.

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'org.apache.ivy:ivy:2.5.0'
    }
}

Com­plete ex­ample build script: dir­ect­ory/​build.gradle

Recipe #6: mirror servers

Down­load a single file from a mir­ror server with the fol­low­ing snip­pet. It first con­fig­ures mul­tiple mir­ror serv­ers and then uses the first one that is work­ing.

task downloadFile {
    doLast {
        def mirrors = [
            'http://repo.maven-non-existing.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
            'https://repo.maven.apache.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
            'http://repo.maven-non-existing2.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
        ] as Queue
        while(true) {
            def mirror = mirrors.poll()
            try {
                download.run {
                    src mirror
                    dest buildDir
                    overwrite true
                }
                break
            } catch (Exception e) {
                if (mirrors.isEmpty()) {
                    throw e
                }
                logger.warn("Could not download file. Trying next mirror.")
            }
        }
    }
}

Com­plete ex­ample build script: mir­rors/​build.gradle

Recipe #7: Lazily evaluate src and dest

The fol­low­ing snip­pet down­loads a single file. It uses Groovy clos­ures to gen­er­ate val­ues for the src and dest prop­er­ties at runtime.

/**
 * Example data and methods
 */
ext {
    downloadToBuildDir = true
}

def getMavenCentralUrl() {
    return "https://repo.maven.apache.org/maven2/"
}

def getStylesJar() {
    return "org/citationstyles/styles/1.0/styles-1.0.jar"
}

/**
 * The actual example
 */
task downloadFile(type: Download) {
    src {
        def mavenUrl = getMavenCentralUrl()
        def stylesJar = getStylesJar()
        return mavenUrl + stylesJar
    }
    dest {
        if (downloadToBuildDir) {
            return buildDir
        }
        return File.createTempDir()
    }
    overwrite true
}

Com­plete ex­ample build script: lazy-src-and-dest/​build.gradle

Recipe #8: temporary .part file (i.e. download and rename)

This snip­pet down­loads a single file con­di­tion­ally us­ing a tem­por­ary file name (<filename>.part). It re­names the file af­ter­wards if the down­load was suc­cess­ful. The .part file is al­ways over­writ­ten if it ex­ists (e.g. from a pre­vi­ous at­tempt to down­load the file).

task downloadFile(type: Download) {
    src 'https://repo.maven.apache.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
    dest new File(buildDir, "styles-1.0.jar")
    overwrite true
    tempAndMove true
}

Com­plete ex­ample build script: temp-re­name/​build.gradle

Recipe #9: unzip downloaded file

The fol­low­ing two tasks down­load a ZIP file and ex­tract its con­tents to the pro­ject’s build dir­ect­ory.

task downloadZipFile(type: Download) {
    src 'https://github.com/michel-kraemer/gradle-download-task/archive/1.0.zip'
    dest new File(buildDir, '1.0.zip')
}

task downloadAndUnzipFile(dependsOn: downloadZipFile, type: Copy) {
    from zipTree(downloadZipFile.dest)
    into buildDir
}

Com­plete ex­ample build script: un­zip/​build.gradle

Recipe #10: verify checksum of downloaded file

The fi­nal snip­pet con­sists of two tasks that first down­load a file and then verify its check­sum.

task downloadFile(type: Download) {
    src 'http://www.example.com/index.html'
    dest buildDir
}

task verifyFile(type: Verify, dependsOn: downloadFile) {
    src new File(buildDir, 'index.html')
    algorithm 'MD5'
    checksum '84238dfc8092e5d9c0dac8ef93371a07'
}

Com­plete ex­ample build script: verify/​build.gradle

More snippets

At the time of writ­ing this post the gradle-down­load-task re­pos­it­ory con­tains 14 ex­ample build scripts in both Groovy and Kot­lin. You can find the in the ex­amples dir­ect­ory:

https://github.com/michel-kraemer/gradle-download-task/tree/master/examples

More ex­amples will be ad­ded in the fu­ture, so make sure to check this dir­ect­ory too. If you have other snip­pets worth adding to the list then send me a pull re­quest on Git­Hub or add them here to the com­ments. Thanks!

More information

If you want to learn more about the gradle-down­load-task plug­in have a look at its README file or at my earlier post.

If you or your com­pany use any of my pro­jects or like what I’m do­ing, please con­sider spon­sor­ing me so I can con­tinue main­tain­ing and de­vel­op­ing my pro­jects and new ones!

Thank you so much for your sup­port!


Posted by Michel Krämer
on April, 3rd 2016.