10 recipes for gradle-download-task

I re­ceived a couple of re­quests lately con­cern­ing com­mon prob­lems when down­load­ing files in Gradle build scripts us­ing gradle-down­load-task. Al­most all of these is­sues could be solved by just us­ing gradle-down­load-task the right way in­stead of ex­tend­ing the plug­in. I have col­lec­ted a num­ber of ex­ample build scripts for these is­sues over the last months and I thought it would be good idea to sum­mar­ize them here.

For those of you who don’t know what gradle-down­load-task is: the plug­in provides a Download task that dis­plays pro­gress in­form­a­tion while down­load­ing files, just like Gradle does when it fetches an ar­ti­fact from a re­pos­it­ory.

Note: the fol­low­ing ex­amples have been tested with Gradle 2.1 or later and gradle-down­load-task 2.1.0 or later.

Recipe #1: download a single file

This is the simplest use case for gradle-down­load-task. The fol­low­ing snip­pet down­loads a single file to the pro­ject’s build dir­ect­ory.

task downloadFile(type: Download) {
    src 'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
    dest buildDir
    overwrite true
}

Al­tern­at­ively, use the download ex­ten­sion to down­load a file any­where in your build script:

task downloadFile << {
    download {
        src 'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
        dest buildDir
        overwrite true
    }
}

Com­plete ex­ample build scripts: sim­pleTask.gradle and sim­pleEx­ten­sion.gradle

Recipe #2: download multiple files

If you want to down­load mul­tiple files at once to the pro­ject’s build dir­ect­ory use the fol­low­ing snip­pet.

task downloadFiles(type: Download) {
    src([
        'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
        'http://central.maven.org/maven2/org/eclipse/jetty/jetty-server/9.1.3.v20140225/jetty-server-9.1.3.v20140225-javadoc.jar'
    ])
    dest buildDir
    overwrite true
}

Com­plete ex­ample build script: mul­tiple­Files.gradle

Recipe #3: download and rename multiple files

If you want to down­load mul­tiple files and spe­cify des­tin­a­tion file names for each of them you have two op­tions.

Op­tion 1 calls the download ex­ten­sion in a loop.

/**
 * Define files to download and destination file names
 */
ext.src = [
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD': 'config.guess',
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD': 'config.sub'
]

/**
 * Call the download extension in a for loop
 */
task downloadMultipleFiles1 << {
    for (s in src) {
        download {
            src s.key
            dest new File(buildDir, s.value)
        }
    }
}

Op­tion 2 cre­ates mul­tiple Gradle tasks to achieve the same thing.

/**
 * Define files to download and destination file names
 */
ext.src = [
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD': 'config.guess',
    'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD': 'config.sub'
]

/**
 * Create multiple tasks
 */
task downloadMultipleFiles2

for (s in src) {
    task "downloadMultipleFiles2_${s.key.hashCode()}"(type: Download) {
        src s.key
        dest new File(buildDir, s.value)
    }
    downloadMultipleFiles2.dependsOn("downloadMultipleFiles2_${s.key.hashCode()}")
}

Com­plete ex­ample build script: mul­tiple­Files­Re­name.gradle

Recipe #4: custom HTTP headers

Down­load a single file and spe­cify a cus­tom HTTP header with the fol­low­ing snip­pet.

task downloadFile(type: Download) {
    src 'http://download.oracle.com/otn-pub/java/jdk/8-b132/jre-8-windows-x64.exe'
    dest buildDir
    header 'Cookie', 'oraclelicense=accept-securebackup-cookie'
}

This ex­ample demon­strates how to down­load the Or­acle JDK 8 in­staller from a build script by auto­mat­ic­ally ac­cept­ing the Or­acle Li­cense.

Com­plete ex­ample build script: cus­tom­Header.gradle

Recipe #5: download all files from a directory

The fol­low­ing snip­pet down­loads all files from a dir­ect­ory on a server. It makes use of Ivy’s URL lister to read the server’s dir­ect­ory list­ing.

task downloadDirectory << {
    def dir = 'http://central.maven.org/maven2/de/undercouch/gradle-download-task/1.0/'
    def urlLister = new org.apache.ivy.util.url.ApacheURLLister()
    def files = urlLister.listFiles(new URL(dir))
    download {
       src files
       dest buildDir
    }
}

From Gradle 2.0 on you need to in­clude a build script de­pend­ency to Apache Ivy in or­der to make this re­cipe work. Put the fol­low­ing right at the be­gin­ning of your build script.

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'org.apache.ivy:ivy:2.3.0'
    }
}

Com­plete ex­ample build script: dir­ect­ory.gradle

Recipe #6: mirror servers

Down­load a single file from a mir­ror server with the fol­low­ing snip­pet. It first con­fig­ures mul­tiple mir­ror serv­ers and then uses the first one that is work­ing.

task downloadFile << {
    def mirrors = [
        'http://central.maven-non-existing.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
        'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar',
        'http://central.maven-non-existing2.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
    ] as Queue
    while(true) {
        def mirror = mirrors.poll()
        try {
            download {
                src mirror
                dest buildDir
                overwrite true
            }
            break
        } catch (Exception e) {
            if (mirrors.isEmpty()) {
                throw e
            }
            logger.warn("Could not download file. Trying next mirror.")
        }
    }
}

Com­plete ex­ample build script: mir­rors.gradle

Recipe #7: Groovy closures for src and dest

The fol­low­ing snip­pet down­loads a single file. It uses Groovy clos­ures to gen­er­ate val­ues for the src and dest prop­er­ties at runtime.

/**
 * Example data and methods
 */
ext {
    downloadToBuildDir = true
}

def getMavenCentralUrl() {
    return "http://central.maven.org/maven2/"
}

def getStylesJar() {
    return "org/citationstyles/styles/1.0/styles-1.0.jar"
}

/**
 * The actual example
 */
task downloadFile(type: Download) {
    src {
        def mavenUrl = getMavenCentralUrl()
        def stylesJar = getStylesJar()
        return mavenUrl + stylesJar
    }
    dest {
        if (downloadToBuildDir) {
            return buildDir
        }
        return File.createTempDir()
    }
    overwrite true
}

Com­plete ex­ample build script: sr­cAnd­Dest­Clos­ure.gradle

Recipe #8: temporary .part file (i.e. download and rename)

This snip­pet down­loads a single file con­di­tion­ally us­ing a tem­por­ary file name (<filename>.part). It re­names the file af­ter­wards if the down­load was suc­cess­ful. The .part file is al­ways over­writ­ten if it ex­ists (e.g. from a pre­vi­ous at­tempt to down­load the file).

task downloadFile << {
    def destFile = new File(buildDir, "styles-1.0.jar")
    if (!destFile.exists()) {
        def tempFile = new File(buildDir, "styles-1.0.jar.part")
        download {
            src 'http://central.maven.org/maven2/org/citationstyles/styles/1.0/styles-1.0.jar'
            dest tempFile
            overwrite true
        }
        tempFile.renameTo(destFile)
    }
}

Com­plete ex­ample build script: tempRe­n­ame.gradle

Recipe #9: unzip downloaded file

The fol­low­ing two tasks down­load a ZIP file and ex­tract its con­tents to the pro­ject’s build dir­ect­ory.

task downloadZipFile(type: Download) {
    src 'https://github.com/michel-kraemer/gradle-download-task/archive/1.0.zip'
    dest new File(buildDir, '1.0.zip')
}

task downloadAndUnzipFile(dependsOn: downloadZipFile, type: Copy) {
    from zipTree(downloadZipFile.dest)
    into buildDir
}

Com­plete ex­ample build script: un­zip.gradle

Recipe #10: verify checksum of downloaded file

The fi­nal snip­pet con­sists of two tasks that first down­load a file and then verify its check­sum.

task downloadFile(type: Download) {
    src 'http://www.example.com/index.html'
    dest buildDir
}

task verifyFile(type: Verify, dependsOn: downloadFile) {
    src new File(buildDir, 'index.html')
    algorithm 'MD5'
    checksum '09b9c392dc1f6e914cea287cb6be34b0'
}

Com­plete ex­ample build script: verify.gradle

More snippets

At the time of writ­ing this post the gradle-down­load-task re­pos­it­ory con­tains el­even ex­ample build scripts. You can find the in the ex­amples dir­ect­ory:

https://github.com/michel-kraemer/gradle-download-task/tree/master/examples

More ex­amples will be ad­ded in the fu­ture, so make sure to check this dir­ect­ory too. If you have other snip­pets worth adding to the list then send me a pull re­quest on Git­Hub or add them here to the com­ments. Thanks!

More information

If you want to learn more about the gradle-down­load-task plug­in have a look at its README file or at my earlier post.


Posted by Michel Krämer
on April, 3rd 2016.