Download large files with Retrofit and show progress

After we hand over a finished project to our clients, we SYNETECH developers often look back at the project and contemplate whether we could implement some parts better, but didn’t, either due to budget reasons, deadlines, or simply because the optimal solution didn’t cross our mind in that moment. Most of the time, we find only minor issues, but then and there we find pieces of code or an application behavior that - to a considerable extent - do not meet the expected standards of android users. An example of an application behavior, which we could have implemented much better, is downloading large files.

Part of an application that we were developing for one of our client included downloading of zip files, which contain all source- and multimedia files of some web page. The downloaded files are then stored in the local storage on the android device. This allowed the users to open and view the downloaded zipped web page even when the device had no access to the internet. Downloading of a zip file can be achieved with a simple HTTP GET request. As per our commonly used tech stack, we used the Retrofit library to provide this HTTP request.

import okhttp3.ResponseBody
import retrofit2.http.GET

interface FileDownloadApi {
    @GET("someFile.zip")
    suspend fun downloadZipFile(): ResponseBody
}

Occurred problems

Unfortunately, we immediately ran into a problem: as per our best practices, we usually set 30 seconds timeout for network requests. However, some of the zip files reached the size of tens (some even hundreds) of MB, and so the 30s timeout was not enough in those cases. As a workaround, we set a much longer timeout (several minutes) for this particular request.

What we didn’t manage to detect and handle during the implementation of the application, is the following issue: the way we defined our HTTP GET request to download a file, Retrofit would try to move the entire file into memory. This can lead to the fatal OutOfMemoryError if the file’s size exceeds your app’s runtime memory.

Last but not least, there was a visible issue regarding user experience. During the file download, we display an infinite circular loader on empty background. Taken in consideration that the process could take minutes for larger files, it may discourage some many users. Unfortunately to add, with the implementation at the time, it was not possible to tell how much data the HTTP GET request already downloaded and how much more is left to be downloaded, so we were unable to inform the users about the download progress.

Looking back at the project, we found rooms for improvements, as well as the opportunity to broaden the knowledge that we lacked in this area. After some research, we discovered the Retrofit annotation @Streaming, which should help us avoid the aforementioned issues.

Prerequisites

You are familiar with Kotlin, Coroutines and Retrofit.

Implementation

If you check out Retrofit @Streaming’s documentation, it reads:

Treat the response body on methods returning ResponseBody as is, i.e. without converting the body to byte[].

which essentially means that for requests returning ResponseBody, if the annotation is applied, we should be able to access the response in its raw form and further handle it in a way we desire. For completion, here’s the documentation of ResponseBody.bytes(), which is used to convert the response body to byte[] and called by default if the @Streaming annotation is not applied. From the documentation, you can see that this is the origin of the OutOfMemoryError.

/**
* Returns the response as a byte array.
* This method loads entire response body into memory. If the response body is very large this may trigger an OutOfMemoryError. Prefer to stream the response body if this is a possibility for your response.
*/@Throws(IOException::class)
fun bytes() = ...

So, for our simple use case, we define the request to download a file as follow:

@Streaming
@GET("large_file_endpoint")
suspend fun downloadLargeFile(): ResponseBody

Next, we need to define our custom way of handling the ResponseBody. We will utilize the ResponseBody.byteStream() that allows us to stream the data instead of waiting for the whole response (like it was by default in the case where @Streaming was not applied), and at the same time we write the incoming data to our desired destination.

private fun ResponseBody.saveFile() {
    val destinationFile = File(destination_path)
    byteStream().use { inputStream->
        destinationFile.outputStream().use { outputStream->
            inputStream.copyTo(outputStream)
        }
    }
}

Then, to download the file, we just call the request inside a Coroutine scope:

scope.launch(Dispatchers.IO){
    apiService.downloadLargeFile().saveFile()
}

And that’s it. With the implementation above, you should no more longer into OutOfMemoryError when downloading a large file using Retrofit. Furthermore, as we no longer wait for the whole response, but instead we now have data stream flowing from the server, the issue with timeout is no longer a concern. With each received data chunk, the timeout is reset. So as long as we receive some data within the timeout interval, timeout should not happen.

One question still remains though: how to show users the download progress?  InputStream.copyTo()internally writes the data chunk by chunk to the destination file, but it doesn’t really tell us it’s current progress. So we will have to come up with our own file saving implementation that would give us the current download progress. Well, not entirely; we could definitely reuse the implementation of InputStream.copyTo() and just add little logic that will report the progress.

Let’s start with a simple class that represents the file download state. The naming of the sub-classes should be self-explanatory:

private sealed class DownloadState {
    data class Downloading(val progress: Int) : DownloadState()
    object Finished : DownloadState()
    data class Failed(val error: Throwable? = null) : DownloadState()
}

Then, we will use Kotlin’s Flow to emit those states correspondingly during the file download process.

private fun ResponseBody.saveFile(): Flow<DownloadState> {
    returnflow{
        emit(DownloadState.Downloading(0))
        val destinationFile = File(destination_path)

        try {
            byteStream().use { inputStream->
                destinationFile.outputStream().use { outputStream->
                    val totalBytes = contentLength()
                    val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
                    var progressBytes = 0L
                    var bytes = inputStream.read(buffer)
                    while (bytes >= 0) {
                        outputStream.write(buffer, 0, bytes)
                        progressBytes += bytes
                        bytes = inputStream.read(buffer)
                        emit(DownloadState.Downloading(((progressBytes * 100) / totalBytes).toInt()))
                    }
                }
            }
            emit(DownloadState.Finished)
        } catch (e: Exception) {
            emit(DownloadState.Failed(e))
        }
    }
    .flowOn(Dispatchers.IO).distinctUntilChanged()
}

Notice that the code of the try/catch block is almost the same as the implementation of InputStream.copyTo(), we just added a simple logic that reports the download progress via our custom defined states. For that purpose, we took the advantage of knowing the final size of the file (ResponseBody.contentLength()) and the cumulative size of the data that has been downloaded (progressBytes). The only thing left to do is to interpret those states to correct UI states:

apiService.downloadLargeFile().saveFile()
    .collect{downloadState->
        state = when (downloadState) {
            is DownloadState.Downloading -> {
                Log.d("myTag", "progress=${downloadState.progress}")
                showProgress(downloadState.progress)
            }
            DownloadState.Failed -> {
                showError()
            }
            DownloadState.Finished -> {
                showSuccess()
            }
        }
    }

And that’s all to it! Now before we end this article, there’s one more issue that I think is worth mentioning. For debugging purposes, you may have HttpLoggingInterceptor’s level set to Level.BODY for your Retrofit instance.

HttpLoggingInterceptor().setLevel(HttpLoggingInterceptor.Level.BODY)

When logging is turned on beyond HEADERS level, the response body has to be read in its entirety into a byte[] in order to both log and hand the data to the converter. This means the @Streaming annotation has no effect in this case, so be aware of this setup when you are debugging your code.

And lastly, I recommend to execute the file download logic on a foreground service, which is meant to host heavy operations. So consider to move your file download logic to a foreground service in order to not block users from using other apps or other parts of your app.

Thank you for reading. You can find all implementation details mentioned in this article in the demo app here.