Building a Media Browser with Go and Google Cloud

For a long time, I've been a fan of running a home media setup with Kodi as a front-end for different media types. I recently had to decommission the ten-year-old PC that was functioning as a media server, and I wanted to explore hosting this media in the cloud. This led me to create Media Browser - a web-based media browser written in Go.

Background

I've been trying to learn about both Go and Google Cloud lately. This project seemed like a great chance to get more experience with both. As such, the first release of Media Browser is effectively a directory browser for Google Cloud Storage.

If you've ever encountered a file server on the web, you know what this looks like.

It's not fancy, but it does the job of providing a navigable view into a file system. I picked this particular representation of a set of media files because one of the media library types that Kodi supports is an HTTP web server.

Targeting a Kodi-compatible Web Server

While there's not a strict specification for how directory browsing should work, the common experience of popular web server's built-in directory browsing capabilities seemed like enough to go on. After my first pass at generating a functional directory browser, I plugged the URL and credentials into Kodi and tested out browsing a set of pictures. The result: nothing showed up.

To start troubleshooting, I spent a bit of time using a little "View source" to help me accurately replicate the Apache HTTPD directory pages. I wondered if the parsing logic in Kodi was specific enough to expect particular elements and structures from Apache's version. I set up a simple Apache server for comparison and found that enabling directory browsing was enough for Kodi to be functional.

After I got to a near-exact replica of the Apache pages and Kodi still wasn't recognizing directories and files, I turned to the logs. I found that Kodi made an OPTIONS HTTP request to directories and files before retrieving them.

After I replicated this behavior in Media Browser and ensured that file names and links matched Apache's behavior, I took another spin with the library in Kodi: success!

Working with Google Cloud

Architecture

One of my objectives for Media Browser is cost management. Cloud hosting can get expensive depending on your technical approach. For this use case, I'm primarily interested in storing a good bit of media and streaming it as inexpensively as possible. Cloud Storage is an obvious choice for basic blob storage. It's practically infinitely scalable and the cost scales simply with the amount of data you store and retrieve. Another nice feature of Cloud Storage is Signed URLs: these provide time-limited "public" URLs that can be used to directly view and stream media hosted in the platform.

Note: AWS and Azure both have very similar offerings that could be plugged in.

The other major factor with considering cost is the compute platform. One of Google Cloud's less expensive options for hosting applications is Cloud Run. Cloud Run is a serverless platform and can scale to zero. The consumption-based pricing model means that we can really limit costs here when the app isn't running.

By combining the Cloud Storage Signed URL feature with Cloud Run's scale to zero capability, we can limit the overall cost of running Media Browser to be roughly the same as the storage cost and network traffic cost. As a user, when you browse the app, you are presented with pages of hyperlinks allowing you to navigate up and down through a directory structure in a Cloud Storage bucket. When you select an individual file, you are redirected to a Signed URL generated on the fly. At that point, the Cloud Run app shuts down because there is no more activity until the next time you start browsing.

Bringing it all together with the CI/CD pipeline in Cloud Build, the architecture looks like this:

Challenges

Working with Cloud Storage, Cloud Run, and Cloud Build were all pretty straightforward. The main challenge I had working with the platform was getting the Signed URL to work in Cloud Run.

When you are developing a Google Cloud app locally and using Google Cloud services, you typically create and use a service account and private key file. This file contains credentials necessary to authenticate to Google Cloud service APIs as well as a private key used for signing.

When an app is deployed to Cloud Run (as well as other compute environments in Google Cloud), best practice suggests using the default runtime service account or assigning a managed service account to the app. The struggle I had with this is that the private key for the managed service accoutns did not seem to be exposed to the app. This meant that the signing functionality provided by the SDK did not work.

To get around this, I ended up storing the private key manually in Secret Manager and extracting it from there during initialization. If I understand the API docs, it seems to be possible to create a signed URL by signing the request with the IAM signBlob method, but I had trouble getting this to work. It does seem to be a better approach to managing credentials than manually handling the service account's private key, so I plan to revisit this in the future.

Retrospective

This was also only my second experience writing a program in Go. I have plenty still to learn, but it was fun to experiment with a less familiar language. I really like that functions are a first-class citizen, formatting is heavily standardized, and execution is super fast. Multiple return values are also a cool feature, but the convention of using them to communicate errors feels uncomfortable to me. I imagine that veteran Go developers like this construct, but I find myself wanting to use an exception-oriented approach. Because most of the languages I've worked in lately (Java, Kotlin, Javascript) do not expose these details, I also am not familiar with how to appropriately use pointers. Maybe I'm just too far away from the C/C++ I learned in high school 😅.

All in all, building Media Browser was a great project to work on. I look forward to doing more with Google Cloud and Go in the future.

If you have questions or suggestions, please reach out!