Advanced Swift Textures

Using Basis Universal Texture Compression with Metal

In this short article, we’ll take a look at a relatively new compressed texture format called Basis. Basis is developed by Binomial, LLC, a company founded by Rich Geldreich (of crunch fame) and Stephanie Hurlburt.

Basis is unique among compression formats in that it emphasizes efficient transcoding between compressed formats. This means that a single .basis file can be transformed, at runtime, into a format that’s optimal for the target platform, without decompressing it in memory, saving space and bandwidth.

Although Basis is a commercial product, targeted predominantly at game developers, Binomial has contributed their Basis Universal reference encoder and transcoder to the Khronos Group in support of the glTF model format, under the Apache License 2.0.

Advanced Basics Intermediate WWDC

What’s New in Metal (2019)

With WWDC 2019 over, it’s a good time to reflect on everything that was announced and start seeing how we can put it into practice. The purpose of this article is to round up the changes to the Metal framework and tools and provide pointers to where you can learn more. This was a banner year, with significant improvements to Metal itself; major advancements in related frameworks like MetalPerformanceShaders (MPS), CoreML, and ARKit; and the exciting introduction of an all-new augmented reality engine, RealityKit.

Since this was such a big year, this post can’t hope to cover all the improvements that are included in macOS 10.15 Catalina and iOS 13. In particular, raytracing with MPS was a hot topic, but you’ll have to look elsewhere for coverage. This session should get you up-to-speed.

Basics Rendering

Vertex Data and Vertex Descriptors

The purpose of this article is to describe and explore a foundational concept in Metal: vertex descriptors. Vertex descriptors are the glue between your application code (written in Objective-C or Swift) and your shader functions (written in the Metal Shading Language). They describe the shape of data consumed by your shaders.

When I was writing the first articles for this site (over four years ago), I didn’t have a firm grasp on the purpose of vertex descriptors, and since their use is optional, this omission persisted for far too long. The sample code for most articles has been updated periodically, and most samples now use vertex descriptors, so it seemed fitting to write about them.

You can download the sample project for this article here.

In order to discuss vertex descriptors, we need to go back to the fundamentals of data and functions.

Interaction Intermediate Swift

Picking and Hit-Testing in Metal

In this article, we’ll see how to interact with our 3D scenes using a technique called picking. Picking helps us answer the question “What did the user just tap on?” More precisely, picking is the process of determining which object or objects in our scene are being indicated by an interaction like a mouse click or a touch on the screen. Picking is frequently used in 3D modeling and CAD programs to select objects for further manipulation, but many applications eventually require the user to interact with the scene, and picking is an essential tool for enabling such interactions.

Picking is mostly synonymous with hit-testing, which is the more commonly-used term when discussing 2D graphical user interfaces. In fact, you may already be acquainted with UIKit’s model for hit-testing. For the purposes of this article, picking and hit-testing are identical.

You can view and download the sample code for this article here.

A demonstration of selecting objects with picking
A demonstration of selecting objects with picking
Intermediate MetalKit ModelIO Rendering Text

Rendering 3D Text with Core Text and libtess2

In this article, we’ll discuss how to create and render 3D text with Metal. Most applications need to render text, and there are many techniques for drawing 2D text with graphics APIs, from pre-rasterized font atlases, to signed-distance field methods like Chris Green’s seminal work at Valve or GLyphy, to cutting-edge vector-based solutions like Slug. Most of these techniques generalize to positioning 2D text in 3D environments as well. Some even allow text to conform to the contours of 3D objects.

The technique discussed here is not intended for general use in graphical user interfaces (GUIs), where you often see 2D text. Rather, this article is about how to create an extruded 3D mesh that represents a string of text. This kind of mesh has many uses, such as labeling objects in virtual/augmented reality or educational applications. Any time you need a piece of text to have some heft to it, you might consider using extruded text.

An example of the 3D text you can create with the technique described in this article.

You can download the sample code for this article here.

Advanced MetalKit ModelIO Physically-Based Rendering

Rendering Physically-Based ModelIO Materials

In this article, we’ll take a look at a portion of the ModelIO framework we haven’t used so far: materials. In previous articles, we’ve used ModelIO to do the heavy lifting of loading 3D geometry, but have loaded and applied textures manually. This ad hoc approach works when we only have a base color map, but quickly grows tedious as materials become more complicated. Here, we’ll take a deep dive into ModelIO materials and learn how to use them to perform basic physically-based rendering (PBR).

You can find the sample code for this article here.

This is not a tutorial on physically-based rendering. PBR has been a hot topic for the past several years, and other people have written excellent tutorials on the subject. Joey de Vries’ series is probably the most approachable. Also consider taking a look at the documentation for Google’s brand-new mobile-first rendering engine Filament. For more theoretical background, consult the following papers and resources:

The focus here is on how ModelIO represents materials, deemphasizing how a shader might render them. For this reason, the shaders for this article are a pared-down version of Apple’s LOD with Function Specialization sample, which is worth studying.

An example asset rendered by SceneKit, from the 2016 WWDC Session, Advances in SceneKit Rendering (
Intermediate MetalKit Rendering Swift

Writing a Modern Metal App from Scratch: Part 2

In the previous article, we wrote enough Metal code to get the spinning silhouette of a teapot on the screen, but that still leaves a lot to be desired as far as a “modern” app is concerned. In this article, we’ll further flesh out the app and introduce lighting, materials, texturing, and managing multiple objects with scene graphs.

Remember that you can clone this Github repository to follow along with the sample code.

Basics MetalKit Swift

Writing a Modern Metal App from Scratch: Part 1

Getting Started

This article is a quick introduction to how to use the Metal, MetalKit, and Model I/O frameworks in Swift. If you know your way around UIKit or Cocoa development, you should be able to follow along for the most part. Some things like shaders and matrices will be foreign to you, but you can learn them as you go about exploring Metal on your own. The purpose here is to give you a template to build on.

If you want to follow along without copy-pasting the code yourself, you can clone this GitHub repository and follow the instructions there.

First things first. Use Xcode to create a new project from the iOS Single View App template. Add import MetalKit at the top of the ViewController.swift file. We could use the Game template instead and have some of the boilerplate written for us, but writing it out long-hand will give us more of an appreciation for the moving parts. The Game template also includes a lot of moving parts that get in the way of understanding the basics.

Book Meta

The Metal by Example Book is Now Available!

Metal by Example is now a book! You can buy it here as a DRM-free PDF.

Since I started this site, I’ve been wanting to turn Metal by Example into a book. Although the blog format is good for quickly publishing techniques one at a time, a book is necessarily a more cohesive experience. For the past month or so, all of my energy has been directed at revising, rewriting, expanding, and editing my Metal content to turn it into a book.

Of course, this project would never have been possible without you, the reader. Thank you for your support over the last year.

If you want to support the work I’ve done here and benefit from the more streamlined, consistent experience of the book, please consider buying it. Thanks again!—Warren

Update (September 13, 2016): Thanks to everyone for the support over the past year! The book is now available on a pay-what-you-want basis. A lot has changed in the world of Metal since I started writing two years ago, and it doesn’t make sense to keep charging full-price for this material, some of which has grown out-of-date. I hope even more people can find it useful, even as it ages, now that price isn’t a factor.

Image Processing Metal Performance Shaders

Metal Performance Shaders in Swift

What is the Metal Performance Shaders Framework?

Announced at WWDC 2015, the Metal Performance Shaders (MPS) framework is a collection of high-performance image filters for iOS 9. Each filter in the library is an efficient black-box implementation of a common image processing task: blur, edge detection, convolution, etc. One useful trait of Apple’s black-box approach with the framework is that Apple can improve the implementation of these filters as new hardware becomes available, and users of MPS can avoid having to code them from scratch.

Download the sample project here.