Creative Coding — Level 3: Advanced

Abstract: This advanced creative coding course opens the frontier systems: GLSL shaders (fragment shader, vertex shader, p5.js WebGL mode), TouchDesigner node architecture for real-time installations, advanced audio-reactive (multi-band FFT + beat detection), reaction-diffusion simulation (Gray-Scott model), evolutionary algorithms and Boids simulation, performance optimization patterns, and exhibition production workflow. The capstone is an exhibition-ready installation with physical output (print, screen or projection) and a complete technical dossier. Outline: GLSL intro → WebGL in p5.js → TouchDesigner → Audio advanced → Reaction-diffusion → Boids + evolution → Performance → Exhibition.

Learning Objectives

By the end of this course, students will be able to:

1. Write (creating) GLSL fragment and vertex shaders from scratch.
2. Integrate (applying) custom GLSL shaders in p5.js WEBGL mode.
3. Build (creating) a node-based pipeline in TouchDesigner for a real-time installation.
4. Implement (applying) an advanced audio-reactive system with multi-band FFT and beat detection.
5. Simulate (creating) a reaction-diffusion system (Gray-Scott model) as a GLSL shader.
6. Build (creating) a Boids (flocking) simulation with separation, alignment and cohesion.
7. Optimize (applying) a p5.js sketch for high performance (60 fps with 10,000+ particles).
8. Produce (creating) a complete exhibition file: work, documentation, technical specs, promotion visuals.

Module 1 — Introduction to GLSL Shaders (2.5h)

Concept

A shader is a small program executed directly on the GPU (Graphics Processing Unit). The GPU can execute thousands of these programs simultaneously — one per pixel, one per vertex. This is what makes real-time graphics and simulations of extraordinary complexity possible.

Two types of shaders:

• Fragment shader: computes the final color of each pixel
• Vertex shader: computes the position of each vertex of a 3D mesh

Fragment shader — structure: `glsl // GLSL fragment shader — minimal #ifdef GL_ES precision mediump float; #endif

uniform vec2 u_resolution; // canvas dimensions (pixels) uniform float u_time; // time in seconds since start

void main() { // gl_FragCoord: pixel position in pixels // Normalize to [0.0, 1.0] vec2 uv = gl_FragCoord.xy / u_resolution;

// Color varies by position and time vec3 color = vec3(uv.x, uv.y, 0.5 + 0.5 * sin(u_time));

gl_FragColor = vec4(color, 1.0); // RGBA } `

GLSL types:

• float: real number
• vec2, vec3, vec4: 2D, 3D, 4D vectors
• mat2, mat3, mat4: matrices
• uniform: value passed from CPU to GPU (constant per frame)
• varying: value interpolated between vertex and fragment shader

Shadertoy and The Book of Shaders:

• https://www.shadertoy.com — write, share, explore GLSL shaders in the browser
• https://thebookofshaders.com — the best pedagogical reference for GLSL (Patricio Gonzalez Vivo & Jen Lowe)

Lab 1.1 — My First Shader on Shadertoy

Steps:

1. Go to https://www.shadertoy.com, create a free account
2. Replace the default shader with:

glsl void mainImage(out vec4 fragColor, in vec2 fragCoord) { vec2 uv = fragCoord / iResolution.xy; vec3 col = vec3(uv.x, uv.y, 0.5 + 0.5 * sin(iTime)); fragColor = vec4(col, 1.0); }

1. Modify the formula to get: gradient, stripes, circle
2. Add sin(uv.x * 20.0 + iTime) to create waves
3. Explore: what happens with fract(), floor(), smoothstep()?

Module 2 — Vertex Shaders and p5.js WebGL (2.5h)

Concept

Vertex shader — structure: `glsl // GLSL vertex shader for p5.js WEBGL attribute vec3 aPosition; attribute vec2 aTexCoord;

uniform mat4 uModelViewMatrix; uniform mat4 uProjectionMatrix; uniform float u_time;

varying vec2 vTexCoord;

void main() { vTexCoord = aTexCoord; vec4 pos = vec4(aPosition, 1.0); // Displacement: move vertices along y by a sin of position + time pos.y += sin(pos.x 5.0 + u_time) 0.1; gl_Position = uProjectionMatrix uModelViewMatrix pos; } `

Integrating a custom shader in p5.js WEBGL: `javascript let myShader;

function preload() { // Load vertex.glsl and fragment.glsl files myShader = loadShader('vertex.glsl', 'fragment.glsl'); }

function setup() { createCanvas(600, 600, WEBGL); // WEBGL mode }

function draw() { shader(myShader); // Pass uniforms to the shader myShader.setUniform('u_resolution', [width, height]); myShader.setUniform('u_time', millis() / 1000.0); myShader.setUniform('u_mouse', [mouseX / width, mouseY / height]); // Draw a rectangle covering the full canvas (geometry for the shader to run on) rect(-width/2, -height/2, width, height); } `

Bypassing p5.js default shader (rect mode): javascript noStroke(); noFill(); plane(width, height); // centered plane in WEBGL

Lab 2.1 — Parametric Mesh with Vertex Shader

Steps:

1. Create a createCanvas(600, 600, WEBGL) canvas
2. Write a vertex shader that displaces vertices sinusoidally as a function of time
3. Pass u_time and u_mouse to the shader
4. Draw a high-resolution plane (many vertices) to make the deformation visible
5. Color each vertex as a function of displacement in the fragment shader

Module 3 — TouchDesigner: Node Architecture (2.5h)

Concept

TouchDesigner (Derivative, Toronto, 2000) is a node-based visual programming environment designed for real-time installations, video mapping, performances and interactive experiences.

The 5 types of operators:

A minimal audio-reactive pipeline: Microphone In (CHOP) → Analyze CHOP (extracting RMS, peak) → Math CHOP (normalization, scaling) → Circle TOP (using CHOP value as radius parameter) → Out TOP (output to screen)

Python in DAT: TouchDesigner executes Python 3 natively in DAT operators. Each execute DAT can run code triggered by events (frame, parameter change, touch): `python

import td

rms = op('audioin1')['rms'][0] # first sample of 'rms' channel

op('circle1').par.radius = rms * 400 `

OSC and MIDI: TouchDesigner communicates natively with OSC (Open Sound Control) and MIDI — it can send and receive messages to/from other hardware and software. A smartphone can control an installation via OSC.

Lab 3.1 — Minimal Audio-Reactive Installation

Steps:

1. Create a new TouchDesigner project
2. Add a Movie File In TOP (or Noise TOP for testing)
3. Add a Level CHOP analyzing the audio
4. Connect the RMS channel to a Circle TOP radius parameter via an Expression in the parameter
5. Add a Composite TOP to merge the original image with the circle
6. Test with a music file — the circle should pulsate in rhythm with the music

Module 4 — Advanced Audio-Reactive: Multi-Band FFT + Beat Detection (2.5h)

Concept

Beyond the simple spectrum, advanced audio-reactive uses precise frequency bands and beat detection.

Multi-band FFT in p5.js: `javascript let fft, mic;

function setup() { mic = new p5.AudioIn(); mic.start(); fft = new p5.FFT(0.8, 1024); fft.setInput(mic); }

function draw() { let spectrum = fft.analyze();

// Frequency bands let subBass = fft.getEnergy(20, 60); // sub-bass: 20–60 Hz let bass = fft.getEnergy(60, 250); // bass: 60–250 Hz let lowMid = fft.getEnergy(250, 500); // low mid: 250–500 Hz let mid = fft.getEnergy(500, 2000); // mid: 500–2000 Hz let highMid = fft.getEnergy(2000, 4000); // high mid: 2000–4000 Hz let treble = fft.getEnergy(4000, 14000); // treble: 4000–14000 Hz

// Map to visual parameters let circleSize = map(bass, 0, 255, 50, 400); let hue = map(mid, 0, 255, 0, 360); let speed = map(treble, 0, 255, 0.01, 0.1); } `

Beat detection (threshold method): `javascript class BeatDetector { constructor(threshold, decayRate) { this.threshold = threshold; this.decayRate = decayRate; this.energy = 0; this.beat = false; this.lastBeat = 0; } update(value) { this.beat = false; if (value > this.threshold && millis() - this.lastBeat > 200) { this.beat = true; this.lastBeat = millis(); this.energy = 1.0; } this.energy *= this.decayRate; } }

let beatDetector = new BeatDetector(180, 0.85);

function draw() { let bass = fft.getEnergy('bass'); beatDetector.update(bass); if (beatDetector.beat) { // Trigger on beat: flash, particle burst... background(255); } // beatDetector.energy: exponentially decaying value after beat let size = map(beatDetector.energy, 0, 1, 50, 400); } `

Lab 4.1 — Multi-Band Composition

Steps:

1. Create a radial composition: each frequency band controls a concentric ring
2. Implement BeatDetector for the bass band
3. On each beat: emit 20 particles in a burst
4. Use highMid to control rotation speed of a spiral
5. Test with 3 musical genres (electronic, jazz, classical) — how does the visualization adapt?

Module 5 — Reaction-Diffusion (Gray-Scott) (2.5h)

Concept

Reaction-diffusion is a class of systems where two chemical species A and B diffuse and react. The Gray-Scott model (1983–1984) produces spectacular organic patterns (spots, spirals, labyrinths, corals) from only 2 parameters (feed rate f, kill rate k).

The equations: dA/dt = Da·∇²A − A·B² + f·(1−A) dB/dt = Db·∇²B + A·B² − (k+f)·B

Where ∇² is the Laplacian (sum of neighbors). Da, Db: diffusion rates of A and B.

Implementation as GLSL shader (GPU — much faster than CPU): `glsl // GLSL fragment shader — Gray-Scott reaction-diffusion step #ifdef GL_ES precision mediump float; #endif

uniform sampler2D u_state; // current state (A in R channel, B in G channel) uniform vec2 u_resolution; uniform float u_feed; // feed rate (typically 0.01–0.08) uniform float u_kill; // kill rate (typically 0.04–0.07) uniform float u_Da; // diffusion A (typically 1.0) uniform float u_Db; // diffusion B (typically 0.5)

void main() { vec2 uv = gl_FragCoord.xy / u_resolution; vec2 px = 1.0 / u_resolution;

// Current state vec4 current = texture2D(u_state, uv); float A = current.r; float B = current.g;

// Laplacian (5-point stencil) float lA = -A; float lB = -B; lA += texture2D(u_state, uv + vec2(px.x, 0.0)).r 0.2; lA += texture2D(u_state, uv - vec2(px.x, 0.0)).r 0.2; lA += texture2D(u_state, uv + vec2(0.0, px.y)).r 0.2; lA += texture2D(u_state, uv - vec2(0.0, px.y)).r 0.2; lA += texture2D(u_state, uv + vec2(px.x, px.y)).r 0.05; lA += texture2D(u_state, uv - vec2(px.x, px.y)).r 0.05; lA += texture2D(u_state, uv + vec2(-px.x, px.y)).r 0.05; lA += texture2D(u_state, uv - vec2(-px.x, px.y)).r 0.05; lB = lA * (B / A); // simplified — proper implementation mirrors the above

// Gray-Scott reactions float reaction = A B B; float dA = u_Da lA - reaction + u_feed (1.0 - A); float dB = u_Db lB + reaction - (u_kill + u_feed) B;

float newA = clamp(A + dA, 0.0, 1.0); float newB = clamp(B + dB, 0.0, 1.0);

gl_FragColor = vec4(newA, newB, 0.0, 1.0); } `

Reference parameter maps: https://mrob.com/pub/comp/xmorphia/index.html

Turing, A.M. (1952). The Chemical Basis of Morphogenesis. Philosophical Transactions of the Royal Society B, 237(641), 37–72. https://doi.org/10.1098/rstb.1952.0012

Lab 5.1 — Interactive Reaction-Diffusion

Steps:

1. Implement the Gray-Scott system with ping-pong buffers (createGraphics × 2)
2. Initialize: state A = 1.0, B = 0.0 everywhere; add a B = 1.0 seed in the center
3. Interactive: click to add a seed of B at the mouse position
4. Expose f and k as keyboard-adjustable parameters (+/- keys)
5. Explore parameter space: spots (f=0.037, k=0.060), spirals (f=0.018, k=0.051), labyrinths (f=0.030, k=0.062)

Module 6 — Evolutionary Algorithms & Boids (2.5h)

Concept

Boids (Reynolds 1987): Craig Reynolds's Boids algorithm simulates flocking behavior from 3 simple rules:

1. Separation: avoid being too close to neighbors
2. Alignment: align direction with the average direction of neighbors
3. Cohesion: move toward the average center of the group

`javascript class Boid { constructor() { this.pos = createVector(random(width), random(height)); this.vel = p5.Vector.random2D().mult(random(2, 4)); this.acc = createVector(0, 0); this.maxSpeed = 4; this.maxForce = 0.1; this.perceptionRadius = 50; }

flock(boids) { let sep = this.separate(boids).mult(1.5); // separation weight let ali = this.align(boids).mult(1.0); // alignment weight let coh = this.cohesion(boids).mult(1.0); // cohesion weight this.acc.add(sep).add(ali).add(coh); }

separate(boids) { let steer = createVector(0, 0); let count = 0; for (let other of boids) { let d = dist(this.pos.x, this.pos.y, other.pos.x, other.pos.y); if (other !== this && d < this.perceptionRadius * 0.5) { let diff = p5.Vector.sub(this.pos, other.pos); diff.normalize(); diff.div(d); // closer = stronger steer.add(diff); count++; } } if (count > 0) steer.div(count); if (steer.mag() > 0) { steer.normalize().mult(this.maxSpeed).sub(this.vel).limit(this.maxForce); } return steer; }

align(boids) { let sum = createVector(0, 0); let count = 0; for (let other of boids) { let d = dist(this.pos.x, this.pos.y, other.pos.x, other.pos.y); if (other !== this && d < this.perceptionRadius) { sum.add(other.vel); count++; } } if (count > 0) { sum.div(count).normalize().mult(this.maxSpeed).sub(this.vel).limit(this.maxForce); return sum; } return createVector(0, 0); }

cohesion(boids) { let sum = createVector(0, 0); let count = 0; for (let other of boids) { let d = dist(this.pos.x, this.pos.y, other.pos.x, other.pos.y); if (other !== this && d < this.perceptionRadius) { sum.add(other.pos); count++; } } if (count > 0) { let target = sum.div(count); return this.seek(target); } return createVector(0, 0); }

seek(target) { let desired = p5.Vector.sub(target, this.pos).normalize().mult(this.maxSpeed); return p5.Vector.sub(desired, this.vel).limit(this.maxForce); }

update() { this.vel.add(this.acc).limit(this.maxSpeed); this.pos.add(this.vel); this.acc.mult(0); // wrap around canvas if (this.pos.x > width) this.pos.x = 0; if (this.pos.x < 0) this.pos.x = width; if (this.pos.y > height) this.pos.y = 0; if (this.pos.y < 0) this.pos.y = height; }

display() { let angle = this.vel.heading() + HALF_PI; fill(200, 80, 90, 80); noStroke(); push(); translate(this.pos.x, this.pos.y); rotate(angle); triangle(0, -10, -5, 5, 5, 5); pop(); } } `

Reynolds, C.W. (1987). Flocks, Herds, and Schools: A Distributed Behavioral Model. SIGGRAPH '87, pp. 25–34. https://doi.org/10.1145/37401.37406

Lab 6.1 — Boids + Obstacle Avoidance

Steps:

1. Implement 150 Boids
2. Add 3 circular obstacles that Boids must avoid
3. Make Boids react to the mouse as a repellent (strong separation)
4. Vary the 3 weights (separation, alignment, cohesion) with keyboard
5. Observe emergent behavior: what does a flock do that individual birds don't?

Module 7 — Performance Optimization (2.5h)

Concept

A p5.js sketch at 60 fps is sufficient for simple compositions. Beyond ~5,000 objects drawn per frame, optimization becomes necessary.

Profiling: Use the browser developer console → Performance tab. Record 5 seconds of animation. Look for the bottlenecks: Is it JavaScript? Drawing? Memory?

Off-screen buffer (createGraphics): javascript // Static background computed once let bg; function setup() { bg = createGraphics(width, height); // Draw complex static background on bg for (let i = 0; i < 10000; i++) { bg.point(random(width), random(height)); } } function draw() { image(bg, 0, 0); // display buffer without recomputing // Only dynamic objects are drawn on the main canvas }

Avoid draw calls — use images: javascript // Instead of drawing each circle separately, use an image let circleSprite; function setup() { circleSprite = createGraphics(20, 20); circleSprite.noStroke(); circleSprite.fill(255, 80); circleSprite.ellipse(10, 10, 18, 18); } function draw() { for (let p of particles) { image(circleSprite, p.x - 10, p.y - 10); // one image() call per particle } }

Spatial partitioning (grid): For Boids/particles with neighbor queries, a spatial grid reduces the complexity from O(n²) to O(n): javascript class SpatialGrid { constructor(cellSize) { this.cellSize = cellSize; this.grid = {}; } clear() { this.grid = {}; } insert(boid) { let key = ${floor(boid.pos.x/this.cellSize)},${floor(boid.pos.y/this.cellSize)}; if (!this.grid[key]) this.grid[key] = []; this.grid[key].push(boid); } neighbors(pos, radius) { let result = []; let r = ceil(radius / this.cellSize); let cx = floor(pos.x / this.cellSize); let cy = floor(pos.y / this.cellSize); for (let dx = -r; dx <= r; dx++) { for (let dy = -r; dy <= r; dy++) { let key = ${cx+dx},${cy+dy}; if (this.grid[key]) result.push(...this.grid[key]); } } return result; } }

GLSL for compute-heavy effects: Particles, reaction-diffusion, complex shaders → always faster as GLSL on GPU than JavaScript on CPU.

Lab 7.1 — 10,000 Particles at 60 fps

Steps:

1. Create a system of 1,000 particles — measure fps with frameRate()
2. Increase to 5,000 — what happens?
3. Implement off-screen buffer + sprite: does fps improve?
4. Implement spatial grid for Boids-type neighbors: compare O(n²) vs. O(n)
5. Target: 10,000 particles at 60 fps (achievable with sprite + grid)

Module 8 — Exhibition Production (2.5h)

Concept

For a wall installation (large format screen/projection):

• Hardware: define the playback machine (laptop/mini-PC), screen/projector resolution
• Reliability: the work must run without intervention for 8h
• Watchdog: automatic restart if the app crashes (PM2 in Node.js, or systemd on Linux)

Technical dossier (for curator, gallery, institution):

1. Work description (max 200 words)
2. Technical requirements (hardware, resolution, connectivity)
3. Installation plan (network diagram, dimensions, power)
4. Demo video (30s–2 min)
5. High-resolution static images (3 frames minimum, 300 dpi)

Performance optimization for 8h of continuous playback: `javascript // Avoid memory leaks function draw() { // Delete completed particles from the array for (let i = particles.length - 1; i >= 0; i--) { if (particles[i].isDead()) particles.splice(i, 1); }

// Limit maximum size while (particles.length > 5000) particles.shift(); }

// Responsive Canvas function windowResized() { resizeCanvas(windowWidth, windowHeight); } `

Generative print (physical output): For a print work:

• Minimum 300 dpi: a 100×100 cm print requires 11,811 × 11,811 px at 300 dpi
• Use pixelDensity(3) or createGraphics(11811, 11811) and draw on this buffer
• Save as PNG → open in Photoshop/Illustrator → export as TIFF or print-ready PDF
• SVG export (with p5.svg): vectors scalable to any size without quality loss

Lab 8.1 — Complete Exhibition File

Steps:

1. Choose your best creation from Modules 1–7
2. Adapt for continuous playback: memory management + windowResized()
3. Produce a 3840×2160 (4K UHD) export
4. Write the complete technical dossier (see §Structure above)
5. Create a 30-second demo video (screen recording + sound if audio-reactive)

Capstone Project — "Exhibition-Ready Installation"

Brief

Produce a complete work ready for exhibition in an institutional or commercial context. The work must demonstrate mastery of at least 3 techniques from Level 3 and include all deliverables for professional presentation.

Technical Constraints

• Real-time: the work must run at 60 fps on a laptop
• Duration: minimum 5 minutes of autonomous variation (without human intervention)
• Interaction: optional but documented
• Resolution: minimum 1920×1080, ideally 4K
• Format: web (p5.js), native (Processing, openFrameworks) or TouchDesigner

Deliverables

1. Functional source code (well-commented, ready to run)
2. Demo video (30s–2 min, including sound if applicable)
3. 5 high-resolution frames (PNG minimum 3000px per side)
4. Complete technical dossier (5 pages): concept + system description + technical specs + installation plan + bibliography
5. Public presentation (15 min + 5 min Q&A)

Evaluation Rubric

Readings & Resources

• The Book of Shaders: https://thebookofshaders.com
• Shadertoy: https://shadertoy.com
• The Nature of Code (Shiffman): https://natureofcode.com
• TouchDesigner documentation: https://derivative.ca/UserGuide/
• Gray-Scott explorer: https://mrob.com/pub/comp/xmorphia/index.html
• Boids (Reynolds 1987): https://doi.org/10.1145/37401.37406
• Gray-Scott / Turing (1952): https://doi.org/10.1098/rstb.1952.0012

→ Exhibition portfolio: Book a 30-min call — free → Hub: Creative Coding — All levels