Shader Hacking

The Linden Lab viewer for SecondLife has 2 main rendering pipelines and 4 sets of shaders.

The two rendering pipelines are called Legacy and Deferred.

The Legacy render pipeline lighting model supports up to 8 light sources: the sun, the moon, and six nearest local lights, and lights/shades triangles by calculating values at their vertices.

The Deferred render pipeline can Shadows and Lights features with “unlimited” numbers of light sources and lights/shades triangles using per-pixel shaders.

The four types of shaders are: Basic Shaders, Windlight Atmosphere Shaders, Windlight Water Shaders, and various lighting/shadows/effects shaders used in the render pipelines.

The Linden shaders contain numerous errors which prevent the shader's from compiling correctly on my system when each of the various shader sets are enabled, preventing either Windlight or Deferred Rendering from working correctly, causing my viewer to show me only 2007-era graphics.

These errors are encountered on

• AMD FX-8150
• AMD Radeon HD 5970 dual GPU card

When I first started running into Shader errors, my software environment was:

• Fedora Linux 18 aka “Spherical Cow”
• AMD Catalyst Proprietary add-on driver and GL 13.4
• Cool VL Viewer 1.26.9.x (based on Linden Lab 3.5 rendering code)

When running in this software environment, the Linden 3.5 rendering code began throwing errors that the variable “i” in the “LIGHT_LOOP(i)” function was undefined.

• ./class2/deferred/alphaNonIndexedF.glsl
• ./class2/deferred/alphaNonIndexedNoColorF.glsl
• ./class1/deferred/materialF.glsl
• ./class1/deferred/alphaNonIndexedF.glsl
• ./class1/deferred/alphaF.glsl

To fix this error, I added “int i = 0;” as the line of code just before the definition of the LIGHT_LOOP(i) function.

Then, Fedora 19 was released and I upgraded.

• Fedora Linux 19 aka “Schrödinger's Cat”
• AMD Catalyst Proprietary add-on driver and GL 13.6-beta
• Cool VL Viewer 1.26.9.1x

When using the 13.6-beta catalyst driver, the viewer would start, then immediately crash after trying to switch to the 3D view.

The viewer log error showed that uki initialization failed, and indirect rendering was being used instead.

However, the Catalyst driver and Xorg logs showed that Direct Rendering and uki* functions were succeeding without errors.

Concluded the viewer's code was incompatible with some change in the ukiOpen function in the Catalyst 13.6-beta driver.

So, I removed the closed-source Catalyst driver and enabled:

• Fedora Linux 19 aka “Schrödinger's Cat”
• Fedora 19 Radeon open-source kernel driver
• Fedora 19 Mesa GL

With this, the viewers run, but with only limited graphics options.

Any attempt to enable shaders throws numerous GLSL compilation errors into the viewer log.

Once the following errors were worked around, the viewers run with almost identical graphical view as the proprietar driver, at a much lower frame rate.

The default Fedora GL extensions “Mesa” apparently do not support the #extension keyword/directive when compiling GLSL.

The following list of files tried to load the GL_ARB_texture_rectangle extension:

• ./class2/deferred/alphaNonIndexedF.glsl
• ./class2/deferred/sunLightF.glsl
• ./class2/deferred/sunLightSSAOF.glsl
• ./class2/deferred/softenLightF.glsl
• ./class2/deferred/spotLightF.glsl
• ./class2/deferred/alphaNonIndexedNoColorF.glsl
• ./class2/deferred/multiSpotLightF.glsl
• ./class1/deferred/fullbrightF.glsl
• ./class1/deferred/emissiveF.glsl
• ./class1/deferred/multiPointLightF.glsl
• ./class1/deferred/fxaaF.glsl
• ./class1/deferred/giF.glsl
• ./class1/deferred/alphaNonIndexedF.glsl
• ./class1/deferred/sunLightF.glsl
• ./class1/deferred/sunLightSSAOF.glsl
• ./class1/deferred/postDeferredGammaCorrect.glsl
• ./class1/deferred/dofCombineF.glsl
• ./class1/deferred/cofF.glsl
• ./class1/deferred/pointLightF.glsl
• ./class1/deferred/softenLightF.glsl
• ./class1/deferred/spotLightF.glsl
• ./class1/deferred/postDeferredF.glsl
• ./class1/deferred/blurLightF.glsl
• ./class1/deferred/waterF.glsl
• ./class1/deferred/alphaNonIndexedNoColorF.glsl
• ./class1/deferred/multiSpotLightF.glsl
• ./class1/deferred/postDeferredNoDoFF.glsl
• ./class1/deferred/alphaF.glsl
• ./class1/deferred/normgenF.glsl
• ./class1/deferred/fullbrightAlphaMaskF.glsl
• ./class1/interface/splattexturerectF.glsl
• ./class1/interface/downsampleDepthRectF.glsl
• ./class1/interface/glowcombineF.glsl
• ./class1/interface/glowcombineFXAAF.glsl
• ./class1/effects/glowExtractF.glsl

In all cases, I simple commented the line by adding /* to the front and */ to the end.

I have no idea if this is impacting how the shaders run.

glxinfo shows the extension is already loaded and available in the X server.

The following two files define 'uniform float delta' while the rest of the shaders define delta a uniform vec2:

• ./class1/interface/downsampleDepthRectF.glsl
• ./class1/interface/downsampleDepthF.glsl

The change I made was to edit both files and change uniform float delta to uniform vec2 delta.

It compiles, but I have no idea if this is correct, or causing other silent code errors.

When attempting to compile the following file, the shader compiler complains that Macros containing __ are for reserved macros only.

• ./class1/deferred/fxaaF.glsl

This looks like the GLSL compiler being paranoid and picky, and nvidia's coding just needs some linting to be cleaner when used with a wider variety of compilers.

This file had to be carefully edited to change FXAA_* macros to replace _ _ with _N_, while leaving otherr x,y,z,w macros untouched.

Search and replace FXAA_QUALITY_ _P with FXAA_QUALITY_N_P and FXAA_CONSOLE_ _P with FXAA_CONSOLE_N_P will get most instances.

Only one difference remains - the WindLight water highlights work when deferred rendering is DISABLED, but disappear if Deferred rendering is enabled.

I can live with that, and the lower framerates for now, especially after knocking the FSAA down to 2x from the normal 48x I keep it at on the proprietary Catalyst driver.

It's just nice to get most everything working again.

UPDATE: I have reported these errors as part of BUG 3397 on the Linden Jira.