Image generation can be used to produce normal maps, a back of the napkin test was able to do it first try in 5 image generation calls, that may get down to 2, with "ai greenscreening". So we start with ai prompts at the top of the pipeline. We get normals and masks, of fictitious/ethereal 3d assets. This is a new resource, but the lack of actual 3d assets will have effect across the whole art pipeline.
Assume the start of the process is a pair of normal and a mask.
ai greenscreening - 2 (mask): you ask an image gen ai for "change the background to $randomhex", twice, then process it into a mask; original algorithm ai greenscreening - 5 (uv+mask): you ask for a "textureless 3d model", get human approval, then itterate on the model to be lit from top, bot, left, right, same random colors ai greenscreening - 2 (uv+mask): You start with a "textureless 3d model" lit from a specific direction, perhaps top left, then you ask for and normal map directly and hope for the best ai greenscreening - 10 (uv+mask+texture): you ask for textureless 3d model and a flat texture version, then 2 set of the 4 directions of lighting, and hope this turns into something worthwhile with some processing
uv compositing - both the production of art assets and the usage of them in games will require taking 2 uv maps and a mask of the 2nd asset, rotating the uv and copying the material data to a working texture where the mask bit is set; additionally flipping, inversing and scaling bulk uv compositing - when rendering 100's assets it will make sense to have a bulk api, while making assumptions the art tools may not want to care about uv patching - ai's are at best controllable like a contractor not a knife, I hate this; using uv compositing and a "clay" render, it will be a small but extant way to do a level of human "Sculpting", 2nd stage after ai greenscreening of the suggested workflow clay render - normal maps with 0 extra details can have a standard view of 3d-ness, this is traditionally done with a light brown color in a matte "material"
Painting - 3rd stage, in a ms paint like interface but with renders, the user is presented swappable views of the asset; we pretend they are painting something like a war hammer figurine with a fixed reference frame.
Bright tab - the asset in full bright, has a sun symbol
dark tab - the asset in full dark, has a moon symbol
material tab - view of the data texture and allows for sub-bit "painting" i.e. assignments, given a "paint events" at some pixel and a mask of "specular", the color picker will change and the assignment must handle that while leaving the uv alone
rendering view - separate from the painting of textures, there will need different modes of viewing the data for different stages of working on an asset, i.e. base painting a new asset is different from adding an extra pixel of glint to an eye, this rendering view should also send "paint events" to the active "tab"; it maybe reasonable to allow for custom ones
edge aware painting - unlike a 3d model the user can not rotate to view a good angle on a "seam" in the asset, it seems like a waste of artist time to not handle the case of a changing color near a seam; tools should be designed with the uv data in mind
ok-hsl - "oklab hue saturation lum" 8 bits per channel hyper-hsl - a variant of ok-hsl where luminance and saturation are allowed to go to -100% to 200% allowing for cartoonish colors and other unreal effects, any such color picker should mark the outside regions as "unrealistic" lighting pallet - a 1x256 hyper-hsl texture + list of names toon settings - 1x256 grey scale texture, used for standard toon shader techniques material setting - 1x256 texture for the material settings, probaly shouldn't be changed by the user ... but if they wanta lerp settings - 3x768 grey scale texture for overloading lerping of the colors, 3 for the 3 channels of hsl, hyper-hsl should be multiplied by 3 before accessing scene lighting - a 1x256 hyper-hsl texture provided by the scene, for time of day lighting effects firefly shader - random point source of hyper-hsl light from the lighting pallet; should be a major shader
"Object assets" such as the art work for a player or a box are made up of 3 textures, possibly encoded in standard pngs zipped together
2 of these textures are for "bright" and "dark", and encoded as ok-hsl
3rd is is the uv data + channel of 1 bit of mask, 7 bits of "material", default encoding is 3 bits of "diffuse" (inverse cross product) 4 bits of "specular" (cross product)
"Background assets" are a pair of a ok-hsl and uv + an l from hsl textures, maybe a default shader, maybe a default light hue, maybe scene lighting reference
an artist hand draws some "baked in" lighting, or some other process
the lighting data - uv, hyper-hsl per-pixel lighting the object layer - object assets "uv composited" together the background - an unchanging asset that has some basic lighting and base texture
main shader code - something somehow generates "the lighting data", some kind of gameplay code generates the object layer, the user must have configured the background at some point
- calculate luminance sources, a) lighting uv + object uv + material specular (use material settings to expand 4 bits into 256) b) lighting uv + object uv + material diffuse c) lighting hyper luminance + material properties (101% is 1% lamination of pure black, ignoring uv) d) background + object uv + material specular e) background + object uv + material diffuse f) background hyper luminance + material properties
- calculate anti-luminance sources(if none return 0,0,0) a) lighting anti-luminance and anti saturation b) background anti-luminance and anti saturation
- calculate top 2 luminance(by l+s) and 1 anti luminance sources, add them together, stored as a hyper-hsl
- using preliminary highlight
lfield + toon settings, lerp between the object layer bright and dark texture - using preliminary highlight
lfield + scene lighting - lerp all these together
I strongly believe in tiny tools, especially when vibe coding
- ai modeler - does one of the kinds of ai greenscreening
- uv patchup - does uv compositing with a clay render, uses 2 different colors of clay
- vivid 2.5d painter - main product see painting section of concepts
- vivid 2.5d backgrounds - creates background assets specifically has a "clay ball" and other reference uv objects that can be placed "in" the scene, needs to handle importing an rgb and outputting an asset
- shader control, generate and handle the settings textures
- reference shader
- demo projects