NIS SDK Programming Guide
NIS SDK Programming Guide
2)
Programming Guide
Revision History
1.0.2 Performance optimizations 2 February 2022
Adjusted minimum sharpness value
Fixed warnings
1.0.1 Performance optimizations 13 December 2021
fp16 coefficients support
GLSL support
DX12 and Vulkan samples
1.0.0 Initial release 16 November 2021
The directional scaling and sharpening algorithm are combined together in NVScaler while NVSharpen
only implements the adaptive-directional-sharpening algorithm. Both algorithms are provided as
compute shaders and developers are free to integrate them in their applications. Note that if you
integrate NVScaler, you should NOT also integrate NVSharpen, as NVScaler already includes a
sharpening pass.
2 Getting Started
2.1 System Requirements
The following is needed to load and run NVScaler and NVSharpen:
The sample app included with the SDK can be compiled for Windows or Linux.
- Support High-level Shader Language (HLSL) model 5.0 or higher, or OpenGL Shader Language
(GLSL) 4.50.7 (version 450)
- The HLSL shader integration requires DirectX11, DirectX12, or Vulkan support
- Support a high-quality anti-aliasing technique like TAA
- Have the ability to negatively bias the LOD for textures and geometry
- On each shader call (i.e., each frame), provide:
o The raw color texture for the frame (in HDR or SDR in display-referred color-space)
o The output texture with the right dimensions
o For NVScaler: scale and sharpness values along with configuration and coefficients
o For NVSharpen: sharpness value and configuration values
To allow for future compatibility and ease ongoing research by NVIDIA, the application should consider
integrating NVIDIA Image Scaling SDK compute shaders without modifications.
Since sharpening algorithms can enhance noisy or grainy regions, it is recommended that certain effects
such as film grain should occur after NVScaler or NVSharpen. Low-pass filters such as motion blur or
light bloom are recommended to be applied before NVScaler or NVSharpen to avoid sharpening
attenuation.
NVIDIA
POST- IMAGE SCALING POST-
ANTI-ALIASED TONE
PROCESSING PROCESSING HUD
RENDER MAPPING
BEFORE SPATIAL SCALING + AFTER
SHARPENING
ALGORITHM
1) LDR
2) HDR PQ
3) HDR Linear
- The recommended range of color values is [0, 12.5], where luminance value (as per BT. 709) of
1.0 maps to brightness value of 80nits (sRGB peak) and 12.5 maps to 1000nits
- The input color texture may have luminance values that are either linear and scene-referred or
linear and display-referred (after tone mapping)
If the input color texture sent to NIS is in HDR format set the NIS_HDR_MODE define to either
NIS_HDR_MODE_LINEAR (1) or NIS_HDR_MODE_PQ (2).
Defines
To get optimal performance of NVScaler and NVSharpen for current and future hardware, it is
recommended that the following API is used to obtain the values for NIS_BLOCK_WIDTH,
NIS_BLOCK_HEIGHT, and NIS_THREAD_GROUP_SIZE. These values can be used to compile
permutations of NVScaler and NVSharpen offline.
enum class NISGPUArchitecture : uint32_t
{
NVIDIA_Generic = 0,
AMD_Generic = 1,
Intel_Generic = 2,
NVIDIA_Generic_fp16 = 3,
};
struct NISOptimizer
{
bool isUpscaling;
NISGPUArchitecture gpuArch;
NIS_SCALER should be set to 1, and the isUpscaling argument should be set to true.
Defines defines;
defines.add("NIS_SCALER", isUpscaling);
defines.add("NIS_HDR_MODE", hdrMode);
defines.add("NIS_BLOCK_WIDTH", blockWidth);
defines.add("NIS_BLOCK_HEIGHT", blockHeight);
defines.add("NIS_THREAD_GROUP_SIZE", threadGroupSize);
NVScalerCS = CompileComputeShader(device, "NIS_Main.hlsl”, &defines);
Note: Compilation of the shader permutations can be performed off-line.
NISConfig config;
createConstBuffer(&config, &csBuffer);
Create SRV textures for the scaler and USM phase coefficients
const int rowPitch = kFilterSize * sizeof(float); // use fp32: float, fp16: uint16_t
const int coefSize = rowPitch * kPhaseCount;
// since we are using RGBA format the texture width = kFilterSize / 4
createTexture2D(kFilterSize / 4, kPhaseCount, DXGI_FORMAT_R32G32B32A32_FLOAT,
D3D11_USAGE_DEFAULT, coef_scaler, rowPitch, coefSize, &scalerTex);
createTexture2D(kFilterSize / 4, kPhaseCount, DXGI_FORMAT_R32G32B32A32_FLOAT,
D3D11_USAGE_DEFAULT, coef_usm, rowPitch, coefSize, &usmTex);
Note: It is also possible to specify an fp16 format for the coefficients such as
DXGI_FORMAT_R16G16B16A16_FLOAT. If you do so, use the fp16 coefficients coef_scaler_fp16 and
coef_usm_fp16.
Create Sampler
createLinearClampSampler(&linearClampSampler);
Use the following API call to update the NVIDIA Image Scaling SDK configuration
bool NVScalerUpdateConfig(NISConfig& config,
float sharpness,
uint32_t inputViewportOriginX, uint32_t inputViewportOriginY,
uint32_t inputViewportWidth, uint32_t inputViewportHeight,
uint32_t inputTextureWidth, uint32_t inputTextureHeight,
uint32_t outputViewportOriginX, uint32_t outputViewportOriginY,
uint32_t outputViewportWidth, uint32_t outputViewportHeight,
uint32_t outputTextureWidth, uint32_t outputTextureHeight,
NISHDRMode hdrMode = NISHDRMode::None
);
NVScalerUpdateConfig returns true if the configuration was successful and false if the configuration
could not be set. The input texture sizes should be less than or equal to the output texture sizes. The
algorithm does not check for viewport inconsistencies.
When viewports are required, compile the shader with NIS_VIEWPORT_SUPPORT = 1. The use of
viewports might impact performance when the output viewport size is close to the output texture size.
To improve performance, consider adjusting the dispatch dimensions to accommodate the viewport
size.
Update the constant buffer whenever the input size, sharpness, or scale changes
NISUpdateConfig(m_config, sharpness,
0, 0, inputWidth, inputHeight, inputWidth, inputHeight,
0, 0, outputWidth, outputHeight, outputWidth, outputHeight,
NISHDRMode::None);
updateConstBuffer(&config, csBuffer.Get());
context->Dispatch(UINT(std::ceil(outputWidth / float(blockWidth))),
UINT(std::ceil(outputHeight / float(blockHeight))), 1);
NIS_SCALER should be set to 0 and the optimizer isUscaling argument should be set as false.
Defines defines;
defines.add("NIS_DIRSCALER", isUpscaling);
defines.add("NIS_HDR_MODE", hdrMode);
defines.add("NIS_BLOCK_WIDTH", blockWidth);
defines.add("NIS_BLOCK_HEIGHT", blockHeight);
defines.add("NIS_THREAD_GROUP_SIZE", threadGroupSize);
NVSharpenCS = CompileComputeShader(device, "NIS_Main.hlsl”, &defines);
Note: Compilation of the shader permutations can be performed off-line.
NISConfig config;
createConstBuffer(&config, &csBuffer);
Create Sampler
createLinearClampSampler(&linearClampSampler);
Use the following API call to update the NVIDIA Image Scaling SDK configuration. Since NVSharpen is a
sharpening algorithm only the sharpness and input size are required. For upscaling with sharpening use
NVScaler since it performs both operations at the same time.
bool NVSharpenUpdateConfig(NISConfig& config, float sharpness,
Update the constant buffer whenever the input size or sharpness changes.
NVSharpenUpdateConfig(m_config, sharpness,
0, 0, inputWidth, inputHeight, inputWidth, inputHeight,
0, 0, NISHDRMode::None);
updateConstBuffer(&config, csBuffer.Get());
context->Dispatch(UINT(std::ceil(outputWidth / float(blockWidth))),
UINT(std::ceil(outputHeight / float(blockHeight))), 1);
$> cd samples
$> mkdir build
$> cd build
$> cmake ..
Open the solution with Visual Studio 2019. Right-click the sample project and select "Set as Startup
Project" before building the project.
$> cd samples
$> mkdir build
$> cd build
$> cmake .. -DNIS_VK_SAMPLE=ON
Note: Carefully check texture clarity when NVScaler is enabled and confirm that it matches the texture
clarity when rendering at native resolution with default AA method. Pay attention to textures with text
or other fine detail (e.g., posters on walls, number plates, newspapers, etc.)
If there is a negative bias applied during native resolution rendering, some art assets may have been
tuned for the default bias. When NVScaler is enabled, the bias may be too large or too small compared
to the default leading to poor image quality. In such a case, adjust the “epsilon” value for the
MipLevelBias calculation.
Note: Some rendering engines have a global clamp for the mipmap bias. If such a clamp exists, disable it
when NVScaler is enabled.
6 Appendix
6.1 Notices
6.1.1 Trademarks
NVIDIA and the NVIDIA logo are trademarks and/or registered trademarks of NVIDIA Corporation in the
U.S. and other countries. Other company and product names may be trademarks of the respective
companies with which they are associated.
6.1.2 License
The MIT License(MIT)
Copyright(c) 2022 NVIDIA Corporation
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
6.2.2 GLFW
Copyright (c) 2002-2006 Marcus Geelnard
Copyright (c) 2006-2019 Camilla Löwy
This software is provided 'as-is', without any express or implied warranty. In no event will
the authors be held liable for any damages arising from the use of this software.
Permission is granted to anyone to use this software for any purpose, including commercial
applications, and to alter it and redistribute it freely, subject to the following
restrictions:
The origin of this software must not be misrepresented; you must not claim that you wrote the
original software. If you use this software in a product, an acknowledgment in the product
documentation would be appreciated but is not required.
Altered source versions must be plainly marked as such, and must not be misrepresented as
being the original software.
This notice may not be removed or altered from any source distribution.
Developed by:
LLVM Team
http://llvm.org
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE
SOFTWARE.
==============================================================================
Copyrights and Licenses for Third Party Software Distributed with LLVM:
==============================================================================
The LLVM software contains code written by third parties. Such software will
have its own individual LICENSE.TXT file in the directory in which it appears.
This file will describe the copyrights, license, and restrictions which apply
to that code.
Program Directory
------- ---------
Autoconf llvm/autoconf
llvm/projects/ModuleMaker/autoconf
Google Test llvm/utils/unittest/googletest
OpenBSD regex llvm/lib/Support/{reg*, COPYRIGHT.regex}
pyyaml tests llvm/test/YAMLParser/{*.data, LICENSE.TXT}
ARM contributions llvm/lib/Target/ARM/LICENSE.TXT
md5 contributions llvm/lib/Support/MD5.cpp llvm/include/llvm/Support/MD5.h
miniz llvm/lib/Miniz/miniz.c llvm/include/miniz/miniz.h
llvm/lib/Miniz/LICENSE.txt
tinyexr uses miniz, which is developed by Rich Geldreich [email protected], and licensed
under public domain.
tinyexr tools uses stb, which is licensed under public domain: https://github.com/nothings/stb
tinyexr uses some code from OpenEXR, which is licensed under 3-clause BSD license.
6.2.6 d3dx12.h
Copyright (c) Microsoft. All rights reserved.
This code is licensed under the MIT License (MIT).
THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.