Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add calculations for YUV / planar images #479

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add calculations for YUV / planar images #479

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
549f79e
Add calculations for YUV / planar images
anishmgoyal Jun 6, 2024
4a9ce60
Address PR comments
anishmgoyal Jun 7, 2024
f9ac2fb
Address PR comments
anishmgoyal Jun 10, 2024
9f6e6a6
Fix headers in graphics_util.cpp
anishmgoyal Jun 10, 2024
ffd6e73
Add more documentation
anishmgoyal Jun 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions include/ppx/graphics_util.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,13 +150,23 @@ class TextureOptions
grfx::Texture** ppTexture,
const TextureOptions& options);

// Splits the frames from a raw video, based on the format of the frames,
// and metadata such as height and width. This assumes raw video, with no
// metadata in the file itself, and no audio tracks (such as a camera
// feed). Returns if the operation succeeded.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Returns if the operation succeeded"? Did you mean to say "Returns X", cause couldn't it return error code as well?

// path: The path to the file containing the video.
// format: Describes the video format; this is important for determining
// the size of a frame.
// width: The width of each frame, in pixels, with no subsampling applied.
// height: The height of each frame, in pixels, with no subsampling applied.
// pFrames: A vector where resulting frames will be stored. This should not
// be null.
friend Result LoadFramesFromRawVideo(
anishmgoyal marked this conversation as resolved.
Show resolved Hide resolved
Keenuts marked this conversation as resolved.
Show resolved Hide resolved
grfx::Queue* pQueue,
const std::filesystem::path& path,
uint32_t width,
uint32_t height,
grfx::Texture** ppTexture,
const TextureOptions& options);
const std::filesystem::path& path,
grfx::Format format,
uint32_t width,
uint32_t height,
std::vector<std::vector<char>>* pFrames);
};

//! @fn CreateTextureFromBitmap
Expand Down
17 changes: 14 additions & 3 deletions include/ppx/grfx/grfx_format.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,13 +285,24 @@ enum FormatPlaneChromaType
FORMAT_PLANE_CHROMA_TYPE_CHROMA,
};

// Note: this is distinct from FormatComponentBit because in the case of a
// member of an image plane, we only want to be able to specify one component
// bit.
enum FormatPlaneComponentType
{
FORMAT_PLANE_COMPONENT_TYPE_UNDEFINED,
FORMAT_PLANE_COMPONENT_TYPE_RED,
FORMAT_PLANE_COMPONENT_TYPE_GREEN,
FORMAT_PLANE_COMPONENT_TYPE_BLUE,
};

struct FormatPlaneDesc
{
struct Member
{
// Note: it's expected that only one bit would be set here. That being
// said, this is mostly to add clarity to plane component definitions.
FormatComponentBit component;
// For debugging purposes: the color component that this plane member
// describes.
FormatPlaneComponentType component;
// This defines whether this is a luma value, chroma value, or neither
// (will be set to undefined for non-YCbCr types).
FormatPlaneChromaType type;
Expand Down
115 changes: 78 additions & 37 deletions src/ppx/graphics_util.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.

#include "ppx/graphics_util.h"

#include <algorithm>

#include "ppx/generate_mip_shader_VK.h"
#include "ppx/generate_mip_shader_DX.h"
#include "ppx/graphics_util.h"
#include "ppx/bitmap.h"
#include "ppx/fs.h"
#include "ppx/mipmap.h"
Expand All @@ -34,27 +37,34 @@ namespace grfx_util {

namespace {

const uint32_t kYuvWidth = 3152; // 3000;
const uint32_t kYuvHeight = 3840; // 3000;

// Start planar image helper functions
Keenuts marked this conversation as resolved.
Show resolved Hide resolved

uint32_t GetPlaneHeightForCopy(
// Gets the height of a single plane, in terms of number of pixels represented.
// This doesn't directly correlate to the number of bits / bytes for the plane's
// height. The value returned can be used in a copy-image-to-buffer command.
// plane: The plane to get the height for (containing information about the
// color components represented in the plane).
// subsampling: The type of subsampling applied to chroma values for the image
// (e.g. 444, 422, 420).
// imageHeight: The height of the image, in pixels, with no subsampling applied.
uint32_t GetPlaneHeightInPixels(
const grfx::FormatPlaneDesc::Plane& plane,
grfx::FormatChromaSubsampling subsampling,
uint32_t imageHeight)
{
bool hasColSubsampling = subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_420;
bool hasColSubsampling = (subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_420);
Keenuts marked this conversation as resolved.
Show resolved Hide resolved
bool hasChromaValue = false;
bool hasLumaValue = false;
for (auto it = plane.members.begin(); it != plane.members.end(); ++it) {
const grfx::FormatPlaneDesc::Member& member = *it;
for (const grfx::FormatPlaneDesc::Member& member : plane.members) {
if (member.type == grfx::FORMAT_PLANE_CHROMA_TYPE_CHROMA) {
hasChromaValue = true;
}
else {
else if (member.type == grfx::FORMAT_PLANE_CHROMA_TYPE_LUMA) {
hasLumaValue = true;
anishmgoyal marked this conversation as resolved.
Show resolved Hide resolved
}
else {
PPX_LOG_WARN("Member " << member.component << "has unknown chroma type.");
}
}

if (hasColSubsampling && hasChromaValue) {
Expand All @@ -76,16 +86,23 @@ uint32_t GetPlaneHeightForCopy(
return imageHeight;
}

uint32_t GetPlaneWidthForCopy(
// Gets the width of a single plane, in terms of number of pixels represented.
// This doesn't directly correlate to the number of bits / bytes for the plane's
// height. The value returned can be used in a copy-image-to-buffer command.
// plane: The plane to get the width for (containing information about the
// color components represented in the plane).
// subsampling: The type of subsampling applied to chroma values for the image
// (e.g. 444, 422, 420).
// imageWidth: The width of the image, in pixels, with no subsampling applied.
uint32_t GetPlaneWidthInPixels(
const grfx::FormatPlaneDesc::Plane& plane,
grfx::FormatChromaSubsampling subsampling,
uint32_t imageWidth)
{
bool hasRowSubsampling = subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_420 ||
subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_422;
bool hasRowSubsampling = (subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_420) ||
(subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_422);
bool hasChromaValue = false;
for (auto it = plane.members.begin(); it != plane.members.end(); ++it) {
const grfx::FormatPlaneDesc::Member& member = *it;
for (const grfx::FormatPlaneDesc::Member& member : plane.members) {
if (member.type == grfx::FORMAT_PLANE_CHROMA_TYPE_CHROMA) {
hasChromaValue = true;
break;
Expand All @@ -102,25 +119,34 @@ uint32_t GetPlaneWidthForCopy(
return imageWidth;
}

// Gets the size of an image plane in bytes.
// plane: The plane to get information for. (Contains information about the
// color components represented by this plane, and their bit counts).
// subsampling: The type of chroma subsampling applied to this image (e.g.
// 444, 422, 420).
// width: The width of the image, in pixels, with no subsampling applied.
// height: The height of the image, in pixels, with no subsampling applied.
uint32_t GetPlaneSizeInBytes(
const grfx::FormatPlaneDesc::Plane& plane,
grfx::FormatChromaSubsampling subsampling,
uint32_t width,
uint32_t height)
{
bool hasColSubsampling = subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_420;
bool hasRowSubsampling = hasColSubsampling || subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_422;
bool hasColSubsampling = (subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_420);
bool hasRowSubsampling = hasColSubsampling || (subsampling == grfx::FORMAT_CHROMA_SUBSAMPLING_422);
bool hasChromaValue = false;
bool hasLumaValue = false;
uint32_t rowBitFactor = 0;
for (auto it = plane.members.begin(); it != plane.members.end(); ++it) {
const grfx::FormatPlaneDesc::Member& member = *it;
for (const grfx::FormatPlaneDesc::Member& member : plane.members) {
if (member.type == grfx::FORMAT_PLANE_CHROMA_TYPE_CHROMA) {
hasChromaValue = true;
}
else if (member.type == grfx::FORMAT_PLANE_CHROMA_TYPE_LUMA) {
hasLumaValue = true;
}
else {
PPX_LOG_WARN("Member " << member.component << "has unknown chroma type.");
}

// We only subsample chroma values.
if (member.type == grfx::FORMAT_PLANE_CHROMA_TYPE_CHROMA && hasRowSubsampling) {
Expand Down Expand Up @@ -152,6 +178,13 @@ uint32_t GetPlaneSizeInBytes(
return (width * rowBitFactor * height) / 8;
}

// Gets the total size of a planar image in bytes, by calculating the size of
// each plane individually.
// formatDesc: Information about the image format, such as the components
// represented, etc.
// planeDesc: Information about the components in the current image plane.
// width: The width of the image, in pixels, with no subsampling applied.
// height: The height of the image, in pixels, with no subsampling applied.
uint32_t GetPlanarImageSizeInBytes(
const grfx::FormatDesc& formatDesc,
const grfx::FormatPlaneDesc& planeDesc,
Expand All @@ -161,8 +194,8 @@ uint32_t GetPlanarImageSizeInBytes(
grfx::FormatChromaSubsampling subsampling = formatDesc.chromaSubsampling;

uint32_t imageSize = 0;
for (auto it = planeDesc.planes.begin(); it != planeDesc.planes.end(); ++it) {
imageSize += GetPlaneSizeInBytes(*it, subsampling, width, height);
for (const grfx::FormatPlaneDesc::Plane& plane : planeDesc.planes) {
imageSize += GetPlaneSizeInBytes(plane, subsampling, width, height);
}
return imageSize;
}
Expand Down Expand Up @@ -1667,36 +1700,44 @@ Result LoadFramesFromRawVideo(
return ppx::ERROR_FAILED;
}

std::optional<grfx::FormatPlaneDesc> formatPlanes = grfx::GetFormatPlaneDescription(format);

uint32_t frameSizeInBytes = 0;
if (formatPlanes.has_value()) {
frameSizeInBytes = GetPlanarImageSizeInBytes(*formatDesc, *formatPlanes, width, height);
uint32_t frameSize = 0; // As measured in bytes, not pixels.
if (formatDesc->isPlanar) {
std::optional<grfx::FormatPlaneDesc> formatPlanes = grfx::GetFormatPlaneDescription(format);
PPX_ASSERT_MSG(formatPlanes.has_value(), "No planes found for format " << format);
frameSize = GetPlanarImageSizeInBytes(*formatDesc, *formatPlanes, width, height);
}
else {
frameSizeInBytes = formatDesc->bytesPerTexel * width * height;
frameSize = formatDesc->bytesPerTexel * width * height;
}

ppx::fs::File file;
if (!file.Open(path)) {
PPX_ASSERT_MSG(false, "Cannot open the file!");
return ppx::ERROR_FAILED;
}
const size_t size = file.GetLength();
const size_t fileSize = file.GetLength();

if (size % frameSizeInBytes != 0) {
PPX_LOG_WARN("There may be partial frames in the raw video file at " << path << "; loading as much as possible.");
}

std::vector<char> buffer(frameSizeInBytes);
std::vector<char> buffer(frameSize);
size_t totalRead = 0;
while (totalRead + frameSizeInBytes <= size) {
const size_t readSize = file.Read(buffer.data(), size);
if (readSize != size) {
PPX_ASSERT_MSG(false, "Cannot read the content of the file!");
return ppx::ERROR_FAILED;
while (totalRead < fileSize) {
const size_t bytesRead = file.Read(buffer.data(), frameSize);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this safe to do after a std::move operation? Would buffer have frameSize capacity?

if (bytesRead < frameSize) {
// If we didn't read as many bytes as we expected to, and we haven't
// reached the end of the file, this is an error.
if (totalRead + bytesRead < fileSize) {
PPX_ASSERT_MSG(
false,
"Unable to load video frame; expected " << frameSize << " but read " << bytesRead << "bytes (previously read " << totalRead << ").");
return ppx::ERROR_FAILED;
}
// Otherwise, fill the rest of the buffer with 0s.
else {
PPX_LOG_WARN("Read " << bytesRead << " bytes for the last frame of the video at " << path << "; filling the rest of the frame with 0s.");
std::fill(buffer.begin() + bytesRead, buffer.end(), 0);
}
}
pFrames->push_back(std::move(buffer));
totalRead += bytesRead;
}

return ppx::SUCCESS;
Expand Down
22 changes: 11 additions & 11 deletions src/ppx/grfx/grfx_format.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ namespace grfx {

#define UNCOMPRESSED_FORMAT(Name, Type, Aspect, BytesPerTexel, BytesPerComponent, Layout, Component, ComponentOffsets) \
{ \
"" #Name, \
#Name, \
FORMAT_DATA_TYPE_##Type, \
FORMAT_ASPECT_##Aspect, \
BytesPerTexel, \
Expand All @@ -35,12 +35,12 @@ namespace grfx {

#define COMPRESSED_FORMAT(Name, Type, BytesPerBlock, BlockWidth, Component) \
{ \
"" #Name, \
#Name, \
FORMAT_DATA_TYPE_##Type, \
FORMAT_ASPECT_COLOR, \
BytesPerBlock, \
BlockWidth, \
-1, \
/*bytesPerComponent=*/-1, \
FORMAT_LAYOUT_COMPRESSED, \
FORMAT_COMPONENT_##Component, \
OFFSETS_UNDEFINED, \
Expand All @@ -50,12 +50,12 @@ namespace grfx {

#define UNCOMP_PLANAR_FORMAT(Name, Type, Aspect, BytesPerTexel, Layout, Component, ChromaSubsampling) \
{ \
"" #Name, \
#Name, \
FORMAT_DATA_TYPE_##Type, \
FORMAT_ASPECT_##Aspect, \
BytesPerTexel, \
/*blockWidth=*/1, \
-1, \
/*bytesPerComponent=*/-1, \
FORMAT_LAYOUT_##Layout, \
FORMAT_COMPONENT_##Component, \
OFFSETS_UNDEFINED, \
Expand All @@ -71,12 +71,12 @@ namespace grfx {
#define OFFSETS_RGBA(R, G, B, A) { R, G, B, A }
// clang-format on

#define PLANE_MEMBER(Component, Type, BitSize) \
FormatPlaneDesc::Member \
{ \
FORMAT_COMPONENT_##Component, \
FORMAT_PLANE_CHROMA_TYPE_##Type, \
BitSize \
#define PLANE_MEMBER(Component, Type, BitSize) \
FormatPlaneDesc::Member \
{ \
FORMAT_PLANE_COMPONENT_TYPE_##Component, \
FORMAT_PLANE_CHROMA_TYPE_##Type, \
BitSize \
}

FormatPlaneDesc::FormatPlaneDesc(std::initializer_list<std::initializer_list<FormatPlaneDesc::Member>>&& planes)
Expand Down
Loading