Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support changing frame size during encoding #1069

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

Support changing frame size during encoding #1069

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
864c34e
Support changing frame size during encoding
tongyuantongyu Aug 26, 2022
3f8f3b3
clang-format
tongyuantongyu Aug 30, 2022
b8356ea
test on ssim
tongyuantongyu Aug 31, 2022
a677a39
review changes
tongyuantongyu Sep 2, 2022
d0d7290
Merge remote-tracking branch 'upstream/main' into var-frame-size
tongyuantongyu Jan 15, 2023
d7fe311
Add tests for more encoding cases
tongyuantongyu Jan 15, 2023
5e208db
AOM version check for encoding
tongyuantongyu Jan 15, 2023
35af2e6
skip test if libyuv is not available
tongyuantongyu Jan 15, 2023
ea312fa
Merge branch 'main' into var-frame-size
tongyuantongyu Mar 3, 2023
86382ae
Two (failing) progressive encoding tests
tongyuantongyu Feb 8, 2023
956a3fd
Add test
tongyuantongyu Mar 7, 2023
b69e549
Update branch
tongyuantongyu Sep 13, 2023
382a42a
Update test (still has crash from AOM)
tongyuantongyu Sep 13, 2023
f19fa0e
Fix gainmap, adjust member location
tongyuantongyu Sep 13, 2023
e7d768f
Fix gainmap test
tongyuantongyu Sep 13, 2023
0086558
Fix test compilation
tongyuantongyu Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions include/avif/avif.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1288,6 +1288,9 @@ typedef struct avifScalingMode
// a combination of settings are tweaked to simulate this speed range.
// * Extra layer count: [0 - (AVIF_MAX_AV1_LAYER_COUNT-1)]. Non-zero value indicates a layered
// (progressive) image.
// * Width and height: width and height of encoded image. Default value 0 means infer from first frame.
// For grid image, this is the size of one cell. Value must not be smaller than the largest frame
// to be encoded.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to think about what this value means for grid images.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the easiest choice: encoder->width and encoder-height overrides image->width and image->height, if present.

Alternatively this can be defined as size of the whole grid, and we verify and compute size of each cell internally.

// * Some encoder settings can be changed after encoding starts. Changes will take effect in the next
// call to avifEncoderAddImage().
typedef struct avifEncoder
Expand Down Expand Up @@ -1334,6 +1337,9 @@ typedef struct avifEncoder
// Defaults to AVIF_HEADER_FULL
avifHeaderFormat headerFormat;

uint32_t width;
uint32_t height;

#if defined(AVIF_ENABLE_EXPERIMENTAL_GAIN_MAP)
int qualityGainMap; // changeable encoder setting
#endif
Expand Down
2 changes: 1 addition & 1 deletion include/avif/internal.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -400,7 +400,7 @@ typedef avifBool (*avifCodecGetNextImageFunc)(struct avifCodec * codec,
typedef avifResult (*avifCodecEncodeImageFunc)(struct avifCodec * codec,
avifEncoder * encoder,
const avifImage * image,
avifBool alpha,
avifItemCategory category,
int tileRowsLog2,
int tileColsLog2,
int quantizer,
Expand Down
51 changes: 46 additions & 5 deletions src/codec_aom.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -547,7 +547,7 @@ static avifBool aomCodecEncodeFinish(avifCodec * codec, avifCodecEncodeOutput *
static avifResult aomCodecEncodeImage(avifCodec * codec,
avifEncoder * encoder,
const avifImage * image,
avifBool alpha,
avifItemCategory category,
int tileRowsLog2,
int tileColsLog2,
int quantizer,
Expand All @@ -558,6 +558,13 @@ static avifResult aomCodecEncodeImage(avifCodec * codec,
{
struct aom_codec_enc_cfg * cfg = &codec->internal->cfg;
avifBool quantizerUpdated = AVIF_FALSE;
avifBool alpha = category == AVIF_ITEM_ALPHA;
#if defined(AVIF_ENABLE_EXPERIMENTAL_GAIN_MAP)
avifBool gainMap = category == AVIF_ITEM_GAIN_MAP;
#else
avifBool gainMap = AVIF_FALSE;
#endif
const int aomVersion = aom_codec_version();

// For encoder->scalingMode.horizontal and encoder->scalingMode.vertical to take effect in AOM
// encoder, config should be applied for each frame, so we don't care about changes on these
Expand Down Expand Up @@ -601,7 +608,6 @@ static avifResult aomCodecEncodeImage(avifCodec * codec,

// aom_codec.h says: aom_codec_version() == (major<<16 | minor<<8 | patch)
static const int aomVersion_2_0_0 = (2 << 16);
const int aomVersion = aom_codec_version();
if ((aomVersion < aomVersion_2_0_0) && (image->depth > 8)) {
// Due to a known issue with libaom v1.0.0-errata1-avif, 10bpc and
// 12bpc image encodes will call the wrong variant of
Expand Down Expand Up @@ -697,6 +703,12 @@ static avifResult aomCodecEncodeImage(avifCodec * codec,
cfg->g_w = image->width;
cfg->g_h = image->height;

// gain map has its own dimension
if (!gainMap) {
cfg->g_forced_max_frame_width = encoder->width;
cfg->g_forced_max_frame_height = encoder->height;
}

// Detect the libaom v3.6.0 bug described in
// https://crbug.com/aomedia/2871#c12. See the changes to
// av1/encoder/encoder.c in
Expand Down Expand Up @@ -747,6 +759,11 @@ static avifResult aomCodecEncodeImage(avifCodec * codec,
if (disableLaggedOutput) {
cfg->g_lag_in_frames = 0;
}
if ((encoder->width || encoder->height) && (cfg->g_lag_in_frames > 1)) {
// libaom does not allow changing frame dimension if
// g_lag_in_frames > 1.
cfg->g_lag_in_frames = 1;
}
if (encoder->maxThreads > 1) {
cfg->g_threads = encoder->maxThreads;
}
Expand Down Expand Up @@ -877,9 +894,33 @@ static avifResult aomCodecEncodeImage(avifCodec * codec,
}
} else {
avifBool dimensionsChanged = AVIF_FALSE;
if ((cfg->g_w != image->width) || (cfg->g_h != image->height)) {
// We are not ready for dimension change for now.
return AVIF_RESULT_NOT_IMPLEMENTED;
if (!gainMap && ((cfg->g_w != image->width) || (cfg->g_h != image->height))) {
static const int aomVersion_3_6_0 = (3 << 16) + (6 << 8);
if (aomVersion < aomVersion_3_6_0) {
// Due to a bug in libaom before v3.6.0 encoding 10bpc and 12bpc images
// with changing dimension will crash the encoder.
if (image->depth > 8) {
avifDiagnosticsPrintf(codec->diag,
"Detected libaom bug with high bitdepth images and changing dimension. Upgrade to libaom v3.6.0 or later.");
return AVIF_RESULT_INCOMPATIBLE_IMAGE;
}

// There exists a bug in libaom's buffer allocation logic before v3.6.0
// where it allocates buffers based on g_w and g_h of first frame instead of
// g_forced_max_frame_width and g_forced_max_frame_height, so encoding frames
// of increasing size will crash the encoder.
//
// This check is stricter than it needs to be, but we don't track the size of
// first image but only the last successful encoded one.
if ((cfg->g_w < image->width) || (cfg->g_h < image->height)) {
avifDiagnosticsPrintf(codec->diag,
"Detected libaom bug with increasing encoding dimension. Upgrade to libaom v3.6.0 or later.");
return AVIF_RESULT_INCOMPATIBLE_IMAGE;
}
}
cfg->g_w = image->width;
cfg->g_h = image->height;
dimensionsChanged = AVIF_TRUE;
}
if (alpha) {
if (encoderChanges & (AVIF_ENCODER_CHANGE_MIN_QUANTIZER_ALPHA | AVIF_ENCODER_CHANGE_MAX_QUANTIZER_ALPHA)) {
Expand Down
3 changes: 2 additions & 1 deletion src/codec_avm.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -530,7 +530,7 @@ static avifBool avmCodecEncodeFinish(avifCodec * codec, avifCodecEncodeOutput *
static avifResult avmCodecEncodeImage(avifCodec * codec,
avifEncoder * encoder,
const avifImage * image,
avifBool alpha,
avifItemCategory category,
int tileRowsLog2,
int tileColsLog2,
int quantizer,
Expand All @@ -541,6 +541,7 @@ static avifResult avmCodecEncodeImage(avifCodec * codec,
{
struct aom_codec_enc_cfg * cfg = &codec->internal->cfg;
avifBool quantizerUpdated = AVIF_FALSE;
avifBool alpha = category == AVIF_ITEM_ALPHA;

// For encoder->scalingMode.horizontal and encoder->scalingMode.vertical to take effect in AV2
// encoder, config should be applied for each frame, so we don't care about changes on these
Expand Down
9 changes: 8 additions & 1 deletion src/codec_rav1e.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ static avifBool rav1eSupports400(void)
static avifResult rav1eCodecEncodeImage(avifCodec * codec,
avifEncoder * encoder,
const avifImage * image,
avifBool alpha,
avifItemCategory category,
int tileRowsLog2,
int tileColsLog2,
int quantizer,
Expand All @@ -61,6 +61,8 @@ static avifResult rav1eCodecEncodeImage(avifCodec * codec,
uint32_t addImageFlags,
avifCodecEncodeOutput * output)
{
avifBool alpha = category == AVIF_ITEM_ALPHA;

// rav1e does not support changing encoder settings.
if (encoderChanges) {
return AVIF_RESULT_NOT_IMPLEMENTED;
Expand All @@ -82,6 +84,11 @@ static avifResult rav1eCodecEncodeImage(avifCodec * codec,
// rav1e does not support disabling lagged output. See https://github.com/xiph/rav1e/issues/2267. Ignore this setting.
(void)disableLaggedOutput;

// rav1e does not support overriding maximum frame width/height in sequence header
if (encoder->width || encoder->height) {
return AVIF_RESULT_NOT_IMPLEMENTED;
}

avifResult result = AVIF_RESULT_UNKNOWN_ERROR;

RaConfig * rav1eConfig = NULL;
Expand Down
6 changes: 5 additions & 1 deletion src/codec_svt.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ static avifResult dequeue_frame(avifCodec * codec, avifCodecEncodeOutput * outpu
static avifResult svtCodecEncodeImage(avifCodec * codec,
avifEncoder * encoder,
const avifImage * image,
avifBool alpha,
avifItemCategory category,
int tileRowsLog2,
int tileColsLog2,
int quantizer,
Expand All @@ -54,6 +54,8 @@ static avifResult svtCodecEncodeImage(avifCodec * codec,
uint32_t addImageFlags,
avifCodecEncodeOutput * output)
{
avifBool alpha = category == AVIF_ITEM_ALPHA;

// SVT-AV1 does not support changing encoder settings.
if (encoderChanges) {
return AVIF_RESULT_NOT_IMPLEMENTED;
Expand Down Expand Up @@ -128,6 +130,8 @@ static avifResult svtCodecEncodeImage(avifCodec * codec,

svt_config->source_width = image->width;
svt_config->source_height = image->height;
svt_config->forced_max_frame_width = encoder->width;
svt_config->forced_max_frame_height = encoder->height;
svt_config->logical_processors = encoder->maxThreads;
svt_config->enable_adaptive_quantization = 2;
// disable 2-pass
Expand Down
46 changes: 30 additions & 16 deletions src/write.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,8 @@ avifEncoder * avifEncoderCreate(void)
encoder->maxQuantizer = AVIF_QUANTIZER_WORST_QUALITY;
encoder->minQuantizerAlpha = AVIF_QUANTIZER_BEST_QUALITY;
encoder->maxQuantizerAlpha = AVIF_QUANTIZER_WORST_QUALITY;
encoder->width = 0;
encoder->height = 0;
encoder->tileRowsLog2 = 0;
encoder->tileColsLog2 = 0;
encoder->autoTiling = AVIF_FALSE;
Expand Down Expand Up @@ -490,6 +492,8 @@ static void avifEncoderBackupSettings(avifEncoder * encoder)
lastEncoder->timescale = encoder->timescale;
lastEncoder->repetitionCount = encoder->repetitionCount;
lastEncoder->extraLayerCount = encoder->extraLayerCount;
lastEncoder->width = encoder->width;
lastEncoder->height = encoder->height;
lastEncoder->minQuantizer = encoder->minQuantizer;
lastEncoder->maxQuantizer = encoder->maxQuantizer;
lastEncoder->minQuantizerAlpha = encoder->minQuantizerAlpha;
Expand Down Expand Up @@ -517,7 +521,8 @@ static avifBool avifEncoderDetectChanges(const avifEncoder * encoder, avifEncode
if ((lastEncoder->codecChoice != encoder->codecChoice) || (lastEncoder->maxThreads != encoder->maxThreads) ||
(lastEncoder->speed != encoder->speed) || (lastEncoder->keyframeInterval != encoder->keyframeInterval) ||
(lastEncoder->timescale != encoder->timescale) || (lastEncoder->repetitionCount != encoder->repetitionCount) ||
(lastEncoder->extraLayerCount != encoder->extraLayerCount)) {
(lastEncoder->extraLayerCount != encoder->extraLayerCount) || (lastEncoder->width != encoder->width) ||
(lastEncoder->height != encoder->height)) {
return AVIF_FALSE;
}

Expand Down Expand Up @@ -1277,6 +1282,10 @@ static avifResult avifEncoderAddImageInternal(avifEncoder * encoder,
return AVIF_RESULT_NO_CONTENT;
}

if ((encoder->width && (encoder->width < firstCell->width)) || (encoder->height && (encoder->height < firstCell->height))) {
return AVIF_RESULT_INCOMPATIBLE_IMAGE;
}

AVIF_CHECKRES(avifValidateGrid(gridCols, gridRows, cellImages, /*validateGainMap=*/AVIF_FALSE, &encoder->diag));

#if defined(AVIF_ENABLE_EXPERIMENTAL_GAIN_MAP)
Expand Down Expand Up @@ -1592,7 +1601,7 @@ static avifResult avifEncoderAddImageInternal(avifEncoder * encoder,
avifResult encodeResult = item->codec->encodeImage(item->codec,
encoder,
cellImage,
item->itemCategory == AVIF_ITEM_ALPHA,
item->itemCategory,
encoder->data->tileRowsLog2,
encoder->data->tileColsLog2,
quantizer,
Expand Down Expand Up @@ -2158,6 +2167,9 @@ avifResult avifEncoderFinish(avifEncoder * encoder, avifRWData * output)
#endif // AVIF_ENABLE_EXPERIMENTAL_AVIR

const avifImage * imageMetadata = encoder->data->imageMetadata;
const uint32_t cellWidth = encoder->width ? encoder->width : imageMetadata->width;
const uint32_t cellHeight = encoder->height ? encoder->height : imageMetadata->height;

// The epoch for creation_time and modification_time is midnight, Jan. 1,
// 1904, in UTC time. Add the number of seconds between that epoch and the
// Unix epoch.
Expand Down Expand Up @@ -2398,14 +2410,16 @@ avifResult avifEncoderFinish(avifEncoder * encoder, avifRWData * output)
}

const avifImage * itemMetadata = imageMetadata;
uint32_t imageWidth = cellWidth;
uint32_t imageHeight = cellHeight;
#if defined(AVIF_ENABLE_EXPERIMENTAL_GAIN_MAP)
if (item->itemCategory == AVIF_ITEM_GAIN_MAP) {
itemMetadata = itemMetadata->gainMap.image;
assert(itemMetadata);
imageWidth = itemMetadata->width;
imageHeight = itemMetadata->height;
}
#endif
uint32_t imageWidth = itemMetadata->width;
uint32_t imageHeight = itemMetadata->height;
if (isGrid) {
imageWidth = item->gridWidth;
imageHeight = item->gridHeight;
Expand Down Expand Up @@ -2646,8 +2660,8 @@ avifResult avifEncoderFinish(avifEncoder * encoder, avifRWData * output)
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 0)); // template int(16) volume = {if track_is_audio 0x0100 else 0};
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 0)); // const unsigned int(16) reserved = 0;
AVIF_CHECKRES(avifRWStreamWrite(&s, unityMatrix, sizeof(unityMatrix))); // template int(32)[9] matrix= // { 0x00010000,0,0,0,0x00010000,0,0,0,0x40000000 };
AVIF_CHECKRES(avifRWStreamWriteU32(&s, imageMetadata->width << 16)); // unsigned int(32) width;
AVIF_CHECKRES(avifRWStreamWriteU32(&s, imageMetadata->height << 16)); // unsigned int(32) height;
AVIF_CHECKRES(avifRWStreamWriteU32(&s, cellWidth << 16)); // unsigned int(32) width;
AVIF_CHECKRES(avifRWStreamWriteU32(&s, cellHeight << 16)); // unsigned int(32) height;
avifRWStreamFinishBox(&s, tkhd);

if (item->irefToID != 0) {
Expand Down Expand Up @@ -2729,16 +2743,16 @@ avifResult avifEncoderFinish(avifEncoder * encoder, avifRWData * output)
AVIF_CHECKRES(avifRWStreamWriteU32(&s, 1)); // unsigned int(32) entry_count;
avifBoxMarker imageItem;
AVIF_CHECKRES(avifRWStreamWriteBox(&s, encoder->data->imageItemType, AVIF_BOX_SIZE_TBD, &imageItem));
AVIF_CHECKRES(avifRWStreamWriteZeros(&s, 6)); // const unsigned int(8)[6] reserved = 0;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 1)); // unsigned int(16) data_reference_index;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 0)); // unsigned int(16) pre_defined = 0;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 0)); // const unsigned int(16) reserved = 0;
AVIF_CHECKRES(avifRWStreamWriteZeros(&s, sizeof(uint32_t) * 3)); // unsigned int(32)[3] pre_defined = 0;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, (uint16_t)imageMetadata->width)); // unsigned int(16) width;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, (uint16_t)imageMetadata->height)); // unsigned int(16) height;
AVIF_CHECKRES(avifRWStreamWriteU32(&s, 0x00480000)); // template unsigned int(32) horizresolution
AVIF_CHECKRES(avifRWStreamWriteU32(&s, 0x00480000)); // template unsigned int(32) vertresolution
AVIF_CHECKRES(avifRWStreamWriteU32(&s, 0)); // const unsigned int(32) reserved = 0;
AVIF_CHECKRES(avifRWStreamWriteZeros(&s, 6)); // const unsigned int(8)[6] reserved = 0;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 1)); // unsigned int(16) data_reference_index;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 0)); // unsigned int(16) pre_defined = 0;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 0)); // const unsigned int(16) reserved = 0;
AVIF_CHECKRES(avifRWStreamWriteZeros(&s, sizeof(uint32_t) * 3)); // unsigned int(32)[3] pre_defined = 0;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, (uint16_t)cellWidth)); // unsigned int(16) width;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, (uint16_t)cellHeight)); // unsigned int(16) height;
AVIF_CHECKRES(avifRWStreamWriteU32(&s, 0x00480000)); // template unsigned int(32) horizresolution
AVIF_CHECKRES(avifRWStreamWriteU32(&s, 0x00480000)); // template unsigned int(32) vertresolution
AVIF_CHECKRES(avifRWStreamWriteU32(&s, 0)); // const unsigned int(32) reserved = 0;
AVIF_CHECKRES(avifRWStreamWriteU16(&s, 1)); // template unsigned int(16) frame_count = 1;
AVIF_CHECKRES(avifRWStreamWriteChars(&s, "\012AOM Coding", 11)); // string[32] compressorname;
AVIF_CHECKRES(avifRWStreamWriteZeros(&s, 32 - 11)); //
Expand Down
9 changes: 7 additions & 2 deletions tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,11 @@ if(AVIF_ENABLE_GTEST)
target_include_directories(avifbasictest PRIVATE ${GTEST_INCLUDE_DIRS})
add_test(NAME avifbasictest COMMAND avifbasictest)

add_executable(avifchangedimensiontest gtest/avifchangedimensiontest.cc)
target_link_libraries(avifchangedimensiontest aviftest_helpers ${GTEST_BOTH_LIBRARIES})
target_include_directories(avifchangedimensiontest PRIVATE ${GTEST_INCLUDE_DIRS})
add_test(NAME avifchangedimensiontest COMMAND avifchangedimensiontest)

add_executable(avifchangesettingtest gtest/avifchangesettingtest.cc)
target_link_libraries(avifchangesettingtest aviftest_helpers ${GTEST_BOTH_LIBRARIES})
target_include_directories(avifchangesettingtest PRIVATE ${GTEST_INCLUDE_DIRS})
Expand Down Expand Up @@ -192,9 +197,9 @@ if(AVIF_ENABLE_GTEST)
add_test(NAME avifpng16bittest COMMAND avifpng16bittest ${CMAKE_CURRENT_SOURCE_DIR}/data/)

add_executable(avifprogressivetest gtest/avifprogressivetest.cc)
target_link_libraries(avifprogressivetest aviftest_helpers ${GTEST_BOTH_LIBRARIES})
target_link_libraries(avifprogressivetest aviftest_helpers ${GTEST_LIBRARIES})
target_include_directories(avifprogressivetest PRIVATE ${GTEST_INCLUDE_DIRS})
add_test(NAME avifprogressivetest COMMAND avifprogressivetest)
add_test(NAME avifprogressivetest COMMAND avifprogressivetest ${CMAKE_CURRENT_SOURCE_DIR}/data/)

add_executable(avifreadimagetest gtest/avifreadimagetest.cc)
target_link_libraries(avifreadimagetest aviftest_helpers ${GTEST_LIBRARIES})
Expand Down
Binary file added tests/data/dog_1080p.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/data/dog_blur_1080p.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/data/dog_blur_540p.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading