Saturday, October 6, 2012

Raspberry Pi as a hardware assisted H264 encoder

One of the amazing things about the Pi is despite its limited CPU power, it has an amazing GPU. This can observed from the fact: 

  • It can do high profile h264 encode/decode (1080p @ 30 fps)
  • It can play Quake 3
  • It can run fancy Qt demos 
Just recently the default Raspberry Pi firmware now has the h264 encoder feature. kulve from the Raspberry Pi forum gave a step by step instruction in the following post: 


I've captured the code here in case the link expires someday: 

# Get The Hobbit in 640 x 272 H264
youtube-dl -f 18 "http://www.youtube.com/watch?v=SDnYMbYB-nU"
 
# Convert 9 seconds of frames to PNG
mplayer -vo png -nosound -ss 61 -endpos 9  SDnYMbYB-nU.mp4
 
# Convert PNG frames to raw RGB (don't ming the weird .png.raw naming)
for i in *png; do convert $i rgb:$i.raw; done

hello_video.bin testout.h264

ffmpeg -f h264 -i testout.h264 -vcodec copy outtest.mp4

mplayer outtest.mp4

I had to install a couple utilities, including youtube-dl, convert, mplayer and ffmpeg. The youtube-dl from the repository seemed to be obsolete already, and I had to download a new one from the official source. 

The hello_video.bin is written by kulve from the Raspberry Pi forum, with the code listed below: 
/*
Copyright (c) 2012, Broadcom Europe Ltd
Copyright (c) 2012, Kalle Vahlman 
                    Tuomas Kulve 
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the copyright holder nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

// Video deocode demo using OpenMAX IL though the ilcient helper library

#include 
#include 
#include 

#include "bcm_host.h"
#include "ilclient.h"

#define RAW_WIDTH    640
#define RAW_HEIGHT   272
#define RAW_BPP      3
#define RAW_SIZE     (RAW_WIDTH * RAW_HEIGHT * RAW_BPP)

static int read_raw_rgb(void *buf, OMX_U32 *filledLen, int filenumber)
{
  char filename[256];
  FILE *f;
  size_t ret;

  snprintf(filename, 256, "%08d.png.raw", filenumber);

  f = fopen(filename, "r");
  if (!f) {
 printf("Failed to open '%s'\n", filename);
 return 0;
  }

  ret = fread(buf, 1, RAW_SIZE, f);
  if (ret < RAW_SIZE) {
 printf("Failed to read '%s': %d\n", filename, ret);
 return 0;
  }

  *filledLen = RAW_SIZE;

  fclose(f);
  return 1;
}

static void print_def(OMX_PARAM_PORTDEFINITIONTYPE def)
{
   printf("Port %lu: %s %lu/%lu %lu %lu %s,%s,%s %lux%lu %lux%lu @%lu %u\n",
          def.nPortIndex,
          def.eDir == OMX_DirInput ? "in" : "out",
          def.nBufferCountActual,
          def.nBufferCountMin,
          def.nBufferSize,
          def.nBufferAlignment,
          def.bEnabled ? "enabled" : "disabled",
          def.bPopulated ? "populated" : "not pop.",
          def.bBuffersContiguous ? "contig." : "not cont.",
          def.format.video.nFrameWidth,
          def.format.video.nFrameHeight,
          def.format.video.nStride,
          def.format.video.nSliceHeight,
          def.format.video.xFramerate,
          def.format.video.eColorFormat);
}

static int video_encode_test(char *outputfilename)
{
   OMX_VIDEO_PARAM_PORTFORMATTYPE format;
   OMX_PARAM_PORTDEFINITIONTYPE def;
   COMPONENT_T *video_encode = NULL;
   COMPONENT_T *list[5];
   OMX_BUFFERHEADERTYPE *buf;
   OMX_BUFFERHEADERTYPE *out;
   OMX_ERRORTYPE r;
   ILCLIENT_T *client;
   int status = 0;
   int filenumber = 0;
   FILE *outf;

   memset(list, 0, sizeof(list));

   if((client = ilclient_init()) == NULL)
   {
      return -3;
   }

   if(OMX_Init() != OMX_ErrorNone)
   {
      ilclient_destroy(client);
      return -4;
   }

   // create video_encode
   r = ilclient_create_component(client, &video_encode, "video_encode",
         ILCLIENT_DISABLE_ALL_PORTS |
         ILCLIENT_ENABLE_INPUT_BUFFERS |
         ILCLIENT_ENABLE_OUTPUT_BUFFERS);
   if (r != 0) {
      printf("ilclient_create_component() for video_encode failed with %x!\n", r);
      exit(1);
   }
   list[0] = video_encode;

   memset(&def, 0, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
   def.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
   def.nVersion.nVersion = OMX_VERSION;
   def.nPortIndex = 200;

   if (OMX_GetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamPortDefinition, &def) != OMX_ErrorNone) {
      printf("%s:%d: OMX_GetParameter() for video_encode port 200 failed!\n", __FUNCTION__, __LINE__);
      exit(1);
   }

   print_def(def);

// Port 200: in 1/1 115200 16 enabled,not pop.,not cont. 320x240 320x240 @1966080 20
   def.format.video.nFrameWidth = RAW_WIDTH;
   def.format.video.nFrameHeight = RAW_HEIGHT;
   def.format.video.xFramerate = 30 << 16;
   def.format.video.nSliceHeight = def.format.video.nFrameHeight;
   def.format.video.nStride = def.format.video.nFrameWidth;
   def.format.video.eColorFormat = OMX_COLOR_Format24bitBGR888;

   print_def(def);

   r = OMX_SetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamPortDefinition, &def);
   if (r != OMX_ErrorNone) {
      printf("%s:%d: OMX_SetParameter() for video_encode port 200 failed with %x!\n", __FUNCTION__, __LINE__, r);
      exit(1);
   }

   memset(&format, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
   format.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);
   format.nVersion.nVersion = OMX_VERSION;
   format.nPortIndex = 201;
   format.eCompressionFormat = OMX_VIDEO_CodingAVC;

   printf("OMX_SetParameter for video_encode:201...\n");
   r = OMX_SetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamVideoPortFormat, &format);
   if (r != OMX_ErrorNone) {
      printf("%s:%d: OMX_SetParameter() for video_encode port 201 failed with %x!\n", __FUNCTION__, __LINE__, r);
      exit(1);
   }

   printf("encode to idle...\n");
   if (ilclient_change_component_state(video_encode, OMX_StateIdle) == -1) {
  printf("%s:%d: ilclient_change_component_state(video_encode, OMX_StateIdle) failed", __FUNCTION__, __LINE__);
   }

#if 1
   printf("enabling port buffers for 200...\n");
   if (ilclient_enable_port_buffers(video_encode, 200, NULL, NULL, NULL) != 0) {
      printf("enabling port buffers for 200 failed!\n");
      exit(1);
   }

   printf("enabling port buffers for 201...\n");
   if (ilclient_enable_port_buffers(video_encode, 201, NULL, NULL, NULL) != 0) {
      printf("enabling port buffers for 201 failed!\n");
      exit(1);
   }
#endif

   printf("encode to executing...\n");
   ilclient_change_component_state(video_encode, OMX_StateExecuting);

   outf = fopen(outputfilename, "w");
   if (outf == NULL) {
  printf("Failed to open '%s' for writing video\n", outputfilename);
  exit(1);
   }

   printf("looping for buffers...\n");
   do
   {
      buf = ilclient_get_input_buffer(video_encode, 200, 1);
      if (buf == NULL) {
         printf("Doh, no buffers for me!\n");
      } else {
//         printf("Got buffer: %p %lu %lu %lu\n", buf, buf->nAllocLen, buf->nFilledLen, buf->nOffset);

         /* fill it */
   read_raw_rgb(buf->pBuffer, &buf->nFilledLen, ++filenumber);

         if(OMX_EmptyThisBuffer(ILC_GET_HANDLE(video_encode), buf) != OMX_ErrorNone) {
            printf("Error emptying buffer!\n");
         } else {
//            printf("Buffer emptied!\n");
         }

         printf("Requesting output...\n");
         fflush(stdout);
         out = ilclient_get_output_buffer(video_encode, 201, 1);

         r = OMX_FillThisBuffer(ILC_GET_HANDLE(video_encode), out);
         if (r != OMX_ErrorNone) {
            printf("Error filling buffer: %x\n", r);
         }

   // Debug print the buffer flags
   if (out->nFlags & OMX_BUFFERFLAG_CODECCONFIG) {
     printf("Got buffer flag: OMX_BUFFERFLAG_CODECCONFIG\n");
   }
   if (out->nFlags & OMX_BUFFERFLAG_ENDOFFRAME) {
     printf("Got buffer flag: OMX_BUFFERFLAG_ENDOFFRAME\n");
   }
   if (out->nFlags & OMX_BUFFERFLAG_SYNCFRAME ) {
     printf("Got buffer flag: OMX_BUFFERFLAG_SYNCFRAME\n");
   }
   if (out->nFlags & 0x400) {
     // FIXME: what is 0x400??
     printf("Got buffer flag: 0x400 flag\n");
   }
   if (out->nFlags & ~(OMX_BUFFERFLAG_CODECCONFIG | OMX_BUFFERFLAG_ENDOFFRAME | OMX_BUFFERFLAG_SYNCFRAME | 0x400)) {
     printf("Got more buffer flag: %lx\n",
      out->nFlags & ~(OMX_BUFFERFLAG_CODECCONFIG | OMX_BUFFERFLAG_ENDOFFRAME | 0x400 | OMX_BUFFERFLAG_SYNCFRAME));
   }

         if (out != NULL) {
            printf("Got it: %p %lu\n", out, out->nFilledLen);
   if (out->nFlags & OMX_BUFFERFLAG_CODECCONFIG) {
     int i;
     for (i = 0; i < out->nFilledLen; i++)
    printf("%x ", out->pBuffer[i]);
     printf("\n");
   }

   r = fwrite(out->pBuffer, 1, out->nFilledLen, outf);
   if (r != out->nFilledLen) {
     printf("fwrite: Error emptying buffer: %d!\n", r);
   } else {
     printf("Buffer emptied!\n");
   }
   out->nFilledLen = 0;
         } else {
            printf("Not getting it :(\n");
         }

      }
   } while (filenumber < 184);

   fclose(outf);

   printf ("Teardown.\n");

   ilclient_state_transition(list, OMX_StateIdle);

   ilclient_cleanup_components(list);

   OMX_Deinit();

   ilclient_destroy(client);
   return status;
}

int main (int argc, char **argv)
{
   if (argc < 2) {
      printf("Usage: %s \n", argv[0]);
      exit(1);
   }
   bcm_host_init();
   return video_encode_test(argv[1]);
}
And the associated make file: 

OBJS=video.o
BIN=hello_video.bin
LDFLAGS+=-lilclient -lpthread

include ../Makefile.include

There really isn't any real application of the encoder yet, I think everyone is still waiting for the official camera module. From what I read the Pi is not able to cope with most USB camera module running at higher resolution (ie. 640x480) and will cause the Pi to crash. I personally tried a Microsoft Cinema HD and the video captured at 160x120 crashed roughly 10 seconds in. But regardless, this now allows the Pi to be used as a standalone transcoder to encode any data to h264 format. 



Posted by at

4 comments:

  1. JeromeFebruary 22, 2013 at 2:21 AM

    Nice job
    Difficult for me to understand clearly
    But could you just bring me light to the following question :
    Is the raspberry pi able to deal with h264 encoding from an hdmi device and then sen it by wifi?

    ReplyDelete
    Replies
    1. Ricky TeeMarch 2, 2013 at 3:04 PM

      No, I don't think so. If you browse the Raspberry Pi forum you'll find this question has been asked. The HDMI port on the Pi is output only. The only video input right now is the CSI port, which is to be used by the camera.

      That being said, the can use any source you want to do the H264 encoding (ie. local file).

      Delete
  2. RamoncioJuly 3, 2013 at 6:18 PM

    And what about DVB-T MPEG2 signals from a usb tuner?
    If realtime hardware transcoding was possible - even for PAL SD channels - many mythtv and vdr users would be really happy!!!

    ReplyDelete
  3. UnknownMay 28, 2014 at 8:53 AM

    TC358749XBG end everything possible

    ReplyDelete

Subscribe to: Post Comments (Atom)