[Android] How to use MediaExtractor and MediaCodec? - Video part

MediaExtractor is used to separate the video data and audio data from media sources.
It can support HTTP streaming or local file.
After separating the video and audio, MediaCodec will take care of the decoding jobs.

Let's check the video part first.
For video, we need to render it after decoding.
Therefore we have to incorporate SurfaceView in our layout.

private MediaExtractor extractorVideo;
private MediaCodec decoderVideo;

extractorVideo = new MediaExtractor();
extractorVideo.setDataSource("myTest.mp4"); 

for (int i = 0; i < extractorVideo.getTrackCount(); i++) {
 MediaFormat format = extractorVideo.getTrackFormat(i);
 String mime = format.getString(MediaFormat.KEY_MIME);
 Log.d(TAG, "mime=>"+mime);
 if (mime.startsWith("video/")) {
  videoTrack = i;  
  extractorVideo.selectTrack(videoTrack);
  decoderVideo = MediaCodec.createDecoderByType(mime);
  decoderVideo.configure(format, surface, null, 0);
  break;
 }
}

if (videoTrack >=0) {
 if(decoderVideo == null)
 {
  Log.e(TAG, "Can't find video info!");
  return;
 }
 else
  decoderVideo.start();
}

extractorVideo  will find out the video track according to the MIME information.
After that, we can pass the video format and surface to render on to decoderVideo using decoderVideo.configure().
decoderVideo will take care of decoding and rendering jobs almost automatically.

Below is the decoding part:

ByteBuffer[] inputBuffersVideo=null;
ByteBuffer[] outputBuffersVideo=null;
BufferInfo infoVideo=null;

if (videoTrack >=0)
{
 inputBuffersVideo = decoderVideo.getInputBuffers();
 outputBuffersVideo = decoderVideo.getOutputBuffers();
 infoVideo = new BufferInfo();
}

boolean isEOS = false;
long startMs = System.currentTimeMillis();

while (!Thread.interrupted()) {
 if (videoTrack >=0)
 {     
  if (!isEOS) {      
   int inIndex=-1;
   try {
    inIndex = decoderVideo.dequeueInputBuffer(10000);
   } catch (Exception e) {
    e.printStackTrace();    
   }

   if (inIndex >= 0) {
    ByteBuffer buffer = inputBuffersVideo[inIndex];
    int sampleSize = extractorVideo.readSampleData(buffer, 0);
    if (sampleSize < 0) {
     // We shouldn't stop the playback at this point, just pass the EOS
     // flag to decoder, we will get it again from the dequeueOutputBuffer     
     decoderVideo.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
     buffer.clear();
     isEOS = true;
    } else {     
     long current = System.currentTimeMillis();
     decoderVideo.queueInputBuffer(inIndex, 0, sampleSize, extractorVideo.getSampleTime(), 0);     
     buffer.clear();
     extractorVideo.advance();
    }
   }
  }
  int outIndex=-1;
  try {
   outIndex = decoderVideo.dequeueOutputBuffer(infoVideo,10000);
  } catch (Exception e) {
   e.printStackTrace();   
  }
  switch (outIndex) {
  case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
   Log.d(TAG, "INFO_OUTPUT_BUFFERS_CHANGED");
   outputBuffersVideo = decoderVideo.getOutputBuffers();
   break;
  case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
   Log.d(TAG, "New format " + decoderVideo.getOutputFormat());
   break;
  case MediaCodec.INFO_TRY_AGAIN_LATER:
   Log.d(TAG, "dequeueOutputBuffer timed out!");
   break;
  default:
   if(outIndex >=0)
   {
    ByteBuffer buffer = outputBuffersVideo[outIndex];    
    buffer.clear();
    decoderVideo.releaseOutputBuffer(outIndex, true);
    // We use a very simple clock to keep the video FPS, or the video
    // playback will be too fast
    while (infoVideo.presentationTimeUs / 1000 > (System.currentTimeMillis() - startMs)) {
     try {      
      sleep(10);
     } catch (InterruptedException e) {
      e.printStackTrace();
      Thread.currentThread().interrupt();
      break;
     }
    }
   }
   break;
  }

  // All decoded frames have been rendered, we can stop playing now
  if ((infoVideo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
   Log.d(TAG, "OutputBuffer BUFFER_FLAG_END_OF_STREAM");
   break;
  }
 }
}
if (videoTrack >=0)
{
 decoderVideo.stop();
 decoderVideo.release();
}

extractorVideo.release();

We can use decoderVideo.dequeueInputBuffer(10000) to get a input buffer first.
Then use extractorVideo.readSampleData(buffer, 0) to feed the video data to buffer from media source.
For decoding, we then use decoderVideo.queueInputBuffer() to queue the data.
decoderVideo will start to decode after that.

We use decoderVideo.dequeueOutputBuffer(infoVideo,10000) to get the result.
If the return value is not -1, it means that decoderVideo has decoded the video data successfully.

To render the decoded data to surface, we use  decoderVideo.releaseOutputBuffer(outIndex, true). The second parameter true means that we need to render.

We have finished the decode and render jobs.
Forward to the next video data again using extractorVideo.advance().

[Corona SDK] How to use graphics.newImageSheet ?

When we want to draw a picture, we can use display.newImage() to load the picture file.
It is quite simple and easy.
However, if we want to draw a lot of pictures, use display.newImage() to load the picture one by one would be tedious.

If these pictures are related, and even they have the same sizes,
to use graphics.newImageSheet()  is a better option.

Assume that we want to draw 52 pictures of poker cards.
Let's us put all the 52 pictures in one picture name "card.png", shown as below:

card.png

Then we can show all the 52 pictures with following codes：

display.setDefault( "background", 0.2, 0.5, 0.2 )

local frames_width = 73
local frames_height = 98
local total_frames = 52
local image_name = "card.png"
local scaleRatio = 0.6
local options =
{
 width = frames_width,
 height = frames_height,
 numFrames = total_frames,
 --sheetContentWidth = 754,  -- width of original 1x size of entire sheet
    --sheetContentHeight = 311   -- height of original 1x size of entire sheet
}

local sheet = graphics.newImageSheet( image_name, options )
cardGroup = {}
for i=0,3,1 do
 for j=1,13,1 do
  cardGroup[i*13+j] = display.newImage( sheet,i*13+j)
  cardGroup[i*13+j]:scale( scaleRatio, scaleRatio )
 end
end
--------------------------------------------------------------------------------
for i=0,3,1 do
 for j=1,7,1 do 
 cardGroup[i*13+j]:translate( (frames_width*scaleRatio+2)*(j-1)+(frames_width*scaleRatio+2)/2,(frames_height*scaleRatio+2)/2+(i*2)*(frames_height*scaleRatio+2)) 
 end 
 for j=1,6,1 do 
 cardGroup[i*13+j+7]:translate( (frames_width*scaleRatio+2)*(j-1)+(frames_width*scaleRatio+2)/2,(frames_height*scaleRatio+2)/2+(i*2+1)*(frames_height*scaleRatio+2)) 
 end 
end

For graphics.newImageSheet( image_name, options ), we only need to pass in the file name and options.
The parameters sheetContentWidth and sheetContentHeight in options are the size of the picture (e.g card.png) . We can leave them without specified if we don't want to scale it.
The parameters width and height are the size of thumbnails (e.g. the size of one card).
The parameter numFrames is the number of thumbnails (e.g. 52 in our case).

When we want to display any card, we can pass in the image sheet we just built and the card index to display.newImage().
The result of the codes:

View As 1080x920

What if the size of thumbnails are not the same?
In such case, we have to specify the size for each thumbnails in the options:

local options =
{
    --array of tables representing each frame (required)
    frames =
    {
        -- FRAME 1:
        {
            --all parameters below are required for each frame
            x = 2,
            y = 70,
            width = 50,
            height = 50
        },

        -- FRAME 2:
        {
            x = 2,
            y = 242,
            width = 50,
            height = 52
        },

        -- FRAME 3 and so on...
    },

}

[Corona SDK] What is the difference between "else if" and "elseif"?

The basic usage of if is:

local i = 0
if(i==1) then
 print(1)  --> will not print
end

Note that there should have a then after if.
Actually, most languages also have similar syntax.
However, they can accept that we omit the then.
But for Lua, it is a must.

How about adding a else ?

local i = 0
if(i==1) then
 print(1)  --> will not print
else
 print(0)  --> will print
end

Nested if in 2 layers:

local i = 0
if(i==1) then
 print(1)  --> will not print
elseif (i==0) then
 print(0)  --> will print
else
 print(123) --> will not print
end

Can elseif be written as else if ?

local i = 0
if(i==1) then
 print(1)  --> will not print
else if (i==0) then
 print(0)  --> will print
else
 print(123) --> will not print
end
end

It seems that the results are the same, except for that one extra end is added.
Why?

Nested for more layers:

local i = 0
if(i==1) then
 print(1)  --> will not print
elseif (i==2) then
 print(2)  --> will not print
elseif (i==0) then
 print(0)--> will print
else
 print(123) --> will not print
end

Again, change elseif to be else if :

local i = 0
if(i==1) then
 print(1)  --> will not print
else if (i==2) then
 print(2)  --> will not print
else if (i==0) then
 print(0)--> will print  
else
 print(123) --> will not print
end
end
end

Well, 3 end are added?

It is because that for else, we will go into its body if previous conditions are all false.
For elseif, it means that we will need to check the condition after it to decide if we should go into its body.
For else if, it means that we have entered the body of else, then we add one condition if.

We can add two lines in the codes (line 5 and line 9) to make it clear:

local i = 0
if(i==1) then
 print(1)  --> will not print
else 
 print("enter 1") --> will print 
 if (i==2) then
 print(2)  --> will not print
else 
 print("enter 2") --> will print 
 if (i==0) then
 print(0)--> will print  
else
 print(123) --> will not print
end
end
end

If it is not clear enough, we can re-layout the codes without modify the content:

local i = 0
if(i==1) then
 print(1)  --> will not print
else 
 print("enter 1") --> will print 
 if (i==2) then
  print(2)  --> will not print
 else 
  print("enter 2") --> will print 
  if (i==0) then
   print(0)--> will print  
  else
   print(123) --> will not print
  end 
 end
end

So, the standard usage is:

if(condition 1) then
 --do some things
elseif (condition 2) then
 --do some things
 .
 .
elseif (condition n) then
 --do some things
else
 --do some things
end

[Corona SDK] Powerful Table

In [Corona SDK] How to use Table?, we learn how to use Table.
Actually, we always encounter Table in Lua.
For example, we have mentioned that we can use t.name for t["name"].
It looks similar with those Lua APIs, right?

Yes, those functions are grouped by Table.
For example, for display related APIs, there exists a table named display.
In the table, there have different index keys which mapped to the correspondent functions.
So, the following two methods mean the same thing.

display.newText( "ee", display.contentCenterX, 80, native.systemFont, 20 )
display["newText"]( "ee", display.contentCenterX, 80, native.systemFont, 20 )

The Table is really powerful....

If we have many functions with related purposes, we can also group them with table.
It will look like the method of object in other languages.

If the function just needs to pass in some parameters.
Use t.name is fine.

What is we need to operate on the object itself?
For example, we draw a rectangle and rotate it.

local rec = display.newRect( 100, 100, 50, 50 )
rec.rotate(rec,45)

We need to pass in the object itself as parameter.
Lua provide a simple way for this:

local rec = display.newRect( 100, 100, 50, 50 )
rec:rotate(45)

That is, use colon ":" for dot ".". 
By doing so, we don't need to pass in the object itself.
It is merely a shortcut.

[Corona SDK] Expressions

Arithmatic Operators
Most basic arithmetic operations are supported: +(addition), -(subtraction), *(multiplication), /(division), %(modulo), ^(exponentiation)
Take a look at the example directly：

print(17+2)      --> 19
print(-17-2)     --> -19
print(17*5)      --> 85
print(17/3)      --> 5.6666666666667
print(17%5)      --> 2
print(17^2)      --> 289
print(17^(-0.5)) --> 0.24253562503633, equal to square root

Relational Operators
There have few types:
==     ~=      <      <=      >       >=
== is used to check if two variables or objects are the same or not.
It will check their types first. If their types are different, it will return false directly.
If their types are the same, then it will compare their values.
That is, the string and number conversion mentioned in [Corona SDK] Basic types are not applied here.
For example, 123 == "123" will return false.

For table and function, the comparison is done by reference, not by value.
As example shown below, the result will be false, even these two tables have the same contents.

t1 = {my,"123"}
t2 = {my,"123"}
print(t1==t2)   --> false

~= is the negation of ==.

Logical Operators
or: Return the first argument if it is not false and not nil. Otherwise, it will return the second argument.
and: Return the first argument if it is false or nil. Otherwise, it will return the second argument.
not: Return true or false, no matter what the original type is.

print(1 or 2)            --> 1
print(nil or true)       --> true
print(false or nil)      --> nil
print(3 and 4)           --> 4
print(not nil)           --> true
print(nil and "test")    --> nil
print(false and nil)     --> false
print(1 or 2 and 3)      --> 1
print(nil or 2 and 3)    --> 3
print(nil and 2 or 3)    --> 3

Concatenation
Use ".." to concatenate two strings.
The string and number conversion mentioned in [Corona SDK] Basic types are applied here.

print("a" .. "b")       --> ab
print("a" .. 2)         --> a2
print(1 .. 2)           --> 12

Length Operator
Add # in front of string or table variable can get the length information.

a = "hello"
print(#a)       --> 5
c = {}
c[1] = "33"
c["a"] = "55"
c[2] = "33"
print(#c)       --> 2
d = {}
d[0] = "33"
d["a"] = "55"
d[1] = "33"
d[3] = "33"
print(#d)       --> 1
e = {}
e[2] = "33"
e["a"] = "55"
e[3] = "33"
print(#e)       --> 0

For table type, the length will be the last number index in continuous sequence.
If the value of index 1 is nil, the length will be 0. 
For example,  c[1],c[2] have continuous index and c[1] is not nil. The length is the last index 2.
For d[0],d[1],d[3], the index sequence is not continuous. The length will be the last index in continuous sequence, which is 1. 
For e[2],e[3], the index sequence is continuous. However, e[1] is nil. The length will be 0.

[Corona SDK] How to use Table?

The table in Lua is very special and important.
It can accept any types of data, except for nil.
We can also put different types of data in the same table.

We can use "{}" to create a new table.
Its index can be number or other types, except for nil.

display.setStatusBar( display.HiddenStatusBar )
t = {} --create a table
t = {he = "today"} --create a table with single property "he"
t[1] = 123
t[5] = "this is 5"
t[true] = 789
t["my"] = 456
t["you"] = "this is you"
display.newText( t[1], display.contentCenterX, 80, native.systemFont, 20 )
display.newText( t[5], display.contentCenterX, 100, native.systemFont, 20 )
display.newText( t["my"], display.contentCenterX, 120, native.systemFont, 20 )
display.newText( t["he"], display.contentCenterX, 140, native.systemFont, 20 )
display.newText( t.you, display.contentCenterX, 160, native.systemFont, 20 )
if(t[true] == 789) then
 display.newText( "this is 789", display.contentCenterX, 180, native.systemFont, 20 )
else
 display.newText( "this is NOT 789", display.contentCenterX, 180, native.systemFont, 20 )
end

To access the table, we can use t[].
There has a special usage when the index is in string type.
That is, t["name"] or t.name are all allowed, like line 13 t.you in above codes.
However, if the "name" of t["name"] has number prefix, we cannot access by t.name then.
For example, t["5r"] cannot be accessed by t.5r.
Below is the result of above demo codes:

[Corona SDK] Basic types

Corona SDK use a so called Lua language.
For other languages, we will need to use some key word, such as int,String and boolean to define the variables.
Once they are declared, we cannot change their types.

For Lua, the types of variables are defined automatically and dynamically.
It will assign the type according to the parameter you provide.
For example:
t = 1  --assign t as number type
t = "hello" --assign t as string type

Can we set t = 1 and then set t = "hello"?
That is, can we change its type for the same variable?
Yes, it is allowed.

There have few basic types:
nil: When you declare the variable without assign any value to it, it will be nil type. It is similar with null in other languages.
For example, local t or local t  = nil both are nil.
If we access the variable in nil type, we will get the error message "got nil".

local t
display.newText( t, display.contentCenterX, display.contentWidth / 4, native.systemFont, 40 )

boolean: It is false or true. We will use it in conditional expression.
If the parameter is false or nil, the result is false.
Otherwise, the result will be true.

if(true) then --will display
display.newText( "hello true", display.contentCenterX, 20, native.systemFont, 20 )
end
if(false) then --will NOT display
display.newText( "hello false", display.contentCenterX, 40, native.systemFont, 20 )
end
if(nil) then --will NOT display
display.newText( "hello nil", display.contentCenterX, 60, native.systemFont, 20 )
end
if(1) then --will display
display.newText( "hello 1", display.contentCenterX, 80, native.systemFont, 20 )
end
if(0) then --will display
display.newText( "hello 0", display.contentCenterX, 100, native.systemFont, 20 )
end

In the above example, if(0) will be taken as true.
number: To express real numbers (double-precision floating-point) ，
The example shown below are all allowed:
12   12.0   12.1895   12189.5e-3   0.121895E2    0xA3   0Xb22f
e or E is for decimal exponent，0x or 0X is for hexadecimal.
string: To present characters array, embedded with zero at the end.
function:
table: It is Lua fundamental data structure, check [Corona SDK] How to use Table?

Another special usage is that number and string can converse mutually.
If we do string operation on a number variable, then it will become string type first.
For example, display.newText() will accept string.
However, we can feed it with number variable.

t = 123
display.newText( t, display.contentCenterX, 100, native.systemFont, 20 )

On other hand, if we do arithmetic operations on a  string variable, then it will become number type first, as example shown below:

t = "456.2"
t = t + 1
display.newText( t, display.contentCenterX, 100, native.systemFont, 20 )

We will get "457.2" for above example.
For the variable t, we can use other number expression as along as it can be converted to number type. 
For example t = "0x3a"  or t = "1E3" are all acceptable.