JFjr - the JMF/FOBS/jME video renderer

[llama]

So here's the result of my latest little attempt to render video to a jME texture. Code can be found here:
http://www.tijl.demon.nl/jme/JFjr/

----------------------------

Setup:

I'm afraid the setup is still a bit complicated, but at least it's not as bad as before. I'm assuming Eclipse is used, however, it should be possible to do with other IDEs too.

(edit: 15-05-2007, check out this post by user unixseb on how to initialize FOBS without ever even opening JMF!  http://www.jmonkeyengine.com/jmeforum/index.php?topic=2908.msg40418#msg40418)

----------------------------

First, make a project, and import the source code from the zip into it. There are several ways you can use this code. I recommend 4, as it has excellent performance, and doesn't do any allocations (except for the initial buffer of course) unlike the other methods, which prevents garbage collection.

Sun downloads: http://java.sun.com/products/java-media/jmf/
FOBS download here: http://sourceforge.net/project/showfiles.php?group_id=105646
(NOTE: for a binary build, choose FOBS4JMF. For source, look in FOBS-src)


1. Sun's build of JMF. You'll have support for a few file formats, which on linux/windows can be "accelerated". Most noteable support is for MPEG and AVI/XVID. Most noteable lack is MP3 in AVI. If you're on a Mac I would not recommend this option.

Tip for windows users: There is no need to keep the options to copy dll files to windows/system on, because we can specify their library.path ourself. (in /system they did not work on my machine)

---

2. FOBS build of JMF. Practically every format for audio and video will be supported, on mac/linux/windows. However, native files for accelerated sound output (such as DirectSound) are not included. And since JavaSound is broken more often than not with me, if you want audio, that's recommended. You'll have to copy the native files from the Sun build to the FOBS build directory, if you want accelerated sound. (or follow explanation on the FOBS page)

---

3. Manual installation of FOBS into existing Sun JMF. basically a combination of both, will always give you proper sound support. Might also be useful to those who want sound and can't figure out how to get 2 with working sound.

instructions here: http://fobs.sourceforge.net/f4jmf_first.html

---

4. FOBS with my FOBS patch. Doing this patch for FOBS is actually quite easy, considering most steps have to be done anyway. You'll need to do the same as either 3 or 2, but also download the java part of the source of FOBS (it's in the \fobs-src\src\jmf-pi\ dir in the archive). Put it in your project. Some errors will show up, related to not having JOGL. Either ignore the error or delete these files.

Now for the patch:

In
com.omnividea.media.parser.video.VideoTrack
there is a method
public void readFrame(Buffer buffer)

now replace the if block:

if(FobsConfiguration.useNativeBuffers) {
...
}

with:

Code:

if(FobsConfiguration.useNativeBuffers)
    {
    if(byteBuffer == null
    && ((byteBuffer = (ByteBuffer) FobsConfiguration.properties.get("ByteBuffer")) == null))
        {
            byteBuffer = ByteBuffer.allocateDirect(4*needDataSize);
            if(Parser.avIsBigEndian()==false) byteBuffer.order(ByteOrder.LITTLE_ENDIAN);
            FobsConfiguration.properties.put("ByteBuffer", byteBuffer);
        }
        //assert(byteBuffer!=null);
        flag = parser.getNextFrame(byteBuffer, 0, needDataSize);
       
        if(flag && data.length > 0)
        {
            byteBuffer.asIntBuffer().get(data);
        }
    }

In case you're using the latest version of FOBS you'll also need to make sure that the code block above that bit looks like this:

Code:

if  ( dataNotOkay(obj, needDataSize) ) {
    int []pixels = (int [])FobsConfiguration.properties.get("BufferedImageIntBuffer"); // <----- add this line
    if (pixels == null) {
    pixels = new int[needDataSize];
    }
    data = pixels;
        buffer.setData(data);
    } else {
        data = (int[]) obj;
    }

---------------------------

Now it's time to add the required libraries to the build path (or classpath). In case of 1, add JMF.jar (in /lib of wherever you installed (e)JMF). In case of 2, 3 and 4, you'll also need JMF.jar, from FOBS4JMF. In case of 2 and 3 (but NOT 4), also add fobs4jmf.jar. Next, set the native library locations for the .dll/.so/.jnilib files with Eclipse or -Djava.library.path . I myself use Eclipse for it (expand the jar in the build path screen), and have all files (jar, dll, etc) in the same dir. In the case of 4, even only setting it for JMF.jar seems to cover all native files in that directory.

---------------------------

Next, we'll test wether our setup is succesful so far, and we'll register our custom renderer. Both can be done with an application included with JMF, names JMStudio. However, we'll need to start it, with the compiled code of our project on the classpath. The application is JMStudio.class inside JMF.jar. With Eclipse this is easy.. create a new Run configuration (Run -> Run...) for a Java Application. On the "Main" tab make sure "include libraries when searching for a main class" is checked ->  search... and find JMStudio.

When you launch, there will be a sort of media player there. Try to play some files to see what files will work and which won't (in the case of FOBS practically all will work). If the player gets frozen trying to play different files, it's most likely the issue with JavaSound that was mentioned in the beginning.

Next, we'll register our renderer. Backup up the settings we're about to change can be done by making a backup of the jmf.properties file. How to do so is well explained in the bottom (step 5) of this page:
http://fobs.sourceforge.net/f4jmf_first.html

Our renderer (so it goes under the render tab) is called "org.llama.jmf.ByteBufferRenderer" Make sure it is at the TOP position, and don't forget to press Commit. If you had those issues with JavaSound, you can delete the JavaSound renderer. You can also completly remove sound support this way, simply remove everything on all tabs with "audio" in it (and Commit!). If you used option 3. make sure all com.omnividea.* are in the right place (near the top).

If the ByteBufferRenderer is correctly installed, you can still play a video in JMFStudio but there won't be any picture. You'll see some output on your console though.

------------------------

Wow! you made it this far.. we're about to see if it paid off. Open TestJMFVideoImage.java , edit the "String file =" line (make sure to use the right type of slashes for your platform). If you've applied the FOBS patch, also change the  "ByteBufferRenderer.useFOBSPatch = false" line to "true".

and Run it..

You should see your video playing on the texture of a quad.

------------------------

If your video does not play at all..

- Check if the ByteBufferRenderer is set at the top of the renderers in JMStudio

- Did it play in JMStudio with a "normal" renderer?
No: something is wrong, check your paths, your -Djava.library.path, etc.
No: in case you choose 1) JMStudio does not play a lot of file formats, and even less on linux/mac. Try another file or FOBS.
No: Remove that damned JavaSound renderer or even everything sound related.
No: gimme the console output, stacktraces, etc.

Yes:..
- Did it "play" (the slider moved) in JMStudio with the ByteBufferRenderer set?
No: did you see (many times) "Error rendering video to texture. No glTexSubImage2D/OpenGL 1.2 support?" in the console? Your video card or driver does currently not support doing this.. but please tell me what you use.

No: gimme the console output, stacktraces, etc.

Yes:
Do not use mipmapping beyond that of the Test.

Change the line "ByteBufferRenderer.printframes = false;" to true. Run again and gimme the console output, stacktraces, etc. and tell me if the app kept running, printing frame information on the console, or crashed.

------------------------

My video plays, but looks very weird:

- How would you describe weird? The image seems ok except for the colors?

The image seems ok except for the colors: This could be a platform/endian issue. Two things you can try:
1. in JMFVideoImage.java there is a method setSize. Change all GL12.GL_BGRA to GL11.GL_ARGB and GL12.GL_BGR to GL11.GL_RGB.
2. (try seperate) In ByteBufferRenderer.java look for the line

nativebuffer.order(ByteOrder.nativeOrder()); //TODO: test on linux.

and change the ByteOrder to ByteOrder.BIG_ENDIAN or ByteOrder.LITTLE_ENDIAN.

- Or does it seem like several interleaved images?

You are likely using 1) and encountered an input format I was not able to test (currently supported: 32, 14 and 16 bit with masked int or byteordering). Try FOBS since this seems to use always the same input format.

Finally, interlaced files are not supported by FOBS afaik, dunno about Sun. It's not common to find interlaced video, except in DV camera footage.

---------------------

(Other) known issues:

- I did not test on linux/mac. Sorry, please let me know if (and what) works.

- WMV9 is not supported (go complain to Bill).

- Audio can make the scene stutter. Even with only video framerate can jump up and down. This is unfortunatly due to multithreading. There are some wait() and sleep() in the code, try playing around with them a bit. Also I might implement double buffering, and a jME audiorenderer some time, or work on some type of locking in FOBS. Don't hold your breath though.

- Multiple video streams are untested, should work in theory.

--------------------

Further stuff:

I've tested several files, including pretty high bit rate MPEG4. FOBS, and ecps. FOBS with patch handled them with pretty good performance. Playing an average XviD AVI file (624 x 352, around 1,500 kbit video only) my CPU usage was 2-5% using FOBS with patch, and about 7-18% using JMF from Sun. Of course the real test comes when playing some high def content, such as this nasty 720P XviD file (about 56 MB):

http://www.rdkleinpowera.de/videosctrendaug1/birch_hd_XviD82.avi

This was completly unplayable (cause of frame drops) using JMStudio with either FOBS or Sun's build. However, with FOBS+patch and using update(Texture, true) it only seemed to be dropping a few frames every now and then in jME. However, the same happened with all my other media players, though general CPU usage was less. MPEG4 really seems to enjoy spiking CPU usage. Mplayer (also based on FFMPEG, like FOBS) was best from my media players. And of course it also had to do audio.

Here's another file (MPEG2) to bring my renderer (and your PC) to it's knees:
ftp://ftp.heise.de/pub/ct/spezial/shuttle.mpg
Size: 396.3 MB

Some things that could be improved: color space conversion done with shaders instead of CPU, double/tripple frame buffering, better packing of data for sending it to the GPU (tried that but still had some bug left in it that made it very slow instead).

Formats I tried with FOBS:
container:
AVI, MPG, MKV, OGM, 3GP all worked, MP4 was buggy, MOV crashed with 1 file, worked for others
video/audio codec:
MPEG1, MPEG2, XviD, DIVX, OGG, MP3, WMA

----------------

What you're still here? So.. hmm yes I know the code is a pain to look at (gotta love that use of static). It's more proof of concept than an actual lib at the moment.

License note: code is currently under LGPL (so is FOBS and so is FFMPEG), except for the example (jME license).

Let me know if this works for anyone else..

[irrisor]

Is there any chance to use it without installing JMF? E.g. can one make a jnlp that starts on machines without JMF?

[llama]

Well you can include JMF / FOBS etc. in the webstart. You can use a premade JMF properties file, but right now there's no premade FOBS crossplatform package (because the FMPEG dll/so/jnilib is so huge perhaps). And Sun, well they're Sun. "Installing" JMF (if you go option 2 or 4+2) is pretty painless, it's just extracting some libs.

The registering of the renderer I made is done manual for now, but mostly cause I didn't find any way to specify at runtime which one should be used. I'm sure that's possible though, I just didn't find it yet. But either way, it's possible to do a webstart. As for legal: Sun allows redistrubition (even in parts) of JMF (eg FOBS does this).

And:
release 0.0.2. I spelled my name wrong (true story) and some other textual changes.
release 0.0.3. Option for framesync left on in example by accident.

[darkfrog]

Great work llama.  It seems pretty complicated for people to be able to add video to their games though.  I wonder how difficult it would be to make it as easy as jME-Physics to include as an extension to jME?  If we could do that I think it would make life much easier for people interested in using JMF.  Also, if I wanted to make a WebStart version that had this all included, aside from the video file and all the other stuff that is already required for jME how much extra download is required for all of this?

darkfrog

[llama]

Yes.. however, I'd like it if people test it like this first. Especially on linux and mac. And also test some different input sources like webcams.

It's only the setup that's complicated for now, webstart will be easier, once I find a consistent cross platform way to do setup. (problem is I lack platforms). If you want to help me with that, you can send me the jmf.properties in the JMF directory after registration (mention wether you used FOBS or Suns install, gimme both if you feel really helpfull), so I can see what renderers are available on other platforms.

You can already make a webstart yourself, just do the registration and stuff,and jmf.jar is just a lib like any other as long as you include the generatated jmf.properties. Again, problem is it won't work on all platforms.

Now don't let the big document scare you.  If you want to test this, it's not rocket science. Copy some sources, include some libs, and then point-click,enter-a-class-name, press commit and you have it working.

I mainly did this cause people wanted to start developing with video/jme. Well now you can.. Now I'll need your help if we'll want to turn this into a crossplatform library. (I'll also try to get my patch in FOBS). The only longer term problem I can see is that including FOBS (if you're gonna use that) in webstart will add quite a bit of size (6,45 MB .dll). However, you should be able to do a recompilation for that, to only include the codecs you need for your game. JMF itself can be cut down to under 1 MB.

That, and for audio we'll need a jME plugin to have 3D sound and such (and JavaSound is the only other real crossplatform solution, which is unreliable). Also, I'm looking at a way to cache (small, I assume) video files completly in RAM.

[darkfrog]

Okay, I'll try to get some tests done on Windows and Linux with this in the near future and send you the info.

That's a pretty large addition to a game, but if you're sticking a video into the game anyway 6 meg is probably nothing in comparison with the media you're adding.

Can we not use JMF to output the sound directly into jME's sound system?

darkfrog

[llama]

Yes that's what I'm suggesting. Not sure how well suited jME's own sound system is at the moment for handling/streaming different sampling rates and such (it mostly seems to be meant for directly decoding ogg/wav), though I haven't taken a closer look yet. Sound also seems a bit more complex with JMF. Advantage would be, support for much more audio (only) formats in jME too.

You could however, always split the video and audio into different files, an use jME's existing OGG support if you really had to. I also think there are some use case where you don't sound anyway (like an animated skybox, or an animated planet with cloud layers and such).

[darkfrog]

The big problem with that is the probability of them getting out of sync is drastically increased. :-p

darkfrog

[Søren]

Hi! Great work! I've started integrating your code into my project. I have some trouble though. When I slam my project into a single jar I can no longer create a player for the avi I'm using. I think the problem is creating the player from a relative filename. Since TestJMFVideoImage.java uses an absolute path there's no help there.

Does someone know how to create a player from a relative path? (preferably through the usual class.getClassLoader stuff)

[llama]

I assume you're using FOBS. FOBS uses the ffmpeg code for reading files, in a way that's a good thing (ffmpeg is rather optimized for this). The bad side is, I don't think you'll be able to read from inside a jar file like this. You could consider extracting the video files to the temp dir from the jar.

[Søren]

okay. Thanks for the quick reply. I'll look into extracting from the jar then. Keep up the good work!

[kidneybean]

This is cool, any new thoughts on it ??

[llama]

Well, it's ready for use. If linux(/mac) people tell me what their result are, and send me their properties files I can try and make cross platform use a little easier.

[Søren]

Hi! I have a memory problem using your code. I have trouble deallocating the memory used for the videos in my game so when I load up the fifth cutscene I run out of memory. Does anyone know how to deallocate?

[Søren]

Here is my current code:

Code:

Player p = image.getJmfplayer();
p.close();
p.removeControllerListener(this);
p.deallocate();

image is an object of type JMFVideoImage.

[Tags:]

jMonkey Forum > Community Activities > User Code & Projects (Moderators: Outrunner, erlend_sh, Momoko_Fan, Vardamir, .:emp...imm0|82:.) > Topic: JFjr - the JMF/FOBS/jME video renderer