Beta release 3.1b

Information and change log about the latest ZGameEditor release.

Moderator: Moderators

Post Reply
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

Update: I've added support for multithreaded processing inside ZGE.

See new menu option

Tools - Enable threaded processing

It is on by default.

At the moment this only affects BitmapExpression but I plan to add it in more places that is CPU intensive and can benefit from parallel execution.

A BitmapExpression is very well suited to run in parallel. So if you have a quad core CPU then it should become at least 4 times faster to recalculate a bitmap.

Note that in order to get this to work I had to change the syntax for BitmapExpressions a little bit. Previously you wrote "this.Pixel.R=this.X" etc. The new syntax is to skip the "this." part. Old projects will convert to this syntax automatically.

Currently it is only implemented in the Windows build.

Let me know if there are any problems.

http://www.zgameeditor.org/files/ZGameEditor_beta.zip
User avatar
Kjell
Posts: 1876
Joined: Sat Feb 23, 2008 11:15 pm

Post by Kjell »

Hej Ville,
VilleK wrote:I've added support for multithreaded processing inside ZGE.
Super nice. The vast majority of systems out there today will benefit from this :) Only thing is that you won't be able to use the following approach anymore.

Code: Select all

<?xml version="1.0" encoding="iso-8859-1" ?>
<ZApplication Name="App" Caption="ZGameEditor application">
  <Content>
    <Bitmap Name="Bitmap1" Width="0" Height="0" Filter="1">
      <Producers>
        <ZExpression Expression="Counter = 0;"/>
        <BitmapExpression>
          <Expression>
<![CDATA[int c = Counter++;
Pixel.R = c & 1;]]>
          </Expression>
        </BitmapExpression>
      </Producers>
    </Bitmap>
    <Variable Name="Counter" Type="1"/>
  </Content>
</ZApplication>
In non-threaded versions ( left ) you end up with vertical bars, but using multi-threading ( right ) you end up with a "random" pattern. Not much of a problem, just something to be aware of.

Image
VilleK wrote:Note that in order to get this to work I had to change the syntax for BitmapExpressions a little bit. Previously you wrote "this.Pixel.R=this.X" etc. The new syntax is to skip the "this." part.
The "this" keyword has been optional for a long time though. It was still mandatory in 1.9.6 but you can omit it in 2.0.0 .. so it / something must have changed somewhere in between those versions.

K
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Post by Rado1 »

A naive question: (probably later, if implemented) would multithreading help also with playing sounds, e.g., long sounds used for background music?

Another problem I have at the moment is loading of ogg files in runtime - large file takes a lot of time to load while animation is stopped. Would it be also possible to load files in background while other application parts are still running?
User avatar
Kjell
Posts: 1876
Joined: Sat Feb 23, 2008 11:15 pm

Post by Kjell »

Hi guys,
Rado1 wrote:Another problem I have at the moment is loading of ogg files in runtime - large file takes a lot of time to load while animation is stopped. Would it be also possible to load files in background while other application parts are still running?
Another problem that's ( somewhat ) related. When you use the File component to read a big file over a number of frames ( using a Repeat component with FileMoveData inside ), you still end up with terrible framerate due to how the file loading mechanism work internally.

For example, try reading a measly 8 bytes per frame of a 1GB file .. it'll grind your computer to a halt :cry:

Code: Select all

<?xml version="1.0" encoding="iso-8859-1" ?>
<ZApplication Name="App" Caption="ZGameEditor application" NoSound="1">
  <OnLoaded>
    <SpawnModel Model="Sprite"/>
  </OnLoaded>
  <OnUpdate>
    <FileAction File="BigFile"/>
  </OnUpdate>
  <OnRender>
    <RenderText Name="Print" X="-0.9" Scale="0.25" Align="1" RenderCharExpression="//" StretchY="2"/>
  </OnRender>
  <Content>
    <File Name="BigFile" FileName="bigfile.bin">
      <OnRead>
        <ZExpression>
          <Expression>
<![CDATA[BigFile.Position = Position;
Print.Text = intToStr(Position)+"\n\n";]]>
          </Expression>
        </ZExpression>
        <Repeat Count="8" WhileExp="//">
          <OnIteration>
            <FileMoveData Comment="Weird Property syntax :-?" Property="Byte"/>
            <ZExpression Expression="Print.Text += intToStr(Byte)+"\n";"/>
          </OnIteration>
        </Repeat>
        <ZExpression Expression="Position = BigFile.Position;"/>
      </OnRead>
    </File>
    <Variable Name="Byte" Comment="Still has to be a float :-("/>
    <Variable Name="Position" Type="1"/>
    <Model Name="Sprite" RotationVelocity="0 0 1">
      <OnRender>
        <RenderSprite/>
      </OnRender>
    </Model>
  </Content>
</ZApplication>
*The example requires a file called "bigfile.bin" to be placed in the same folder as the executable.

K
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Post by Rado1 »

What about to have an asynchronous File component? E.g. having something like File.Asynchronous property set to 1 would allow to read/write a file in a separate thread and when finished, the thread is finished as well. The OnRead or OnWrite can be executed in the same thread, or after the thread is finished in the next processing loop to avoid clashes.
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

The delay of reading from large files is because ZGE reads the whole file to memory before calling OnRead. Moving it to a thread would let other tasks move on but it would still mean calling OnRead a thousand million times for a 1gb file which won't be very fast either. I have to wonder if it makes sense to use a tool like ZGE that makes small executables when you have datafiles that are 1gb :-). We could let a thread read from a file and write to TargetArray in the background but we would need a way to let the script know how much of the file currently is available. Or maybe you only want to read the first 8 bytes? Then we could have a property that controls how much of the file to read (if zero read whole file).

OGG-delay is not from file access but from the decompression. It is probably a good idea to move this decompression to a thread, that would allow the app to start up quicker. (I assume the delay you notice is on Android, on PC the decompression is so fast anyway)

When BitmapExpressions are threaded then expressions that write to global variables need to change, just like your example Kjell. I noticed this in several of my older projects that I wrote when local variables were not supported so I used global Temp1, Temp2 etc instead.

I hope to make ZGE fully thread-safe at some point so that I can run things like Models.OnUpdate in parallell which could mean a good speed benefit.

MeshExpression is a good candidate to thread too.

I'm interested in knowing whether you get the same speed improvements on bitmapexpression that I do? Try on a 1024x1024 bitmap that takes at least 5 seconds to refresh. Then turn on threading and let me know how much faster it is. If you start the Task Manager you should notice that CPU usage is close to 100% during bitmap refresh.

Threading makes realtime RefreshContent more usable. See attached project.

Btw, there was a small bug that could make the threading lock up in the last update so download the beta again if you experience that.
Attachments
RealtimeNoise.zgeproj
(700 Bytes) Downloaded 390 times
User avatar
Kjell
Posts: 1876
Joined: Sat Feb 23, 2008 11:15 pm

Post by Kjell »

Hej Ville,
VilleK wrote:The delay of reading from large files is because ZGE reads the whole file to memory before calling OnRead.
I'm aware :wink:
VilleK wrote:I have to wonder if it makes sense to use a tool like ZGE that makes small executables when you have datafiles that are 1gb :-)
Of course not, but using a 1GB file demonstrates the problem clearly. Whereas when you're reading 8 bytes a frame from a 1MB file, you probably won't notice much .. even though you're still wasting 99%.
VilleK wrote:Or maybe you only want to read the first 8 bytes? Then we could have a property that controls how much of the file to read (if zero read whole file).
Something like that would work yes ( in combination with position* controlling Seek ). Or you cache a limited amount ( 256K or something ) .. there are a bunch of techniques that are better than simply reading the entire file.

*I guess you could store the previous value of Position internally, automatically increment it after each read and compare it to Position before each read to determine if a Seek call is required.
VilleK wrote:OGG-delay is not from file access but from the decompression. It is probably a good idea to move this decompression to a thread, that would allow the app to start up quicker.
Still think it would be good to have a option that only decodes the samples required for playback each frame ( like any other music player ).
VilleK wrote:I'm interested in knowing whether you get the same speed improvements on bitmapexpression that I do? Try on a 1024x1024 bitmap that takes at least 5 seconds to refresh.
From 6.0 down to 2.2 seconds :)

K
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

So continuing with making ZGE multithreaded there are two new features:


1. MeshExpression is now thread based so it will run 4 times faster on a quad core.

Just like the BitmapExpression, the mesh expression syntax had to be changed to drop the "this." prefix.

The syntax is converted automatically but this can cause compatibility problems in expressions like these:

float V=1.0; //local variable called "V"
this.V.X=V;

After conversion this will become
float V=1.0;
V.X=V; //error: the local variable V hides function parameter V

Fix it by renaming local variable:
float V2=1.0;
V.X=V2; //ok



2. Thread component

You can now use threading in your own scripts.

See attached example.

The benefit of using a thread is that you can launch things that take a long time and still keep your script running. For example, load a large file or do heavy computations (generating the level map). If you do this in a thread you can still use App.OnRender or models to display a progress bar.

Thread component only has a single property "Expression". The code you write in the expression is the thread. When the expression exits, then the thread quits.

You can control threading using two new script functions:
- startThread(thread, parameter). This launches a thread, using "parameter" as parameter to the thread expression. The parameter is meant to be used for giving each thread a context when you launch several instances of the same thread component. In the attached example parameter is used to allow each thread to write a separate piece of the main Data array.
- sleep(milliseconds). This pauses the current thread and allow other threads to execute. If you use threads and notice your CPU usage is too high, insert a sleep in your computation loop to lower CPU usage.


NOTE: There are very few safety nets here, you can hang/crash ZGE if you do things like:
- create hundreds of threads
- call sleep with a very high number
- access data that is no longer available

So as precaution safe your projects often :-)

Also note:
- You can make no OpenGL calls in a thread, so don't use any render components. This is a limit of the OpenGL API.
- ZGE is not yet completely thread safe (it is a work in progress), so if you notice any trouble with your threading code let me know and I'll try to fix it.
- Any threads you created will keep running until finished, even if you stop your project in the designer. Check the log window to see when a thread is started and finished.

Threading is now also supported on Android.

http://www.zgameeditor.org/files/ZGameEditor_beta.zip
Attachments
ThreadExample.zgeproj
example on how to use threads
(2.1 KiB) Downloaded 365 times
Imerion
Posts: 200
Joined: Sun Feb 09, 2014 4:42 pm

Post by Imerion »

This sounds powerful, and implemented in a smart way! Will probably wait a bit and learn more before using threading, as none of my current projects are very demanding. But it's cool to know it's possible!
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Post by Rado1 »

This multithreading seems to be an interesting thing and needs some more exploration. Can be ordinary variables used for semaphores to achieve thread synchronization?

There are several things I observed:
1. Started threads are not stopped in ZGE Preview after the Stop button is pressed.
2. If "Enable threading processing" is switched on, it consumes more CPU, but the overall FPS is better (more than 50% on 8 cores).
3. Some of my previous projects stopped working at assignments to vector variables by ?: operator ("Identifier is not an array" syntax error is reported); e.g.,

Code: Select all

vectorXY = isTrue ? vector2(1,2) : vector2(0,3);
4. Improvement: V, N, C, TexCoord, X, Y, and Pixel variables (or at least their items) in Mesh/BitmapExpression components could be auto-completed.
5. This simple project crashes when running with F9:

Code: Select all

<?xml version="1.0" encoding="iso-8859-1" ?>
<ZApplication Name="App" Caption="ZGameEditor application" ClearColor="0 0 0 1" AmbientLightColor="0 0 0 1" CameraPosition="0 -1.93 1.96" CameraRotation="-0.11 0 0" NoSound="1">
  <OnUpdate>
    <ZExpression Expression="@RefreshContent(Component: Mesh1);"/>
  </OnUpdate>
  <OnRender>
    <UseMaterial Material="Material1"/>
    <RenderMesh Mesh="Mesh1"/>
  </OnRender>
  <Content>
    <Mesh Name="Mesh1">
      <Producers>
        <MeshBox XCount="100" YCount="100" Grid2DOnly="255"/>
        <MeshExpression Expression="V.Z = noise2(V.X * 3 + App.Time, V.Y * 3);"/>
      </Producers>
    </Mesh>
    <Material Name="Material1" WireframeWidth="1" Color="1 1 0.502 1" SpecularColor="0 0 0 1" EmissionColor="0.502 0 0 1" DrawBackFace="255"/>
  </Content>
</ZApplication>
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

Thanks for testing.

Posted fixes for problem 3 and 5: http://www.zgameeditor.org/files/ZGameEditor_beta.zip
User avatar
Rado1
Posts: 775
Joined: Wed May 05, 2010 12:16 pm

Post by Rado1 »

Thanks Ville for fixing.

I observed one unexpected behavior of vectors. There's a difference between changing vector by items (X,Y,Z,R,G,B,A) and by assigning values created by vector*() functions. Let's demonstrate on example:

Code: Select all

vec3 A, B;

A = vector3(1,2,3);
B = A;
trace(intToStr(B.X) + ", " + intToStr(B.Y) + ", " + intToStr(B.Z));
// Output: 1, 2, 3

A.X = 4;
A.Y = 5;
A.Z = 6;
trace(intToStr(B.X) + ", " + intToStr(B.Y) + ", " + intToStr(B.Z));
// Output: 4, 5, 6  - unexpected

A = vector3(7,8,9);
trace(intToStr(B.X) + ", " + intToStr(B.Y) + ", " + intToStr(B.Z));
// Output: 4, 5, 6
It seems like vector*() functions create new "instances" of vectors, while changing particular items change the current vector "instance". This is not bad, but should be considered when assigning vector variables to other vector variables. At the moment it is done "by reference". Previously it was done "by value". Have you Ville changed this behavior in some of the recent versions? Are instances of vectors garbage collected when there is no reference to them (no variables are holding their values)?
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

You are right, it is supposed to be by reference. But by accident it hasn't been that way for a couple of beta releases. And yes they are automatically garbage collected when there are no reference to them.
shviller
Posts: 4
Joined: Tue Nov 10, 2009 4:47 pm

Post by shviller »

The latest beta breaks CleanseCube, the cube gets all glitchy:
Image
Maybe the problem is that the Temp1 variable ends up shared across threads? The the situation seems to be similar to this one:
Kjell wrote:In non-threaded versions ( left ) you end up with vertical bars, but using multi-threading ( right ) you end up with a "random" pattern. Not much of a problem, just something to be aware of.
Image
but instead of an explicit counter, it's just a local variable.

Also, the standalone exes seem to get created with multi-threading enabled regardless of the setting in the IDE.

Oh yeah, since I don't post on the forum pretty much at all, I should explain: I'm not a grumpy jerk that appears outta nowhere and starts complaining, Kjell made me report this! :D
User avatar
VilleK
Site Admin
Posts: 2274
Joined: Mon Jan 15, 2007 4:50 pm
Location: Stockholm, Sweden
Contact:

Post by VilleK »

Hi shviller, always nice to see more people on the forum than the usual familiar faces. Even if Kjell made you do it ;)

Yes, you are right, since many of the older projects was made before ZGE supported local variables they behave strange when multi-threaded (because they use global variables). I had changed several other projects but missed CleanseCube. I'll fix it. If you want to help, give the other demo projects a quick look too and see if something else has broken.

And yes, the multi-threaded option controls the IDE only. Runtime is always multi-threaded. I want to keep it that way if possible but if many problems are reported I may have to add an option for runtime too.
Post Reply