[OPabst]

Hello,

based on the situation, that several FS user reports problems regarding OOM (out of memory) or CTD (crash to Desktop) over the last time and as the "3GB-Patch" which tweaks the FS.exe to be able to handle more memory is more and more used, I have take some time to look deeper into the problem.

I noticed, that while my Airplane was parked in Ibiza (Simwings Scenery) I got a OOM on my PC after some time I was not infront of the PC. So nothing was done, expect the AI Traffic around.
So my first idee was, that a AI Airplane with buggy textures may be the reason, as the CTD was based on the FE.DLL

But I don't see a problem an the aircraft, so I made some more test, by deactivating the AI traffic, chaning my plane to default Cessna and set clear weather. When I monitor the memory in the Process page of the Windows taskmanager, I can see that the memory used by FS is growing from time to time, which can not be normal.

So have have setup the following test scenario:

- removed all DLLs not default from the Modules subdir in FS (back to basic FS 9.1)
- removed all non default scenery entries in the Scenery.cfg (back to default installation)
- Switch of AI Traffic, set to fix clear weather, disable dynamic scenery
- Place the default Cessna on the Apron of Ibiza, outsideview, so that I have the plane and the terminal in view
- Save this as defaultflight, so that the FS starts allways with this settings.

Then I open the Taskmanager in the process page and ordered the list to virtual Memory usage (FS is now allways top in the list when loaded)

Now I start the FS and don't touch it anymore, only monitoring the memory usage:

1 - Default: Memory is allocated after complete start, within the first Minute the memory is growing around 20 to 40 MB but then it is stayable (wait 1 hour, no more memory needed)

In the next step, I reactivated all DLL's in the modules folder:

2 - With all DLL's, more memory is needed, but after one Minute, the memory is stayable too

Then I bring back the UTE and freeware Europemesh into the scenery.cfg

3 - With mesh and UTE, more memory needed, but also stayable after a 1-2 minutes.

In the next step, I add the two simwings Ibiza scenery entry (Photo and scenery) to the scenery.cfg

4 - with simwings scenery, the memory is growing after some time (1-2 minutes interval) by 20-30 MB, it is NOT stayable over longer time and is not reducing anymore.

Then I made some test to deactivate only the Photo part of the scenery -> memory grows
When activating only the photo part, memory was stayable
Then I deactivate single BGL and I could identify the LC_048_018.BGL as the problem generator.

5 - When I deactivate the LC_048_018.bgl, memory is stayable

In the next step, I have checked if maybe the combination with UTE and this file is the problem, so I deactivate all UTE entries in the scenery.cfg

6 - UTE disabled, LC file enabled -> memory is still growing, when the LC file is active.

Then I moved this file to a new IBIZA_LC\SCENERY directory (without a TEXTURE Subdir) and add this new Directory to the Scenery.cfg

7 - UTE active, LC file in a seperate LC Directory (-entry) -> NOW the memory is STAYABLE, not change over the time.

What does that mean:

It looks like, that FS has a problem in memory handling, when a Landclass file is placed within a Scenery directory mixed up with other BGL's, maybe because of the know issue regarding the TEXTURE directoy. Before, I only know, that a empty TEXTURE subdir in a active scenery entry generates problems with memory allocations. Now I think, it is not the empty TEXTURE subdir, it is the exiting TEXTURE directory when LC files are in the SCENERY directory.
So, Landclass files must be allways in a seperate entry, where only a SCENERY directory is included.

To double check this, I generate a emty TEXTURE subdir in the LC entry:

8 - With empty TEXTURE subdir beside the LC file -> Memory grows as expected.

Now I copyed all BMPs of the scenery to the empty TEXTURE directory of the LC entry.

9 - Even with BMP in the Texture directory, the memory grows !

Last test, I move a 3D objekt BGL file to the scenery directory beside the LC file

10 - also with the BGL file in it, the memory grows over the time.


Result of the Tests:

Never place a LC file in a SCENERY directory, where a TEXTURE Sub dir is existing, LC File have to be in there own entry without a TEXTURE Subdir.

I expect, that serveral Addons (Freeware but also payware) have LC file in there normal Entry, when a user is flying long times over such areas, the FS will eat the memory and the OOM is "normal" at the end.

Now, I check all my installed Sceneries regaring this problem, here the first results:

1.) Ibiza of Simwings Balear and Gibratar (as explained above):
File LC_048_018.BGL and WC_048_018.BGL move to a seperate LEIB_2004_LC subdirentry solved it.

2.) FranceVFR Mediterranee Basepack
Two files generate memory usage: wc_fmbp_e.bgl and wc_fmbp_w.bgl
Moving this two files to a seperate LC Scenery directory entry will solve the problem for me.
Maybe other products of this line are effected too (which I don't have)

Some more Tests:

I made some other tests around FranceVFR in LFML, which are very interesting and shows, how critical
a wrong installation can be.

First I disable UTE, any addon Scenery in LFML, save the startflight with Cessna in SLEW Mode and outsideview.

I also start Filemon to see, what happens on the disk. One point here shows, that the FS will read all ground/LC BMP's

every 2 Minutes. Based on the long list of BMP in FranceVFR, the read needs long and so it looks like the FS will read

all the time the files. Before he is at the end, the 2 minutes are over and he starts again.
So, if you wonder, why the harddisk don't stop, now you know.

Then I made 2 Test with the same rotation, after each Jump I wait some Minutes to see, how the memory grows or not.

1.) both files WC*.BGL are in seperate Scenery entry.

Start of FS at LFML: Memory usage 209 MB growing to 214 MB after 2 Minutes - then stayable
Jump via Menu to EDDN (default): Memory 243 MB gowing to 283 MB after 2 Minutes - then stayable
Jump to LEPA (Simwings): Memory 250 MB growing to 286 MB after 2 Minutes - then stayable
Jump back to LFML: Memory 262 MB reduceing to 221 MB after 2 Minutes - seams stayable

This all looks ok, different places needs different Memory, but when you change position, memory is getting free
and at the end, the Memory is more or less the same as at start time.


2.) both files WC*.BGL moved to the Scenery Directory of FranceVFR (as in default installation)
all the rest the same as in test 1)


Start of FS at LFML: Memory usage 398 MB growing to MB after 2 Minutes 408MB - after 10 Minutes 650 MB still growing
Jump via Menu to EDDN (default): Memory 681 MB gowing to 722 MB after 2 Minutes - then stayable
Jump to LEPA (Simwings): Memory 690 MB growing to 726 MB after 2 Minutes - then stayable
Jump back to LFML: Memory 860 MB growing to 950 MB after 5 Minutes - still growing (after 15 Minutes at 1.5GB)


What we see is the typical memory leak. The once allocated memory is not given free anymore. When you fly some minutes

in this area, all memory is eaten and you get a OOM, latest at the destination, when a complex Scenery is loaded.

The only difference between Test 1 and 2 is the location of the two Landclass files (here they are Waterclasses).
When this files are in the normal Scenerydirectory, you can see in filemon, that allways when the memory is growing,
this files generate a search for a Bitmap first in the parallel TEXTURE dir, which fails. Then the FS search this BMP

in FS9\scenery\world\texture, where the file could found, but the memory is gone.

When this files are in a seperate SCENERY directory, where no TEXTURE Dir is existing, the FS maybe notice this on

startup time and so the BMP is directly search in FS9\Scenery\World\Texture without a error before. This keeps the

memory low.


Here the result of testing all AES related Addons in the last 2 days:
It's only a overview of my personal results, maybe will changed when other solutions found.

1.) Simwings/Aerosoft Balear Ibiza

Files to Move to LC Entry: LC_048_018.bgl,WC_048_018.BGL


2.) FranceVFR MediterraneeBP

Files to Move to LC Entry: wc_fmbp_e.bl,wc_fmbp_w.bgl


3.) GMMN - Aerosoft Casalanca:

Files to Move in seperate LC Entry: airportedges.bgl,casablancaeast.bgl,casablancaedges.bgl

4.) FSQuality Croatia & Slovenia

Disable the two LC Entrys (Croatia Landclass NonCostal and Costal), no other solution yet.

5.) FISD EFRO
Small problem (around 1 MB every two minutes lost), file not yet identified.

6.) Greece Airport Team LGTS

File to Move in LC Entry: LD_5417.BGL

7.) Sochi 2007

File not identified yet, usage small (2-3 Mb/2 Minutes)

8.) VOZ 7 Bribane Area

VOZ Scenery
VOZ Matts custom landclasses
VOZ Matts dry custom landclasses

Massiv memory usage: 30 MB over 2 Minutes

Files are all over, so we need to wait for a VOZ solution.


As you see, not every Scenery has problems here, but I only check some of your Addons, so maybe other are effected too. As any fly over such an area can effect the memory for the complete session, it could be, that here is the reason for most crashed at the end of a longer fight. But maybe to early to say that.





Other Issues seen:

When you have WOAI Traffic installed, it looks like that some of the aircraft use a Effect file called fx_strobex.fx, which is not present in the FS9 effects Subdir, so that the FS try to find this file 100 times a second.
Solution: copy the file ..\EFFECTS\fx_strobe.fx to ..\EFFECTS\fx_strobex.fx




will be continued ....

[zneih]

Wow, great work!!

One quick question though: I'm looking through all my addon scenery folders right now, sometimes there are "LC" and "landclass" files. Is there a difference between those two and should I move both LC and landclass files to a separate scenery folder?

Best regards, Manu

[OPabst]

Hi,

when a addon has allready a seperate entry in the Scenery library (scenery.cfg) which is mostly call LC or Landclasses, then this product is not critical (I don't see any, where the developer has done a LC entry with a memory problem). Ibiza and FranceVFR has non of this additational entries.

But not all addons have LC files in there scenery and if so, they are maybe not located by the filename. Thats way I will check all airports,only when I see the memory growing i try to find the related files.

[MeatWater]

Great find! Is it possible to generate a tool that identifies landclass bgl files? These don't contain any calls for 3D-objects (right?), so it should be possible.

This should also apply to FSX, if I got that right that part of memory handling obviously didn't change.

[OPabst]

Great find! Is it possible to generate a tool that identifies landclass bgl files? These don't contain any calls for 3D-objects (right?), so it should be possible.

This should also apply to FSX, if I got that right that part of memory handling obviously didn't change.

Hi,

thats my goal, but first I want to get an overview, which addons are effected. Maybe then we know, that the OOM problem at all is fixable (maybe with a tool) and the 3GB patch lost his need (and all the problems behind that).

[tetiaroa]

Hi Oliver,


WOOOW!! What an incredibly huge effort, you put into this just to help us out! THANK YOU !!!!

That´s a fantastic finding! Hope, we´ll really be able to solve this OOM-problem without /3Gb-patch in the near future!

Do you think, it´ll be possible to create such an "OOM-Fix"-tool?

Regards,
Tetiaroa!

[KingCat]

Do these issues only apply to when using the 3GB patch, or also the "regular" use of FS9? Personally I don't use the 3GB patch and I haven't experienced an OOM message yet. I don't want to change any files if I don't have to.


Jon

[OPabst]

Do these issues only apply to when using the 3GB patch, or also the "regular" use of FS9? Personally I don't use the 3GB patch and I haven't experienced an OOM message yet. I don't want to change any files if I don't have to.


Jon

This is a basic "feature" of the FS and has nothing todo with the 3GB patch. The 3GB Patch only maybe let popup the problem a littlebit later, but it did not generate or solve it.

The problem depents on the addons you use, so if you have no OOM, ignore it yet. This thread is only a loud thinking and no solution for the problem. Maybe we find one sometimes to fix it generally, but at the moment it is only to let you know that if such problems (like OOM,CTD's) happend after long flight in a approach to a addon Airport, must NOT be a problem of the addon itself, it's more a result of the memory leak of other addons overflying before.

[mackintosh]

Do these issues only apply to when using the 3GB patch, or also the "regular" use of FS9? Personally I don't use the 3GB patch and I haven't experienced an OOM message yet. I don't want to change any files if I don't have to.


Jon

Jon, they most certainly do happen without using the 3GB patch. My FS9 installation is so crippled that rarely can I make a flight between two large airports that does NOT end in an OOM CTD (most recently I had an OOM having just taken off from Gary's Gatwick, maybe 20 minutes into the flight, the fastest CTD yet). I found that OOMs are more likely to happen the larger the memory of your gfx card is (I'm getting way more OOMs with my 9800GX2 setup than I used to get with a single crappy 256Mb AGP card, something about memory allocation sharing between the gfx card and system memory). I'm pretty sure Oliver is onto something here and since a lot of us have a ton of add-ons, a utility that would identify and move landclass files would definitely be welcome.

On a side-note, the now obsolete FligtSim Manager did have a facility that would identify landclass files, but it was mostly trial and error, it didn't catch them all and sometimes would flag the wrong files.

[VulcanB2]

Hi,

I removed all add-on landclass from the few add-ons I have installed that had custom landclass - NO CTD!!!

THANKYOU!!!

Best regards,
Robin.

[AaronMyers]

What about something like KORD from FSDT? I notice their Zurich does have a separate entry but KORD does not. How would one figure out what needs to be moved out and what does not?

[OPabst]

What about something like KORD from FSDT? I notice their Zurich does have a separate entry but KORD does not. How would one figure out what needs to be moved out and what does not?

KORD seams to be ok, no Landclass included.

The easiest Test to check, if a area is critical, you can do the test I have done:

1) download and unzip Filemon into a directory and make a link on the desktop for easy start.
2) Save a default flight in FS with Cessna, clear weather, AI traffic off
3) Start filemon, go to Volumes Menu and select the drive of your FS
4) Apply a filter like this:
include = FILE NOT FOUND
exclude = .fx;.CAB;.agn;uteur
(the uteur is to exclude UTE, so if you have other UT maybe change or add that)

Now, jump via Goto Menu to a Airport, where you want to check the area, then check if you see
FILE NOT FOUND events, where the Path is the Addon Texture directory and the Filenames are very long numbers like 012121103311011Su.bmp or in this schema: 058b2su2.bmp
Both indicate, that a Landclass file in the addondirectory looks for BMPs not available in that local Texture dir.
When you leave the fs at this position and only monitor the filemon, you will see that this events are happens every some minutes, even when you don't touch the FS.
When you open the Taskman and check the virtual memory in the processpage for the FS9.EXE, you will see that the memory grows every loadcycle.

I have now a tool, that can identify Lanclass BGLs in addon directories, where a Texture subdir exists, but I can't say, if the needed BMP's are all present, because therefor I must decrypt the BGL format to get the needed BMP names, which are not stored there. Very complex process. I hope, that after my holidays I can go forward here.

But: Keep in mind, that if you remove Landclass files or move them to a seperate entry, you may change the visual effect of your addon. Allways make a backup and document, where you make the change, so you can go back to default.

[psolk]

KORD seams to be ok, no Landclass included.

The easiest Test to check, if a area is critical, you can do the test I have done:

1) download and unzip Filemon into a directory and make a link on the desktop for easy start.
2) Save a default flight in FS with Cessna, clear weather, AI traffic off
3) Start filemon, go to Volumes Menu and select the drive of your FS
4) Apply a filter like this:
include = FILE NOT FOUND
exclude = .fx;.CAB;.agn;uteur
(the uteur is to exclude UTE, so if you have other UT maybe change or add that)

Now, jump via Goto Menu to a Airport, where you want to check the area, then check if you see
FILE NOT FOUND events, where the Path is the Addon Texture directory and the Filenames are very long numbers like 012121103311011Su.bmp or in this schema: 058b2su2.bmp
Both indicate, that a Landclass file in the addondirectory looks for BMPs not available in that local Texture dir.
When you leave the fs at this position and only monitor the filemon, you will see that this events are happens every some minutes, even when you don't touch the FS.
When you open the Taskman and check the virtual memory in the processpage for the FS9.EXE, you will see that the memory grows every loadcycle.

I have now a tool, that can identify Lanclass BGLs in addon directories, where a Texture subdir exists, but I can't say, if the needed BMP's are all present, because therefor I must decrypt the BGL format to get the needed BMP names, which are not stored there. Very complex process. I hope, that after my holidays I can go forward here.

But: Keep in mind, that if you remove Landclass files or move them to a seperate entry, you may change the visual effect of your addon. Allways make a backup and document, where you make the change, so you can go back to default.

Hey Oliver, this is really great stuff. My only issue is that for ~ 2 years now I have been disabling ALL scenery other than the two airports I am flying between. I seem to be getting the error when going between KEWR and EGCC with no UTE running at all...
Any ideas on how I could still be getting the G3d.DLL error in those circumstances?
Thanks in advance Oliver, your work is tremendous;y appreciated!
-Paul

[OPabst]

Hey Oliver, this is really great stuff. My only issue is that for ~ 2 years now I have been disabling ALL scenery other than the two airports I am flying between. I seem to be getting the error when going between KEWR and EGCC with no UTE running at all...
Any ideas on how I could still be getting the G3d.DLL error in those circumstances?
Thanks in advance Oliver, your work is tremendous;y appreciated!
-Paul

I never say, that this is the reason for every FS engine problem.

[psolk]

I never say, that this is the reason for every FS engine problem.

Was just hoping you had another idea in that brilliant head of yours I will try the diagnostics on those two sceneries now and report back shortly....
Thanks Oliver,
-P

[psolk]

Was just hoping you had another idea in that brilliant head of yours I will try the diagnostics on those two sceneries now and report back shortly....
Thanks Oliver,
-P

OK Oliver, no luck using just the sceneries and default A/C or the sceneries and the CS 757 which produced the errors BUT
when I introduced ASV6.5 which I am sure we all use I would see alot of NO SUCH FILE
This happens every 3rd or 4th cycle that ASV looks for:
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ NO SUCH FILE
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ Success
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ Success
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ Success
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ Success
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ NO SUCH FILE

Do you think this could be a factor? The only other thing see is a RANGE NOT LOCKED when trying to unlock the state.cfg of one of my AI...

[OPabst]

I don't have AS, so I can't say it. If the memory did not grow, I don't think it is critical.

But when you say you use AI Traffic, did you try to set the Traffic to 0 (off)? AI Aircraft can also generate CTD's and Memory problems, when Textures missing or have the wrong Format.

[psolk]

I don't have AS, so I can't say it. If the memory did not grow, I don't think it is critical.

But when you say you use AI Traffic, did you try to set the Traffic to 0 (off)? AI Aircraft can also generate CTD's and Memory problems, when Textures missing or have the wrong Format.

That will be my next test... I don't see any glaring issues with either of the sceneries or the aircraft and I know there is nothing wrong with AES so it is either AI or just the combination of all of the above with the /3GB switch that triggers the error.

Incidentally between 2 AES enabled airports I have done 16 hours nonstop from KEWR to VHHX in the PSS T7 and 12 hours nonstop from EGLL-VHHX in the PMDG 744 using Aerososft EGLL, Imaginsim KEWR, FlYTampa VHHX, UTUSA, PFE by Dave March, FDC by Dave March or FS2Crew by Bryan York, FSGensis Mesh 75% AI and all my other add-on Airports disabled in my scenery.cfg with NO issues at all using the /3GB switch...

It is only when I throw in the non-aes airport with AES enabled using the /3GB switch that I start to see the G3D.DLL errors...

Solution is you just need to put "some" sort of AES lite at ALL add-ons LOL

Still testing though,
-Paul

[mackintosh]

OK Oliver, no luck using just the sceneries and default A/C or the sceneries and the CS 757 which produced the errors BUT
when I introduced ASV6.5 which I am sure we all use I would see alot of NO SUCH FILE
This happens every 3rd or 4th cycle that ASV looks for:
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ NO SUCH FILE
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ Success
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ Success
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ Success
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ Success
C:\Program Files\Microsoft Games\Flight Simulator 9\Modules\ASGraphics\ NO SUCH FILE

Do you think this could be a factor? The only other thing see is a RANGE NOT LOCKED when trying to unlock the state.cfg of one of my AI...

ASGraphics is the AS add-on that feeds cloud and sky textures depending on the current METARs. It has been a source of some of my OOM errors, so I no longer use its cloud textures and have disabled it (I think some of them are prone to leaking...maybe a bad texture format, or could be what you are referencing above, I haven't checked ASG with Filemon).

[OPabst]

It is only when I throw in the non-aes airport with AES enabled using the /3GB switch that I start to see the G3D.DLL errors...

But you say you fly from KEWR to EGCC, then you have no AES at the destination!?

And for the other direction (from non to AES) it was for the most the helping to go short to a AES airport after start the FS and then jump to the non AES Airport to start the flight.