Server freezes (Fixed!)

  1. 8 years ago

    Ok, forum seems to keep losing my password or booting me out for some reason.

    We have been using auto-generation or Player generation of missions from the Player Operations in the HQ to run our missions.

    What you're saying is that missions should be generating for us if we go out and do active patrolling, with interaction with civilians bringing us these missions?

    The current build of M4RTI4L's mission that I'm attempting to get working has Spyder's Civilian interaction scripting. So, that should bring about missions for the unit?

    If so, I'll try to get the unit trying that out, both with and without the C2ISTAR module in. I may have to turn off all auto-generation of missions with it in though to hopefully keep the freezing from happening.

  2. @ ski2060 Yep exactly what we are experiencing, We have the same mission running, create "things" to do with Zues and it all runs perfectly well.
    Stupid ammounts of AI spawning, CQB, civilians, server handles it nicely and will run it for days no freeze.

    As soon as C2Istar is there are missions are requested/auto generation is used. Boom, freeze will occur.

    Other thoughts I have to test, but have not yet are:
    OPFOR Faction. We use the RHS insurgent faction or the SUD_Insurgents factions.

    Island choice, we have been using SUP islands. May give it a run on Stratis/Altis again. - this may be a factor as I have noticed occasionally when creating a C2Istar mission it will spawn units in the lower left corner of the map. well away from the objective, they are still grouped etc, just off in the middle of nowhere. its the the entire group either, useusally 1 or 2 men out of a section.
    So perhaps its a compounding problem ? cannot spawn units together, cannot move them together etc etc.

  3. I'm willing to bet at this point it's just server branch issues. I ran M4RTI4L's mission with AIATP, and then with CUP and it hung the server with either one active.

    If it could be nailed down and get stabilized so I can run our tweaked version, that would be phenominal.

    I'll see if I can get the performance binaries put up on our server.
    Do all the clients have to be running the same EXE's? Or just the server running one?

  4. Edited 8 years ago by ski2060

    I have 4 days off starting tomorrow (but tomorrow is daughter's birthday party). So, I'll see about throwing a mission together with a basic ALiVE setup, CBA, and ACE. Now, what ACE modules should I put down? Because there are a ton of modules for it- Advanced Medical, ballistics, mechanics, etc. Should I just drop em all down and see how that runs? Or maybe just Advanced Medical and Ballistics?

    Also, I noticed from streaming my RPT file off host site, that I get the same Server: Object not found error repeating in significant amounts bloating the RPT. BUT, it's not crashing or freezing the server.

  5. I ran Operation Landlord with @AliveServer no problem. CBA, ALive, AliveServer and ACE. Ran several days 12-16 hours each time with no crash or Freezing.

    I get massive amounts of the Object not found errors if I am running ACE and RHS togther.
    I tried dropping LeightsOpFor for CAF Aggressors and still got the freezing. RHS was involved both times.

    I found a modded version of RHS Russian Insurgency the other night that seemed to be running stable. So, I swapped the Russian stuff out for US and have it up on my server now. @CBA_A3;@ace;@ALiVE;@task_force_radio;@IWAR- Irregular Warfare;@Leights OPFOR Pack OFFICIAL;@Spyder Addons;@RHSUSAF;@RHSAFRF;@Ares;@CUP Terrains - Core;@CUP Terrains - Maps;@Aliveserver

    It hasn't crashed or frozen yet as far as I can see. Although It is throwing the Object not found error, and I've been the only one on the server.

    I'll see if removing @aliveserver does anything if the currently running mission freezes.

  6. So, I have a question and maybe something to narrow down the freezing.
    SOAR team, are you using an RHS BLUFOR or OpFor OpCom in your missions?

    Whenever I have a mission using an RHS faction as BLUFOR (typically USarmy_d) I have the freezing issues.
    I have a modified RHS Russian Insurgency mission that I swapped out the player faction from Russian to US Army.
    However, I am not using a BLUFOR OpCom. I have 2 enemy OpComs, an IWAR USML_AIF Occupation commander, and a Leights OpFor ISTS Insurgent commander.
    the plan is to have the map heavily occupied by the main AIF forces, and the ISTS are recruiting from the local civilians. This will put some heavy pressure on the players when they are in the field.

    I have this mission up and running now, and it appears to not be freezing. I had 3 unit members on yesterday for several hours with no problems.
    I left the server up and running overnight, so it's probably been on 16 hours or so at this point without crashing. I have no logs though, I turned off logging because I was getting bloated RPTs with the Server: Object #:###### not found (message type_91 or _119) messages, but the server was not freezing.

    I will try to leave this mission running for a couple of days and see what happens. It is on 0.9.10 right now, and I plan to update to 1.0 middle of next week when I am off work again.

  7. @SOAR_Jooce I guess I don't really understand why you're testing with ACE now. Are you simply wanting to eliminate it as well or are you hoping that the freezes will give you more info than other mod combos? We had crashes without ACE running so it looks ALiVE simply doesn't play nice with other mods on 1.54+.

    Because internal testing has only indicated crashing with ACE active.

  8. Good to see the issue is be narrowed down.
    I'll see if I can get any testing done over the next couple of days.

    I have the standard Operation LandLord and an ACE version I put together. I'll see if I can get some warm bodies to help me test out both versions if possible and look for crashing.

  9. @SpyderBlack723 Because internal testing has only indicated crashing with ACE active.

    Ah, internal testing. Now that our weekend ops are over, I'll test the ALiVE mission that Tupolov sent back to us tonight. Really curious to see if it was JUST module configuration when RHS is involved.

  10. Edited 8 years ago by uk_apollo

    First I'm going to say what a wonderful mod @alive is, and how it has been enjoyed by all of us at 3 Commando Brigade on every single one of our busy public server missions for well over a year. Awesome job, thanks!

    Unfortunately we have also been suffering with the "freezes" in this winter weather. Identical to all the reports you've previously had here. We do run a large modset so I won't bore you with rpt's as I know you need minimal mod installations to help find the solution. But I thought it worth reporting the issue to add to the data-set, especially as we have built up a lot of experience running Alive on 24/7 dedi servers catering for 20-40 players on most days of the year.

    Symptoms

    • players see others freeze in place or walking on the spot
    • vehicles rubber band
    • players will claim to be at the same place but not see each other
    • players may not be able to join in progress
    • red chains sometimes, but not always
    • desync = 0 in map/player info
    • suddenly occurs for all players, seemingly randomly from 5 minutes to 3 hours into a mission
    • server rpt huge, spammed with specific error messages (example below)
    • server reports very low on virtual memory (memory leak)

    Facts

    • it started happening sometime after the A3 v1.54 update and/or updates to @alive and @ace3 at that time. The same modset and same missions prior to that were stable.
    • the problem goes away if you remove @alive from a larger modset
    • it's still freezing with the Eden update
    • not all of our missions freeze: we have a number which use @alive/@ace3/@rhs_ amongst other mods, which appear to be 100% stable (10 hours+ up-time)

    Our conclusions so far

    • it's not related to spawn density (happens with minimal ai numbers spawned, even with all players still at base)
    • it's not player number related (has happened with just 2)
    • it's not related specifically to being in combat (has happened with all players in a blacklisted base)
    • it's probably not a specific @ace3 module setting, as all of our missions have identical parameters (ballistics etc); some work, others don't
    • it's probably not the 3rd party Opfor type selected in the @alive modules - server freezes with BIS AAF enemy too (although to be fair, RHS/Leights is still loaded in the modset, but not being used)
    • it doesn't appear to be related to a particular @alive module (such as C2Istar); I knocked it back to the bare minimum last night and still saw rpt spam; asymmetric or conventional warfare alike
    • it probably is a conflict between @alive and either @ace3 or @rhs_ (possibly @leights_opfor) apparent only since an update from about a month ago, in agreement with the consensus opinion here

    Server RPT sample
    17:39:30 WARNING: SuppressCache::Trace called with huge input. Function exited to prevent program stall inside. tBeg[-0.000000,165.000000,-0.818642]; tEnd[0.000000,0.574304,12276.468750]; ammoHit(1.745155); distance(0.000000)
    17:39:30 In last 500 miliseconds was lost another 2 these messages.
    17:39:49 Virtual memory total 4095 MB (4294836224 B)
    17:39:49 Virtual memory free 57 MB (59785216 B)
    17:39:49 Physical memory free 25890 MB (27147853824 B)
    17:39:49 Page file free 56668 MB (59421413376 B)
    17:39:49 Process working set 3530 MB (3701755904 B)
    17:39:49 Process page file used 3762 MB (3944747008 B)
    17:39:49 Longest free VM region: 3604480 B
    17:39:49 VM busy 4233289728 B (reserved 150560768 B, committed 4082728960 B, mapped 74436608 B), free 61546496 B
    17:39:49 Small mapped regions: 24, size 106496 B
    17:41:40 NetServer::finishDestroyPlayer(1970388062): DESTROY immediately after CREATE, both cancelled

    The WARNING message is spammed, thousands of times.

    www.3commandobrigade.com

  11. I guess i can piggy-back on this thread regarding the server hangs and crashes.

    My friend and I tried to make an Asymetric mission the other week and started to have these errors, mostly the server hangs and crashes after its memory has been filled up.

    So we dug into the problem and started a dev with @ALiVE (1.0.0.1602091) and @CBA (2.3.1) only (using the stable branch). The mission is made on Altis and in the "Eden" editor the mission.sqm binarization is turned off. Here are my impressions:

    -It seems to come from the Installation placement in the MIlitary AI commander, when activated, in local server i get the following error "Error undefined variable in expression: _state" "x\alive\addons\fnc_analysis\fnc_liveanalysis.sqf, line 514" when you start running trough the map towards the enemy "bastion" once the modules loaded up (until the Intel appears).

    -Removing the Assymetric mode or just simply deactivating the Installations stops the error. Other modes like Invasion and Occupation doesnt generate the error AFAIK.

    When loaded up on the dedicated, you can start the mission, load it up at 100%, leave it run for as many hours/days as you want it wont crash, its only when you log in and start walking towards the enemy that the error occurs. It freezes the server and chugs up all the RAM until the dedicated server crashes. You won't notice this if you play alone, only that the server on "#monitor" doesn't post anymore after a while.

    I did a "portfolio" with the mission file, and the two RPT's (one from my local machine testing in local host, and the other from our dedicated (*HUGE RPT 24mb*)) and also a screencapture of the error i got in local host running this mission. The mission inside is quite light (couple of kbs) but its made to stress test our machine also, we didnt want to overlook certain aspects, so if im correct this mission takes a bit to load up and has 86 objectives triggering the "80" limit warning. Also, if the mission is badly "structured" with the custom objectives let me know, for all i know maybe im trying to do something impossible with ALiVE and just plainly breaking it because im overwhelming the Military Commander with multiple Custom Objectives (is there a limit to the number you can synch to it?)

    Ive read that it might be the error that HighHead is working on but i might be mistaken. Hoping for the best.

    FTP download

    Hope this helps