Server freezes (Fixed!)

  1. 8 years ago
    Edited 8 years ago by ski2060

    That sounds exactly like the problem I'm having on my server. I've been trying to run MArtials Takistan NATOFOR ALiVE mission in it's most recent form, as well as modified versions to fit my unit's play style, and this is happening to me.

    I can run the server, it will populate with units as normal.
    I can join, others can join, an we can play with player or ALiVE generated missions. But at some random point ,2 minutes in or 2 hours in, it will freeze up for one or more players. It will not crash, just freeze things in place while one or others can still move about.
    We have to ALT+F4 out, or if possible exit Arma, and try to come back in. Server will show as up, but we cannot load back in.
    Most recent RPTs I have for it I think:
    https://www.dropbox.com/s/ayzs4i6tlcnpt89/arma3server_2016-01-22_04-24-45.rpt?dl=0
    https://www.dropbox.com/s/efxmsxdoaoxxqxi/arma3server_2016-01-22_00-53-34.rpt?dl=0

    I will try to remove the C2ISTAR module and see if only running it with Player generated missions will fix the problem.

    Edit:
    Just loaded up my version of NATOFOR (just tweaked medical and TFAR a little, changed bad guys to ISTS) with the removal of C2ISTAR. So, autogeneration of tasks is gone like I expected, but it also removes manually generated missions completely. So, there are no tasks that are able to be generated.
    I see that OpFor units are still being placed by ALiVE OpCOm, so I assume the Mission just becomes Sandbox at that point , with players tasking missions for themselves, or Zeus placing missions.

    @ Monty, Would that be a fair assessment of what you guys are experiencing?

  2. It's not ALiVE related..it's ARMA. We just had 2 desyncs/Out of Memory crashes tonight within 15 minutes of starting the mission using Dynamic Combat Generator.

    1.54 totally FUBARed everything server wise. The desyncs/OOM crashes are widespread amongst the community right now. Sometimes it will desync 5 min in..sometimes 2 hours in....but it ultimately will crash and after almost 2 full months...the issue still has not been resolved. Very aggravating, especially when trying to run long term persistent campaigns as you end up desyncing before making real progress in your campaign.

    I love you ARMA..and yet I hate your freaking guts.

  3. Edited 8 years ago by HeroesandvillainsOS

    Thanks Savage. I'm on V15 right now and again played for another 2 hours with no issues. Granted, I'm not trying to host the session for other players, but on my end things are much more stable now.

    One BIG thing I noticed is that placing @aliveserver anywhere other than last in your server load order will almost 100% cause desync and crashes so I'd recommend to Ski to look at that.

    I've also seen an early build of his mission and we use a lot of the same stuff and principles. However, last I saw he was using some mods I haven't played with in forever so in my un-trained gut, I still feel like he must have some kind of mod conflict or script conflict going on. Or someone in his crew is running something the others aren't. Just my totally uninformed two cents.

  4. And think, with the new update coming, even more things may be broken. I think they're are making some changes to file architecture and other stuff. I'm not looking forward to half the mods I use being broken :(

    I am currently in progression testing to see which mods are working and which aren't, also.
    Basic AliVE Takistan map with CBA and Alive, ran fine for 12 hours yesterday.

    Put anything with ACE up, and it hangs. Not even a hard crash, but teh same as you, a Freeze, with no messages generated on screen.
    The server always shows Cannot Communicate Errors.

    I think I'll either find or put together a mission with just Alive and RHS to test also.
    My unit would hate to lose ACE, so I would probably be relegated to NOT using Alive for my experimental server.

  5. I think it's ACE. I am using RHS Insurgency ALIVE and it is not crashing. I had it on overnight with a couple unit mates playing on it, no crashes.
    Put NATOFOR Insurgency that has ACE on it, and it Freezes in anywhere from 2 minutes to 2 hours.

    There is something in ACE that is conflicting with ALIVE. No clue what it might be though.

  6. So I have Operation Landlord with ACE running on my server right now if anyone want to see if they can get it to Freeze. It's been up about 5 hours now with no freezing.
    But I've only had one player on it so far as I'm at work.
    If anyone wants to jump on and test it out to see if they can get it to freeze, please feel free.
    69.162.125.22 -port=2347
    ski2060 Testing Server
    @CBA; @ACE, @ALiVE. Everything is vanilla otherwise.

  7. I ran Operation Landlord with @AliveServer no problem. CBA, ALive, AliveServer and ACE. Ran several days 12-16 hours each time with no crash or Freezing.

    I get massive amounts of the Object not found errors if I am running ACE and RHS togther.
    I tried dropping LeightsOpFor for CAF Aggressors and still got the freezing. RHS was involved both times.

    I found a modded version of RHS Russian Insurgency the other night that seemed to be running stable. So, I swapped the Russian stuff out for US and have it up on my server now. @CBA_A3;@ace;@ALiVE;@task_force_radio;@IWAR- Irregular Warfare;@Leights OPFOR Pack OFFICIAL;@Spyder Addons;@RHSUSAF;@RHSAFRF;@Ares;@CUP Terrains - Core;@CUP Terrains - Maps;@Aliveserver

    It hasn't crashed or frozen yet as far as I can see. Although It is throwing the Object not found error, and I've been the only one on the server.

    I'll see if removing @aliveserver does anything if the currently running mission freezes.

  8. Tupolov

    Show in context Administrator
    Edited 8 years ago by Tupolov

    @SOAR_Jooce honestly, your point about CBA and ALiVE working fine was lost in the noise :) If that is the case then we just need to ID the mod it conflicts with.

    We appreciate your testing and effort absolutely. As you can imagine we want to focus on specific ALiVE issues. If we can nail down if its ALiVE and ACE (for example), then we know where to start. This thread just is full of people reporting issues with a whole mix of mods, we just don't have the spare hours to chase those combination of mods down.

    We are interested in fixing any conflict with ACE (or RHS).

    If someone can confirm the issue with CBA, ALIVE and ACE - that would be great!

    Also, if someone can confirm the issue exists with ALiVE 0.9.9 (and any modset) that would be useful too!

    Thanks for your patience and apologies for any perceived "aggro" from us :)

    If you do confirm that issue - please pass us RPT log, plugin log and the mission SQM/files.

    Cheers

  9. @Tupolov @SOAR_Jooce honestly, your point about CBA and ALiVE working fine was lost in the noise :) If that is the case then we just need to ID the mod it conflicts with.

    We appreciate your testing and effort absolutely. As you can imagine we want to focus on specific ALiVE issues. If we can nail down if its ALiVE and ACE (for example), then we know where to start. This thread just is full of people reporting issues with a whole mix of mods, we just don't have the spare hours to chase those combination of mods down.

    We are interested in fixing any conflict with ACE (or RHS).

    If someone can confirm the issue with CBA, ALIVE and ACE - that would be great!

    Also, if someone can confirm the issue exists with ALiVE 0.9.9 (and any modset) that would be useful too!

    Thanks for your patience and apologies for any perceived "aggro" from us :)

    If you do confirm that issue - please pass us RPT log, plugin log and the mission SQM/files.

    Cheers

    Your own Developers have said they have reproduced it with 100% success rate. Using CBA/ALIVE an ACE is pointless at this time. Are you not in communication with your developers?

    If they have a 100% success rate of reproducing the issue, why are we going back to square one? We have done nothing but support ALiVE and give information non stop. We tested Alive 1.0 within 30 minutes of hearing it was released. And as soon as we report it does not fix the issue Tupolov goes full defensive on the Arma subreddit, Tupolov acts like ALiVE cannot possibly be conflicting with the mods, even though your developers have now clearly stated that it does.

    "It does not mean that ALiVE is at fault. Until we can get a reproducible issue without other mods (i.e. just CBA and ALiVE) then we can actually go about fixing something." - Tupolov

    Guess what, if removing ALiVE from our modpack brings stability. It does mean ALiVE is partially to blame.
    Now please work with your developer who says he has a 100% chance of reproducing the issue, it seems like he is the only one actually doing positive work to get this done and fixed.

  10. @dixon13 That means the server hasn't given up and crashed yet.

    Yeah, but when it sits there for hours without fully crashing it kind of impacts your ability to use the server to play on. So it just gets restarted or the mission changed.

    If I can find a suitable replacement for RHS I would use it. CUp would most likely work, but lots of my unit guys don't seem to like CUP, and think their models are too low quality for some reason.

    I wouldn't mind it, and may switch on a map build just to see how it looks and plays.

  11. @ski2060 Yeah, but when it sits there for hours without fully crashing it kind of impacts your ability to use the server to play on. So it just gets restarted or the mission changed.

    If I can find a suitable replacement for RHS I would use it. CUP would most likely work, but lots of my unit guys don't seem to like CUP, and think their models are too low quality for some reason.

    This is the exact situation we have been put in as well. The server will just freeze and will not create any meaningful logs. If we don't shut down the server manually it will hang there forever.

    CUP is a quantity over quality mod (weapon/unit) so its not an option for replacing RHS which is quality over quantity. I used to be a CUP supporter until I realized this fact.

    And we can't drop RHS to support ALiVE to only have CUP as a backup. Its why we have chosen RHS over ALiVE at this point. We can replace some of the functionality of ALiVE with DAC if we need to. Obviously we would prefer ALiVE but seeing as there really isn't hope for a fix in the near future, we will have to focus on implementing DAC with Headless Client.

    If the ALiVE devs can give us something to test other than "CBA, ALiVE, and ACE" we will help, but testing redundant mod packs is out of the question at this point. We know those three are not conflicting, and we know RHS is.

  12. Tupolov

    Show in context Administrator
    Edited 8 years ago by Tupolov

    @SOAR_Griz "our developer" didn't reproduce the issue with just ALiVE, CBA and ACE (or ALiVE, CBA and RHS). He reported his clan gets the issue most of the time when running his clan's set of mods - which there are many - it can happen 30mins in, it can happen 4 hours in. He is in the same boat as everyone here reporting the issue with many mods enabled. This doesn't help us locate which mods are conflicting. Its about as useful as anyone else reporting the issue with a fuck ton of mods enabled.

    "Our developer" didn't state that ALiVE conflicts with other mods. He simply confirmed he has seen the issue with his clan. Again, its no different to what is being reported here. There are many people reporting issues with MP play, hangs/freezes/crashes that are not using ALiVE too.

    Anyway, he isn't really a developer as reported, he's mainly our build/release guy. I'm the main developer of ALiVE at this point in time while ARjay and Highhead are away. If this problem is ALiVE related and its going to get fixed, them I'm the one who will have to fix it. So, I'll need as much help as possible. Unfortunately all I can do at this point is ask for people to nail down exactly which mod conflicts, the mission used, the number of players, the logs and any other relevant info. I don't play Arma regularly with any clan as most of my spare time is spent coding ALiVE :(

    I'm not getting defensive about anything here or on reddit :) Just saying the facts as reported, no one yet has been able to identify which two mods (if at all) are causing the issue.

  13. So, I have a question and maybe something to narrow down the freezing.
    SOAR team, are you using an RHS BLUFOR or OpFor OpCom in your missions?

    Whenever I have a mission using an RHS faction as BLUFOR (typically USarmy_d) I have the freezing issues.
    I have a modified RHS Russian Insurgency mission that I swapped out the player faction from Russian to US Army.
    However, I am not using a BLUFOR OpCom. I have 2 enemy OpComs, an IWAR USML_AIF Occupation commander, and a Leights OpFor ISTS Insurgent commander.
    the plan is to have the map heavily occupied by the main AIF forces, and the ISTS are recruiting from the local civilians. This will put some heavy pressure on the players when they are in the field.

    I have this mission up and running now, and it appears to not be freezing. I had 3 unit members on yesterday for several hours with no problems.
    I left the server up and running overnight, so it's probably been on 16 hours or so at this point without crashing. I have no logs though, I turned off logging because I was getting bloated RPTs with the Server: Object #:###### not found (message type_91 or _119) messages, but the server was not freezing.

    I will try to leave this mission running for a couple of days and see what happens. It is on 0.9.10 right now, and I plan to update to 1.0 middle of next week when I am off work again.

  14. Tupolov

    Show in context Administrator

    Hey,

    Yeah to have asymmetric work with CQB, the Mil module has to have same faction as CQB. So, I updated that. I ensured that OPCOM had groups to work with by doing this.

    Also, to remove any doubt, I got rid of extra scripts etc. I know that this mission has been pushed out as ALiVE Insurgency, but tbh the additional scripts etc are not part of ALiVE and could impact performance.

    So, I ran the mission for 6 hours on dedicated with RHS using rhs_faction_insurgents. Just me though and a local dedicated server. I placed the player in a position where OPFOR would spawn and start taking over the island.

    I didn't experience a freeze, but as you say there was no competing BLUFOR force other than AI at the base.

    Mission is here https://drive.google.com/open?id=0Bydf4NQlStBjTm1JSFRvcmNBenM

    I'll do more testing this weekend.

    Cheers

  15. Tupolov

    Show in context Administrator
    Edited 8 years ago by Tupolov

    Looks like 1.52 is stable so far - 4 hours of testing... With ALiVE 1.0 and same mission that was freezing with 1.54+

  16. @Tupolov Looks like 1.52 is stable so far - 4 hours of testing... With ALiVE 1.0 and same mission that was freezing with 1.54+

    SOAR also rolled back and attempted to use 1.52 prior to ALiVE 1.0 as well. It appeared to resolve the freezing issues but created more issues for us as a result.

  17. Might the issue be cause by a single module in ACE and ALiVE conflicting?
    I put up my Operation Landlord with ACE today, and it's been running about 12 hours with no crash/hang.

    I had 4 players on earlier an nothing happened. We used Arty, called Supplies, used transport and CAS helis. Ran around in a stolen T100 and killed a bunch of CSAT.

    I am still using V16 for the Perf EXE as I haven't been able to get it updated to the latest binary yet.
    I will get you a RPT if needed.

  18. Edited 8 years ago by JerkinMerkin

    Hi guys.

    I have tested with 1.54 running only @ALiVE, @CBA, and @AliveServer and my server froze around 7 hours 30min. With @ace the server froze anywhere from 10min to 1 hour. Therefore it appears to me that @ace is not the issue, it just makes the issue worse.

    More info can be found here:

  19. Edited 8 years ago by uk_apollo

    First I'm going to say what a wonderful mod @alive is, and how it has been enjoyed by all of us at 3 Commando Brigade on every single one of our busy public server missions for well over a year. Awesome job, thanks!

    Unfortunately we have also been suffering with the "freezes" in this winter weather. Identical to all the reports you've previously had here. We do run a large modset so I won't bore you with rpt's as I know you need minimal mod installations to help find the solution. But I thought it worth reporting the issue to add to the data-set, especially as we have built up a lot of experience running Alive on 24/7 dedi servers catering for 20-40 players on most days of the year.

    Symptoms

    • players see others freeze in place or walking on the spot
    • vehicles rubber band
    • players will claim to be at the same place but not see each other
    • players may not be able to join in progress
    • red chains sometimes, but not always
    • desync = 0 in map/player info
    • suddenly occurs for all players, seemingly randomly from 5 minutes to 3 hours into a mission
    • server rpt huge, spammed with specific error messages (example below)
    • server reports very low on virtual memory (memory leak)

    Facts

    • it started happening sometime after the A3 v1.54 update and/or updates to @alive and @ace3 at that time. The same modset and same missions prior to that were stable.
    • the problem goes away if you remove @alive from a larger modset
    • it's still freezing with the Eden update
    • not all of our missions freeze: we have a number which use @alive/@ace3/@rhs_ amongst other mods, which appear to be 100% stable (10 hours+ up-time)

    Our conclusions so far

    • it's not related to spawn density (happens with minimal ai numbers spawned, even with all players still at base)
    • it's not player number related (has happened with just 2)
    • it's not related specifically to being in combat (has happened with all players in a blacklisted base)
    • it's probably not a specific @ace3 module setting, as all of our missions have identical parameters (ballistics etc); some work, others don't
    • it's probably not the 3rd party Opfor type selected in the @alive modules - server freezes with BIS AAF enemy too (although to be fair, RHS/Leights is still loaded in the modset, but not being used)
    • it doesn't appear to be related to a particular @alive module (such as C2Istar); I knocked it back to the bare minimum last night and still saw rpt spam; asymmetric or conventional warfare alike
    • it probably is a conflict between @alive and either @ace3 or @rhs_ (possibly @leights_opfor) apparent only since an update from about a month ago, in agreement with the consensus opinion here

    Server RPT sample
    17:39:30 WARNING: SuppressCache::Trace called with huge input. Function exited to prevent program stall inside. tBeg[-0.000000,165.000000,-0.818642]; tEnd[0.000000,0.574304,12276.468750]; ammoHit(1.745155); distance(0.000000)
    17:39:30 In last 500 miliseconds was lost another 2 these messages.
    17:39:49 Virtual memory total 4095 MB (4294836224 B)
    17:39:49 Virtual memory free 57 MB (59785216 B)
    17:39:49 Physical memory free 25890 MB (27147853824 B)
    17:39:49 Page file free 56668 MB (59421413376 B)
    17:39:49 Process working set 3530 MB (3701755904 B)
    17:39:49 Process page file used 3762 MB (3944747008 B)
    17:39:49 Longest free VM region: 3604480 B
    17:39:49 VM busy 4233289728 B (reserved 150560768 B, committed 4082728960 B, mapped 74436608 B), free 61546496 B
    17:39:49 Small mapped regions: 24, size 106496 B
    17:41:40 NetServer::finishDestroyPlayer(1970388062): DESTROY immediately after CREATE, both cancelled

    The WARNING message is spammed, thousands of times.

    www.3commandobrigade.com

  20. ditto to UK apollo. Ran my British Steel mission with another friend for over 2 hours with no issues. Thought we were in the clear. Called for CAS and a Helo Transport. As soon as we are about to land on the helo....desync.

    Checked the RPT...same out of memory crash that apollo mentioned above.

    Immediately restarted the server and launched the helicopters again, told them to land...landed fine this time.

    So....in conclusion....still getting crashes....not happening "consistently" in any one instance.

    My British Steel mission uses ALiVE + ACE...no RHS. 3CB and CAF Aggressors are the unit packs.

    I will say...there is a definite difference though in performance. I think we're close.

  21. Newer ›