|
PacketFlow - For Ashita and Windower
By RadialArcana 2022-06-03 12:18:38
Would you be interested in trying a mod of that zone to see if it fixes your issue?
Someone asked me about this years ago and I made a mod for it, I can give you a link to try if you want.
I was lurking the thread when this issue was brought up and I believe the root of the problem was the smoke from the chimneys coming off of almost every building or something that was lagging the game more than normal. Was it your mod that "removed" the smoke with a texture edit?
The slow motion effect with 60 fps mode happens in other circumstances with above average stress which is why I thought PacketFlow might actually help relieve this problem.
Yeah, posted it ages ago. Some said it helped at the time.
Cerberus.Kylos
サーバ: Cerberus
Game: FFXI
Posts: 4,570
By Cerberus.Kylos 2022-06-03 16:23:47
Thanks very much for this, it was one of the most annoying parts of the game which I assumed would never be resolved. Might be another 9 months or so until I can try this, but just wanted to show my appreciation for the hard work.
If SE goes about banning people for this it would be disappointing, because it's not seriously affecting them, nor is it giving anyone an unfair advantage. It cleans up the information sent to us for a smoother experience, which makes for better and less frustrating gameplay.
However, if I were still playing, I wouldn't try this for at least six months, because if you really care about your accounts, then you've got to give SE time to confirm if this is something they choose to ignore. Fingers crossed they do ignore it, and you guys can make this one of the best addons to the game in years. Cheers!
PS: Does anyone know if this helps with animation lock? Like when you pull too many mobs and the game freezes you in place because it can't handle it?
Shiva.Thorny
サーバ: Shiva
Game: FFXI
Posts: 2,974
By Shiva.Thorny 2022-06-03 17:02:21
PS: Does anyone know if this helps with animation lock? Like when you pull too many mobs and the game freezes you in place because it can't handle it?
It doesn't, but there are a lot of ways to deal with that. One easy one would be to make an addon that temporarily blocks all action packets that hit you, but aren't initiated by you. You load it, stop seeing the mobs attack, get out of animation lock, then unload it. You could even implement it on a timer, like '/blockpackets 10' to give yourself 10 seconds of mob attacks hidden. Then you don't have to bother with an on/off.
[+]
サーバ: Asura
Game: FFXI
Posts: 5,163
By Asura.Daleterrence 2022-06-03 17:09:03
When using 60 fps mode in the config addon there are very apparent slow downs in some areas such as Western Adoulin. Would PacketFlow reduce this issue for 60 fps mode? I wouldn't think so but just to make sure, I figured I'd ask.
Would you be interested in trying a mod of that zone to see if it fixes your issue?
Someone asked me about this years ago and I made a mod for it, I can give you a link to try if you want.
I'd be interested in a link to this too, if at all possible. Anything that reduces that horrible slowdown would be very helpful
Cerberus.Kylos
サーバ: Cerberus
Game: FFXI
Posts: 4,570
By Cerberus.Kylos 2022-06-03 17:37:55
PS: Does anyone know if this helps with animation lock? Like when you pull too many mobs and the game freezes you in place because it can't handle it?
It doesn't, but there are a lot of ways to deal with that. One easy one would be to make an addon that temporarily blocks all action packets that hit you, but aren't initiated by you. You load it, stop seeing the mobs attack, get out of animation lock, then unload it. You could even implement it on a timer, like '/blockpackets 10' to give yourself 10 seconds of mob attacks hidden. Then you don't have to bother with an on/off.
If someone could make this I'm sure the community would love it, lol. The only workaround I had was making sure I'm engaged and stay close enough to my target to not disengage, but it can still be tricky. It's the worst feeling when you pull too many and get locked helplessly for over 5 minutes until you die.
サーバ: Asura
Game: FFXI
Posts: 3,185
By Asura.Geriond 2022-06-03 18:52:02
JaZero also makes you ignore incoming hits when it comes to pulling out your weapon. If you're fine with that aspect but not the other things it does, you can load it then unload it when you engage.
Shiva.Thorny
サーバ: Shiva
Game: FFXI
Posts: 2,974
By Shiva.Thorny 2022-06-03 20:58:02
JaZero also makes you ignore incoming hits when it comes to pulling out your weapon. If you're fine with that aspect but not the other things it does, you can load it then unload it when you engage. pretty sure since ja0 blocks client changing the state of the lock , rather than directly blocking the lock, that will just permalock you if you load it while locked
but i am also phoneposting and havent verified, so grain of salt
サーバ: Bahamut
Game: FFXI
Posts: 25
By Bahamut.Immortalknightx 2022-06-04 11:45:31
Bahamut.Immortalknightx said: »The best question yet. How do I get it?? I don't see a download link
Built in to windower. Konami code on plugin tab to unlock few hidden plugins including this one.
I stupidly dismissed this as sarcastic reply.. nope.
Go to windower's app with the 3 tabs, choose the plugins tab, on your keyboard: ↑↑↓↓←→←→BA
By DaneBlood 2022-06-04 12:11:39
Does this also help with input lag (IE, you issue a command and it does go through, but it takes like 1-2+ seconds to initiate)?
What you experience as "input lag" might as well be "output lag"
you click a button
action is sent to server
sever does the action
results is sent back
delay
delay
more delay
Your clients receive results
You see the results
its my understanding we send very little informmation to the sever vs how much we receive that most likely lag is on the feedback rather than the execution
Asura.Saevel
サーバ: Asura
Game: FFXI
Posts: 9,997
By Asura.Saevel 2022-06-04 12:21:02
Does this also help with input lag (IE, you issue a command and it does go through, but it takes like 1-2+ seconds to initiate)?
What you experience as "input lag" might as well be "output lag"
you click a button
action is sent to server
sever does the action
results is sent back
delay
delay
more delay
Your clients receive results
You see the results
its my understanding we send very little informmation to the sever vs how much we receive that most likely lag is on the feedback rather than the execution
Nah man, he keeps missing those sweet headshots when he fires with the controller.
サーバ: Sylph
Game: FFXI
Posts: 26
By Sylph.Theodren 2022-06-04 14:52:17
Has anyone experienced crashes while using this plugin? I gave it a try on Thursday, and ended up crashing to the desktop within 5 minutes of my first segment run.
It could be a complete fluke of course. Just curious if anyone else has had the same issue. I'll probably give it another try here in the near future.
Definitely appreciate all the hard work.
By DaneBlood 2022-06-04 16:53:31
So I have no idea about any of this stuff but does increasing bandwidth by 60% increase server cost for SE? Assuming enough people are using this.
I can picture it in my mind:
data analyst/business intelligence at square enix presenting in front of the producer/higher ups for ffxi
"...and this graph shows that our 10k monthly bandwith/server costs in june increased to 20k, we're investigating the causes but we believe it might be due to a third party addon."
I'm sure this will go very well for the future... the producer and execs will definitely turn a blind eye, what could go wrong affecting the bottom line of square enix on a game in its sunset stages?
Unless the claim it increases the bandwith has been exaggerated I would not risk it considering they will notice this and some higher up will get this shut down instantly
just for clarity
- it only 60% increase not 100% incres
- it would only be 60^in crase if everyone ran the addon
- it would only be a 60% increase in worstcase heavy traffic situations
So concidering all these affect reducing factor. the above scenario is not as likely as you potray it
By DaneBlood 2022-06-04 21:26:37
One easy one would be to make an addon that temporarily blocks all action packets that hit you, but aren't initiated by you. [/quote
What would be the drawback. you want see dmg to you in chat log? but hp would still be update?
would it be possible to just do it for "auto-attacks: si we can still see spell and TP moves ?
im thinking this would be very helpful addon if it didn't blind us to much
By DaneBlood 2022-06-05 00:01:43
Yeah, posted it ages ago. Some said it helped at the time.
link ?
サーバ: Leviathan
Game: FFXI
Posts: 3,753
By Leviathan.Celebrindal 2022-06-05 11:58:01
in general loving the results even with my piece of ***toaster that I play this game on, but this morning I had severe freezing of the game while dumping sparks/accolades with this running along with using SellNPC.
I don't have near the knowledge to accurately represent what was happening, I just know I've never experienced the game freezing that severely in just a town before, and thought any info was good info in this stage of the game.
[+]
Lakshmi.Zaps
サーバ: Lakshmi
Game: FFXI
Posts: 194
By Lakshmi.Zaps 2022-06-05 17:07:25
Internet usage is measured in bits, not Bytes, a bit is 1/8th of a Byte. The following math is in bits
10k people at 6.25kbps is 62.5mbps
add 60% and you have 100mbps
So, from a purely network usage perspective, we would have to decide is SE really cares about an increase of roughly 40mbps
The increased bandwidth is pretty small compared to anything their other services provide. Usually in the types of facilities SE would host the game from bandwidth is purchased by the gigabit (1000 kilobits is a megabit, 1000 megabits is a gigabit)
to put that in perspective, 40mbps is 4% of a gigabit.
a 60% increase in bandwidth, on a game made for DIAL UP, is still basically nothing.
*The math is meant to be close enough to illustrate how insignificant this should be to SE, nothing more.
Asura.Raelia
サーバ: Asura
Game: FFXI
Posts: 75
By Asura.Raelia 2022-06-05 20:48:07
Outside of consumer connections, egress isn't charged by speed, it's charged by data transferred. I already covered on page 3 that even at exorbitant egress rates their cost is a drop in the bucket. Their egress still costs less than 1% of their gross revenue at an absolutely insanely cranked to 11 worst case estimate.
Byrth and Thorny have also pointed out well enough that the game isn't sending you any more data than it would have anyway except in the laggiest instances, it's just sending the same set of chunks more rapidly.
But anyone else who needs to come in here and throw remedial networking knowledge around to feel better about using this plugin is welcome to, I guess.
サーバ: Asura
Game: FFXI
Posts: 941
By Asura.Iamaman 2022-06-05 20:57:19
So I'm trying to wrap my head around the way this works a little more. I'm assuming the server and client both keep a queue of chunks waiting to be sent that is purged, prioritized, coalesced into a UDP packet, then sent at certain intervals. If the queue fills before the interval, then lower priority chunks are discarded (or sent later?). I'm also assuming the server side purge/sendparse is triggered either by certain time intervals or the queue filling, but it also sounds like it can be triggered by receiving a packet from the client.
If all that is the case (assuming I'm not way off base), where does the 6.25 kb/s limit come from? Is that the frequency the server sends packets/purges regardless of what is in the queue? I'm reading that as the client isn't solely responsible for the send frequency.
Based on what Thorny said in reply to the CPU utilization question, I'm also assuming that this plugin works because the increased packet frequency from the client results in this purge sooner, meaning it's less likely that chunks waiting in the queue to be sent will be deprioritized or discarded since the purge is happening sooner. It'd also mean the only real consistent increase in traffic is whatever outbound/inbound messages are being sent at that 250ms interval (some keepalive or position packet?), which presumably is small, and the chance of increased bandwidth usage would only occur when the queue is filling at a higher rate and the response packets are larger because there are more chunks from the queue. In other words, the frequency remains fairly consistent, but bandwidth usage is dictated by queue size and CPU usage is a non issue since the only additional usage would come from having to prioritize multiple chunks.
Is that right or am I way off? I'm mostly curious about the implementation details, but also trying to understand the risk of there being an issue.
Asura.Raelia
サーバ: Asura
Game: FFXI
Posts: 75
By Asura.Raelia 2022-06-05 21:09:20
That's as well as I understand it, thinking of it as a fixed 400ms tick rate with a maximum 1.25kb payload that gets populated with queued chunks. To go back into speculation land, I bet they're cramming what they can into a 1500 byte max MTU with some wiggle room to avoid fragmentation, which means the true maximum bandwidth of the game is 3.75KBbps if someone wanted to include all possible overhead.
I'm also curious what kind of packet is being sent to the server to get it to spit one out sooner. If it's just an empty packet then I worry about it being atypical and causing the server to error/log it. If it's a player input packet of some sort I actually worry less because there'll just be more of them.
[+]
サーバ: Asura
Game: FFXI
Posts: 941
By Asura.Iamaman 2022-06-05 21:16:19
I'm also curious what kind of packet is being sent to the server to get it to spit one out sooner. If it's just an empty packet then I worry about it being atypical and causing the server to error/log it. If it's a player input packet of some sort I actually worry less because there'll just be more of them.
I'd expect the server is smart enough to discard any packets that are abnormal or don't pass basic parsing. I also think Thorny mentioned position packets earlier, my assumption being these are typically high frequency, small size packets that would not raise alarm (I'd also assume both server/client have some form of keepalive, but maybe position fills that role also)
Asura.Raelia
サーバ: Asura
Game: FFXI
Posts: 75
By Asura.Raelia 2022-06-05 21:23:19
Never realized until now that this was RAS Syndrome.
Shiva.Thorny
サーバ: Shiva
Game: FFXI
Posts: 2,974
By Shiva.Thorny 2022-06-05 21:42:08
-client sends a udp packet every 400ms under normal conditions
-this contains a 0x15 chunk, which is auto generated and includes your position, amount of movement, current target, last received sequencing id, etc. any actions youve taken since the last packet get appended to it
-server responds with a udp packet containing pending chunks and a sequencing id
the plugin alters the clients logic so that if you have received a server-client packet since the last client-server packet you sent, and 250ms have elapsed, this 400ms window is considered met
the client will generate the 0x15 entirely on its own, appending anything thats pending, the plugin only alters the timings. the higher frequency means more s>c chunks are sent out prior to being discarded
any additional cpu burden on the server would be from the greater frequency allowing for more position updates from both the player using the plugin to other players, and from mobs/other players to the player using the plugin. attacks, actions, etc.. would not be sent again. in most models this is not a significant burden, but we cant be 100% sure since distance calculations can be costly if applied to many other targets each movement packet
[+]
By SimonSes 2022-06-06 00:01:28
This plugin is awesome. Thanks a lot!
If I get banned for using it in few instances where I need it, then I don't care. I can understand lack of more content, but when I see something like this being possible to do by 1-3 people with no access to all the toys SE has access to, I just lose my temper. They could have done it so many years ago...
サーバ: Asura
Game: FFXI
Posts: 33
By Asura.Specialkid 2022-06-06 00:26:06
Kinda feels like that skeleton crew of a skeleton crew just doesn't know how to do modern programming imo
we can do all this stuff with external tools fixing the problems from the outside in, but these guys are hardfocused on fixing what they can from the inside out (which is not a bad idea, but that requires NOT being on a skeleton crew of a skeleton crew)
[+]
By Odin.Deridjian 2022-06-07 02:03:02
Matsui-P should get an intensive full-time English course to at least B2 to maybe C1 as part of a professional company training and speak to some persons of the Windower/Ashita community, then push to Yoshi-P and the CEO's to hire them freelance for a while. Can't even imagine what some people could accomplish if they just had the real back end on their hands. UX would be fixed in no time. You guys doing so much work around this game, it's astonishing. Great job actually posting this. Much love to all the devs and data miners out there!
Now announcing, PacketFlow for both Ashita and Windower!
Players have been having persistent issues with "lag" causing messages to drop in instances for years. In reality, this "lag" can be decomposed into two separate problems:
1. Chunk prioritization is messed up in instances
2. Instances generate a lot of chunks
PacketFlow attempts to address the second problem.
For a brief overview of FFXI network dynamics and terms, I recommend reading the spoiler: FFXI communicates between the server and client using UDP packets that have a max size around 2.5KB. Those packets are composed of multiple "chunks" of FFXI data, carrying information to either tell the server what you're doing (client -> server) or for the server to update you on the state of the game (server -> client). People frequently use the term "packet" to refer to chunks, but I'm going to keep the terms separate here because it is mechanistically important.
Examples:
When my FFXI client displays a message like "Byrth gains the effects of Phalanx" in the chat log, it knows to do so because it received a chunk telling it to display a gain buff message for my character within a larger UDP packet that may also contain a lot of other information. When you see my character run around, it is because my FFXI client sent the server a message telling it my new position (one chunk in a client->server UDP packet), and the server relayed my new position to your FFXI client (another single chunk in a server->client UDP packet).
Prioritization
In non-instanced zones, SE appears to use a prioritization system to make sure that your client receives more important chunks. If you pull all of the monsters in Korroloka Tunnel at once, for instance, and cast Phalanx on yourself, you will always see the "gain the effect of Phalanx" message. More generally, chunks containing information involving your character are prioritized over more contextual chunks (like monsters moving around). Most of the time, this prioritization system does not matter because FFXI is sending you less than 2.5KB of data and so everything makes it into the UDP packet. However, sometimes you're in a very active environment and there's more than 2.5KB of data to send you.
You can experience this when by pull multiple monsters. If 5 monsters are chasing you, they typically move pretty reasonably. If 20 are chasing you, some monster start to jump between locations because the chunk that updates those monsters' positions have low priority and do not fit in the packet. However, even with 20 chasing you the messages about their attacks are generally included (and fill up your chat log). This prioritization system lets FFXI get away with a pretty low bandwidth requirement, and the fact that most of us have never really thought about it indicates that it works pretty well.
However, in instances the prioritization system is at best nonfunctional or worse is functioning in the opposite way from normal zones. You will frequently miss messages about yourself in a Dynamis - D instance (buff gain, action completed, etc.), that you would never miss in a vanilla Dynamis zone. This is all server-side, so we can only guess at the specifics of the packet prioritization system and how it's messed up in instances, but it's clearly not working as it was designed to 2 decades ago.
What are solutions here?
1. Minimize the number of chunks you are sent: fight one monster at a time, turn off party buff information, don't use GearSwap, etc. Anything that decreases the number of chunks you are sent for FFXI is going to help you avoid missing chunks, because prioritization doesn't matter unless the packet is full.
2. Increase the server -> client bandwidth.
Option #1 sucks and is very limiting, so option #2 is the interesting one.
Bandwidth is more or less max packet size * frequency. For the FFXI server -> client connection, this is (2.5 kb / 1 packet) * ( 1 packet / 0.4s) = 6.25 kb/s most of the time. We cannot change either of those parameters directly because they are part of the server software.
However, years ago I noticed that the server responds to any UDP packet it is sent with another UDP packet. Thus, we can indirectly control the incoming UDP packet frequency by adjusting the outgoing packet frequency. This idea came to nothing for the better part of a decade mostly because I had no idea how to trigger the FFXI client to send UDP packets more frequently.
I was reminded of it earlier this year when KnugenJulian started posting in the Windower Discord asking if anyone had a fix for Odyssey lag. I brought up my purely theoretical fix, and Thorny made it real by connecting it with atom0s' UDP packet handler reverse engineering.
Thus, there are now plugins (both named Packetflow) for Ashita and Windower that decrease that time-per-packet from 400ms to 250ms, effectively increasing bandwidth by 60%. They do not completely solve the problem of dropped chunks in instances, but they do make it better. Additionally, they make loading zones quite a bit faster. I trained 20 Orcs in Dynamis D - Sandy on my RUN, running them 20' back and forth so they'd keep moving with two WHMs to keep me alive. The only purpose of this was extra packet generation (many movement and attack chunks from the monsters).
I also made a party of 6 (brd x1, rdm x5). Each RDM cast 6 barspells in rotation 20x (120 casts total). They precast naked and midcast in full 16 slots to generate equipset chunks. I set up a Lua to log cast completions and loaded it on two of the RDMs, one with PacketFlow and one without.
600 casts actually occurred, and:
* the character with packetflow saw 588, and all 12 missed were their own. (10% missed of their own, 0% of others)
* the character without packetflow saw 517, and 36 of those missing 83 were their own. (30% missed of their own, 10% of others)
So it works and that's the theory, but is it safe?
This is a major concern. As with all 3P tools, but perhaps even more here, you risk your account when you use this plugin. Packetflow is unambiguously detectable and increases the outgoing bandwidth costs for FFXI. Thus, for Windower it is in the Konami code section of the launcher.
As far as due diligence, we actually more or less finished these plugins in mid-April and I have been using the Windower one on my main and mule for the last month ~24/7. I'm not banned yet. We don't have a way to test it better than that and that's all of the testing that has been done. Caveat Emptor!
Attempts have been made to reach out to SE about these problems and to get a temperature read on the proposed solution:
* 2016 AMA - Bandwidth - basically "it would be too hard to fix"
* 2022 AMA - Prioritization
* 2022 AMA - Bandwidth
We waited a week for SE to post their hypothetical 2022 AMA forum follow-up where they answer those questions, but no dice.
Credit to:
* Thorny for the implementation and empirical testing
* atom0s for the reverse engineering that made it possible
* KnugenJulian for starting the conversation that revived the concept
|
|