Also worth mentioning is that LuaJIT and JS engines have different constraints. For example, JS engines must be very careful about security, LuaJIT does not. So JS engines need to do more cleanup/initialization for example, to prevent things like heap spray attacks, which LuaJIT does not. Also, JS engines are written to be maintainable by a large team, LuaJIT is really focused entirely on speed.
LuaJIT is written with security in mind. LuaJIT 2.x had permanent write-protection enabled for the generated machine code in the first release, i.e. before anyone else did so.
I do not see how maintainability could somehow have an inverse relation to the engine speed. IMHO the LuaJIT code base is significantly more maintainable than the Mozilla code base.
Can you elaborate on what you mean by "permanent write-protection"? Is that to prevent code being overwritten? If so, that is just one possible attack.
Does LuaJIT implement techniques to mitigate heap spray attacks and heap inspection problems? JS engines work against that by initializing things like typed arrays to 0, and by overwriting objects when they are free'd. These have a runtime cost though, I'm curious if LuaJIT incurs it as well.
Regarding maintainability, your handwritten interpreter in assembly is not something I would expect other people to easily get up to speed on. The JS engine codebases are more approachable IMHO.
Also worth mentioning is that LuaJIT and JS engines have different constraints. For example, JS engines must be very careful about security, LuaJIT does not. So JS engines need to do more cleanup/initialization for example, to prevent things like heap spray attacks, which LuaJIT does not. Also, JS engines are written to be maintainable by a large team, LuaJIT is really focused entirely on speed.