Jump to content
llunak

RFC: Modular system for mods

Recommended Posts

As you probably noticed, while the game is well moddable, the situation is worse when it comes to actually using the mods. Currently the standard technique is overwriting game's files with ones provided by the mod, which has several problems, such as game updates breaking mods, several mods breaking each other, and mods reverting game updates. There is a more flexible mod system currently in works, and I'd like to collect some feedback from more experienced modders.

Modified game executable: [ATTACH]4832[/ATTACH] . It is built on top of X:CE, so it presumably should be used on top of X:CE.

Set of two testing mods: [ATTACH]4833[/ATTACH] . There are two mods, called mod1 and mod2, they belong in assets/mods directory.

How the system works. Note that this is work in progress (there will be eventually UI, some things will be improved/changed):

- All subdirectories of assets/mods/ are treated as mods. Each mod directory acts as that mod's assets/ subdirectory (so mod foo would provide assets/aircombatimages/paused.png as assets/mods/foo/aircombatimages/paused.png).

- When the game wants to load an asset, it will merge it from all mods (if they provide/change the asset) and the stock game. Mods will overwrite what the game provides, higher priority mods overwrite lower priority mods (currently mod priority is in their alphabetical order, later in alphabet is higher priority).

- There are several types of resources:

* Binary files, such as images (currently only images work). These obviously cannot be merged. The asset will be used from the mod with the highest priority that provides the asset.

* Excel XML files, such as xenopedia.xml. These always use the first column as the identifier of the whole item.

+ Only items which are to be changed need to be present in the mod's file (e.g. mod1 in examples changes only two xenopedia items, so it has only two rows).

+ Higher priority mods replace items with same identifier of lower priority mods, or add new ones if they do not exist yet.

+ If a mod provides an item that has all columns except for the identifier empty, the item will be actually removed (e.g. mod2 in examples removes xenopedia entry for Storeroom).

+ If only some columns are empty, they will be kept from the item they are replacing (mod2 in examples replaces only MedicalCenter description, other columns come from lower priority mods or game itself). This allows the mod to change only data it intends to change, keeping game data for the rest it doesn't want to change.

* "Normal" XML files, such as gameconfig.xml . These always have one root element and merging is currently done on (direct) children elements of the root element.

+ Mods need to provide only children of the root element that the mod wants to change.

+ If a higher priority mod has a child element that has attribute called Name or name, whole xml subtrees of such a child from a lower priority mod is replaced by the xml subtree (mod1 in examples replaces <Sound name="Click" ... </Sound> xml tree with its own that changes the click sound to the cannon OGG file).

+ If there's no replacement, the whole xml subtree is appended as the last child. This also works for other XML element's that do not have Name/name attribute, and it is assumed that the same XML tag being later in the XML will overwrite any possible previous XML tag when the game reads the data (mod1 in examples changes region border settings this way).

* Others. These are currently not supported.

The game prints debug information to the standard output about the merging process, these are prefixed with "MODS:".

First question: Does this make sense?

The system should have the following advantages:

- No modification of game files (so no problems with Steam updates, etc.).

- Files of each mod are kept in one separate place.

- Mods should much less likely conflict with each other.

- Mods should be much less likely to revert game changes in newer versions.

- It should be more obvious what a mod changes when looking at its files.

Now, merging of (non-Excel) XML files could be still improved, as currently it's necessary to include a whole XML subtree, even if one wants to change just a single value. For example, currently it is not possible to just change accuracy of a snap shot of weapon.pistol in weapons_gc.xml without providing a copy of its whole <Weapon ... </Weapon> XML tree, which may again lead to conflicts just as when replacing a whole file.

(Second question: Does somebody have a good idea for a solution here? Meaning, different and better than the idea presented below.)

It looks like a solution for this would be merging XML also at lower levels. That is however not trivial because of XML attributes that may or may not match, and some of them being different may mean different items while others being different just mean changing the same item. For example, different name attribute in <Weapon> in weapons_gc.xml means a different weapon, while different emptySound just means changing an aspect of the item. Therefore, the current idea is that mods would add an additional tag that would say how merging should exactly be done. For example:

<Weapons> <Weapon name="weapon.pistol" emptySound="Empty Click 2" modmergename="name">   <SingleShot modmerge="simple">     <Set1 ap="28" accuracy="65" />   </SingleShot> </Weapon></Weapons>

This would change weapon.pistol to have a different empty sound and different TU cost and accuracy for snap shot. Specifically, modmergename="name" would say to merge with XML element whose attribute with the given name has the same value, modmerge="simple" would simply merge with another tag of the same name. Not shown here, but modmerge="replace" would just replace the whole XML subtree, like is currently done with Name/name XML subtrees. Existing attributes would add/replace (if emptySound wasn't specified here, it'd be unchanged, the same with ap or accuracy).

Third, fourth, fifth and sixth questions:

- Does this make sense?

- Would it be needed/useful in practice?

- Is it still simple enough?

- Are there other ways of merging needed?

Seventh and eighth question:

- Do you know what diff/patch are and would you be capable of creating one?

- Is there anything else important/useful to say?

Xenonauts.exe.zip

mods.zip

Xenonauts.exe.zip

mods.zip

Edited by llunak
Added mention of the game printing debug output

Share this post


Link to post
Share on other sites

Yes, this makes sense.

Such system will definitely be useful - more mods compatibility means more mods and less problems with installation most of the time.

And, as you describe, it should be simple to use and functional.

Only thing i have concerns about - diff/patch may be not versatile enough for xml merging, at least to my experience.

Found but not yet tried this thing that may or may not be more helpful - diffxml

Share this post


Link to post
Share on other sites

I think this makes sense to me. I'd probably need to play around with it to get it settled in my head and to make sure I've not misunderstood, but it sounds like a really good way of doing things to me (from a layman's perspective, anyhow).

Share this post


Link to post
Share on other sites

I don't think you should bother going down to a modding granularity so small as the one where you only want to replace some values withing a given data record. If a data record (a weapon, an alien) is present in a mod, then all values must be filled in, and it will replace the one previously loaded.

But that's IMHO.

As for loading priority, a numeric tag or something per file would be best probably.

Share this post


Link to post
Share on other sites
Only thing i have concerns about - diff/patch may be not versatile enough for xml merging, at least to my experience.

That is why the game will merge the XML itself, as described. And not by using diff/patch, that is meant to be for the hard cases (.lua scripts).

Found but not yet tried this thing that may or may not be more helpful - diffxml

I think this would be more fragile and harder to use. Also, it's GPL-ed, so (depending on one's interpretation of the licence) it would be problematic to include it with the game.

Share this post


Link to post
Share on other sites
I think this makes sense to me. I'd probably need to play around with it to get it settled in my head and to make sure I've not misunderstood, but it sounds like a really good way of doing things to me (from a layman's perspective, anyhow).

Then do play around with it a bit :), I can wait a bit if needed, that's why I'm asking for feedback so early. It's important that this works well for modders, so input from people like you matters. If somebody finds out a month later that it's all wrong, I'm not going to change it then.

Share this post


Link to post
Share on other sites
I don't think you should bother going down to a modding granularity so small as the one where you only want to replace some values withing a given data record. If a data record (a weapon, an alien) is present in a mod, then all values must be filled in, and it will replace the one previously loaded.

So how exactly would you expect to work a mod called, let's say, "Make lasers have longer range", that would only alter ranges of the stock laser weapons?

As for loading priority, a numeric tag or something per file would be best probably.

I don't see how that would work. How would a mod author know the mod's priority compared to all the other possible mods out there?

Share this post


Link to post
Share on other sites
Then do play around with it a bit :), I can wait a bit if needed, that's why I'm asking for feedback so early. It's important that this works well for modders, so input from people like you matters. If somebody finds out a month later that it's all wrong, I'm not going to change it then.

Yeah, if I get chance (so many things to do) I'll have a go at building one of my mods using this system at the weekend. For me, that will be the best test case, since I'm more likely to spot issues if I'm working through it.

Out of wonder, for things like config.xml, am I right in thinking that most of the values in there would require replacing an entire 'section' (I'm not sure what the proper turn is. Is subtree right?).

E.g. If I wanted to change the damage range from +/-50% to +/-100%, would that require including the whole <shot></shot> section that it is contained in?

If so, is it possible to resolve that using the system you outline for dealing with subtree issues?

(Apologies in advance for any other quite possibly stupid questions I ask about this stuff. Just want to make sure it's straight in my head.)

Share this post


Link to post
Share on other sites

I have some questions about the setup.

I make the complete structure in "mod" folder for example mod/mod1 I run the executable this read the "assets" folder with priority over "mod" folder, that can include multiple mod and give priority in alphabetical order "modA", "modB, "modC", etc.

Can you help me to run some test? for me is not clear hot to implement.

Share this post


Link to post
Share on other sites
Out of wonder, for things like config.xml, am I right in thinking that most of the values in there would require replacing an entire 'section' (I'm not sure what the proper turn is. Is subtree right?).

Not in the case of config.xml . That's the case of

"

If there's no replacement, the whole xml subtree is appended as the last child. This also works for other XML element's that do not have Name/name attribute, and it is assumed that the same XML tag being later in the XML will overwrite any possible previous XML tag when the game reads the data (mod1 in examples changes region border settings this way).

"

The game will read first the <Shot> from the game's config.xml, set damage range based on the setting there, and eventually it would get to the appended entry from the mod and set damage range again base on that setting.

This approach wouldn't work with e.g. weapons_gc.xml, where each <Weapon> section describes another weapon.

Share this post


Link to post
Share on other sites
I make the complete structure in "mod" folder for example mod/mod1 I run the executable this read the "assets" folder with priority over "mod" folder, that can include multiple mod and give priority in alphabetical order "modA", "modB, "modC", etc.

Sorry, I don't understand what this question is actually about (and it's in fact not a question).

Share this post


Link to post
Share on other sites
That is why the game will merge the XML itself, as described. And not by using diff/patch, that is meant to be for the hard cases (.lua scripts).

My bad - somehow i thought that "Only items which are to be changed need to be present in the mod's file" meant making diff instead of stripped xml.

Trying it now, 1 bug so far:

1) cities.xml - simple xml, works as intended (London disappears, Hastings appears)

<?xml version="1.0"?><Cities> <City>   <Name>London</Name> </City> <City>   <Name>Hastings</Name>   <Country>United Kingdom</Country>   <Population>7</Population>   <Location>4108,0890</Location>   <PopupDistance>0.6</PopupDistance>   <DetailDistance>0.7</DetailDistance> </City></Cities>

2) strings.xml - excel xml, works as intended (London string changed, Hastings string created)

   <Row>   <Cell><Data ss:Type="String">London</Data></Cell>   <Cell><Data ss:Type="String">London_1</Data></Cell>  </Row>  <Row>   <Cell><Data ss:Type="String">Hastings</Data></Cell>   <Cell><Data ss:Type="String">Hastings</Data></Cell>  </Row>

3) strings.xml - excel xml, does not work (London string displays as empty string instead of being removed, Hastings string properly displays as "####" - nonexistent string)

   <Row>   <Cell><Data ss:Type="String">London</Data></Cell>  </Row>

Overall very easy to use for simple changes that i tried so far - Good work, llunak!

Share this post


Link to post
Share on other sites
This approach wouldn't work with e.g. weapons_gc.xml, where each <Weapon> section describes another weapon.

Ah, there we go. I think it was this point that I was overlooking. Thanks.

Share this post


Link to post
Share on other sites
3) strings.xml - excel xml, does not work (London string displays as empty string instead of being removed, Hastings string properly displays as "####" - nonexistent string)

That's because strings.xml merging is actually handled by the game engine, not the game itself. I don't see any practical use for removing strings from there though, so it shouldn't matter in practice.

Share this post


Link to post
Share on other sites

@llunak - Not a bug, but a feature then =) Good to know.

I'm very impressed with the whole thing and the fact that it understands stripped to the essentials excel xmls.

Share this post


Link to post
Share on other sites

This should be of huge benefit to the community, imo.

Share this post


Link to post
Share on other sites

Ok, so I've been trying this out with a small mod, but I can't seem to get it to work properly.

I've placed the modded files assets/mods/[MODFOLDER]

The modified files are moraleconfig_gc and psionicpowers_gc

Since I'm modifying most of the stuff in these files, I've not removed anything and just used the two files wholesale.

When trying the changes in game, however, they haven't taken. It's not an error in the files themselves: I've tried replacing the vanilla files as I'd usually do and it works fine that way. Furthermore, I've tried some other changes (making an image change, and changing the starting money) and they've worked fine. So it's something specifically to do with those files (well, moraleconfig_gc anyway. I've not tested any of the psionics changes so that might be fine).

Any thoughts why this would be?

Share this post


Link to post
Share on other sites
The modified files are moraleconfig_gc and psionicpowers_gc

Code reading moraleconfig_gc.xml always uses the first matching group it finds, so the appended ones from the mod aren't used. This needs the better XML merging (or recoding the code in question).

Share this post


Link to post
Share on other sites

How is it I'm always at work when the best discussions happen. Erm, I have a question. I do all my editing in notepad++. This is because I can't afford Excel, and both Open Office and Libre Office seem to have problems with the Excel-xml files. (Open office can't open them, Libre Office can't save them). Is there anything in the system you've devised that would have problems with Excel-xml file edited purely in notepad++, Do I have to break open the piggy bank and find the pennies for a copy of Excel?

Share this post


Link to post
Share on other sites
How is it I'm always at work when the best discussions happen. Erm, I have a question. I do all my editing in notepad++. This is because I can't afford Excel, and both Open Office and Libre Office seem to have problems with the Excel-xml files. (Open office can't open them, Libre Office can't save them). Is there anything in the system you've devised that would have problems with Excel-xml file edited purely in notepad++, Do I have to break open the piggy bank and find the pennies for a copy of Excel?

Why about Apache Open Office?

Share this post


Link to post
Share on other sites
Is there anything in the system you've devised that would have problems with Excel-xml file edited purely in notepad++

No, I don't see why. I actually do all editting by manually too. BTW, what exactly is the problem with LibreOffice saving, it seemed to be ok when I just tried (incidentally I might know somebody who could possibly fix it)?

Why about Apache Open Office?

There's no other OpenOffice now than the one from Apache (and its development is generally lagging behind LibreOffice).

Share this post


Link to post
Share on other sites

This is the single most important modding support feature that the game needs.

Just take a look at game seris with undoubtedly largest modding community ever: Elder Scrolls III, IV and V. The games are not that great on their own, but with incredible modding capabilities, they've built communities that extend the games' lifetime for more than a decade. And in my opinion, the most important factor is the ability to combine those millions of mods with relative ease, through a simple load order system. Without that, the user is limited to what a single modder offers, instead of being able to pick, choose and combine. By measure of agency, the numbers don't lie.

Anyway, Great work Ilunak! I'll check this out and how it works with my mod works.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×