RFC: Modular system for mods

[Nibelung44]

So how exactly would you expect to work a mod called, let's say, "Make lasers have longer range", that would only alter ranges of the stock laser weapons?

I don't see how that would work. How would a mod author know the mod's priority compared to all the other possible mods out there?

Well, you provide the data records of all edited lasers. Not just the values for the new ranges you want in.

[llunak]

Well, you provide the data records of all edited lasers. Not just the values for the new ranges you want in.

And so your "Make lasers have longer range" mod will conflict with mod "Make lasers have better accuracy", and we're basically back to the current situation, only files do not get overwriten on the disk but when processing.

[kabill]

Two (or more) questions:

1) Are there any plans to roll this out with the next X:CE build? (Please say yes!)

1a) If not, is there anything holding it back that non-coders like myself could help with (i.e. is there anything in particular that needs testing?)?

2) Does this function currently work the maps/submaps/props? If not, it would be a useful function to allow overwriting of existing map-related things without having to replace them (e.g. Fire in the Hole would be much easier to uninstall as you wouldn't have to delete the extra UFO maps).

[llunak]

1) Are there any plans to roll this out with the next X:CE build? (Please say yes!)

Hmm ... well, if you insist, yes.

1a) If not, is there anything holding it back that non-coders like myself could help with (i.e. is there anything in particular that needs testing?)?

"It" needs testing. See at the bottom. Everybody else is welcome to test it too.

The small catch is that I haven't written any docs for it except for the in-code comments, also see at the bottom. Do not get immediately scared by the length of the description, it's meant to be flexible, in practice the (admitedly somewhat simple) two testing mods require only one special tag. Knowing XML basics such as what an element or attribute is can be helpful. I'll write better docs or example mods whenever I get to it, hopefully in time for X:CE 0.25 .

2) Does this function currently work the maps/submaps/props? If not, it would be a useful function to allow overwriting of existing map-related things without having to replace them (e.g. Fire in the Hole would be much easier to uninstall as you wouldn't have to delete the extra UFO maps).

I actually don't know, I'd expect it should work on anything, but why don't you tell me ?

Executable (based on X:CE 0.24) and two example mods: http://ge.tt/1Pv1Wsn1

Description:

- mods are installed and deleted using modding tools in the launcher, the "mod to load" combobox is currently useful only for deleting, all mods are loaded and sorted in alphabetical order

- mod files are .zip files with no toplevel directory (i.e. the contents should be e.g. xenopedia.xml, not mymodname/xenopedia.xml)

- each mod file must include modinfo.xml (currently empty is enough)

- any errors are printed to standard (console) output

- if Xenonauts.exe is passed -modinfo , there is additional information about the asset merging printed

/*

How merging of XML spreadsheets works (for modding):

First row is header, it must match.

First column is identifier of each item (i.e. row).

For each row from a merged XML spreadsheet:

- if the item doesn't exist yet, it is appended

- if the item already existed

- if the item has all columns except for identifier empty, the item is removed (even if it previously existed)

- if the item has all columns non-empty, the item is replaced

- if the item has some columns empty, the item is updated - empty columns will keep previous values

*/

/*

How merging of XML documents works (for modding):

- if an element has a special attribute MODMERGEATTRIBUTE, this attribute specifies the name of attribute

that will be used for matching, i.e. elements match if their names match and values

for the given attribute match (this is for merging e.g. <Weapon name="weapon"...> elements in weapons_gc.xml),

if the attribute does not exist, two elements match if their names match

- if an element has a special attribute MODMERGE, this attribute specifies how the element will be merged

with a possibly already existing matching element

- if the value is "insert", if there is no matching element, insert the element, it is an error if a matching

element already exists

- if the value is "replace", the already existing matching element is replaced by this element (including

all its children), it is an error if there's no or more than one element to replace

- if the value is "replaceifexists", an already existing matching element is replaced by this element (including

all its children) if it exists, otherwise nothing happens, it is an error if there are more than one elements

to replace

- if the value is "insertreplace", use "insert" if there is no matching element, otherwise use "replace"

- if the value is "delete", the already existing matching element is deleted if it exists, it is an error

if there are more than one element to delete

- if the value is "deleteall", all already existing matching elements are deleted, if they exist

- if the value is "update", it is an error if there are no or more than one matching elements, otherwise

update possible attribute values and (direct) content, descent one level down and repeat for all children

- if the value is "updateifexists", it is an error if there is more than one matching element, otherwise

nothing happens if there is no matching element, otherwise update possible attribute values and (direct)

content, descent one level down and repeat for all children

- if the value is "updateall", use "update" for all matching elements

- if an element has no MODMERGE attribute

- if a matching element doesn't exist yet, it is appended in the right place including all its child elements

- if an element has no children elements

- if there's already exactly one matching element, update attribute values and content (this is mainly

for simpler modding of files like gameconfig.xml)

- if there are already more matching elements, that is an error

- (the no-element-yet case is handled above)

- if an element has children elements but no attributes besides MODMERGE, MODMERGEATTRIBUTE or the attribute

specified by MODMERGEATTRIBUTE (in this case an already existing element can match

even if it has attributes, this allows for easy updating of config elements that have attributes,

such as in moraleconfig_gc.xml)

- if there's already exactly one matching element, descend one level down and repeat

for all children

- if there are already more matching elements, that is an error

- (the no-element-yet case is handled above)

- otherwise it is an error (the element has both chidlren and attributes and there already exists

an element with the same name - in this case it's not easy to say if the element should be

merged (which would be suitable for settings) or added or replaced (either of which could be suitable

e.g. for <Weapon> elements in weapons_gc.xml)

*/

[kabill]

Ok, I've found an instance that the current system can't handle:

Modding the UFO mission files (xml spreadhseets), it's impossible to add in the same ufo type twice (e.g. to produce a dynamic ufo system) since the first column is used as the identifier.

This is what I started with:

This is what I changed it to:

When I did this, the mod wouldn't load*. I presume this is because there's multiple identifiers of the same type in the file which it can't cope with.

I've not tested this, but I think there will be issues with the UFO crew loadouts too as the columns don't actually bear any reference to one another (i.e. they're independent lists in the same file) and the identifier will in fact be one of the crew loadouts. If you change anything in the first column, it's going to get confused and replace data in other columns in an unhelpful way.

I'm wondering whether an easy fix for this wouldn't be simply to add in a 'null' column as the first one to use as the identifier instead and just fill it with an incremental number of something.

[Solver]

I presume the problem here is that the first column isn't actually the identifier, it's the first column together with the second that forms a unique identifier.

[kabill]

Ugh, posted on the other thread but yes, it's just using the first column but that's not necessarily a unique element (and this is even more true in the crew lists).

EDIT: For what it's worth, I just had a quick look and these two sets of files are literally the only two which experience this problem.

Edited July 11, 2014 by kabill

[llunak]

Modding the UFO mission files (xml spreadhseets), it's impossible to add in the same ufo type twice (e.g. to produce a dynamic ufo system) since the first column is used as the identifier.

Hmm, I see. I think the easiest change here would be to allow something like /[number] as a suffix in the first column that would be silently stripped, so you could write the second lightscout as m:airplane.alient.lightscout/50.

When I did this, the mod wouldn't load*. I presume this is because there's multiple identifiers of the same type in the file which it can't cope with.

There should be errors printed to standard output (and also more info with -modinfo).

I've not tested this, but I think there will be issues with the UFO crew loadouts too as the columns don't actually bear any reference to one another (i.e. they're independent lists in the same file) and the identifier will in fact be one of the crew loadouts. If you change anything in the first column, it's going to get confused and replace data in other columns in an unhelpful way.

Is that the files in ufocontents/ ? That is indeed a rather poor format. I think I'll need to change the format of that, so that the alien type is the first column and the counts are other columns.

Edited July 12, 2014 by llunak

[kabill]

Right, having finally got everything working:

It seems the suggestion I made above can be added in without any further coding changes. I added in an "ID" column as the first one and just filled it with incremental numbers. Doing this for the default file didn't affect the game at all - everything worked as it should. I then tried adding in a modded file as well which extended the list and that seemed to work fine too.

There might be some more elegant solutions, but this seems to be the simplest since all it involved is making a few minor changes to the xml files.

EDIT: I assume that this would work for the ufocontents files as well. Although restructuring them wouldn't necessarily be a bad idea, as I'm unclear whether you could add in extra mission types (i.e. columns) using the modular function.

[llunak]

The "ID" workaround is possible because only the merging code refers to columns by their number, everything else in the game refers to them using their names. So you basically explicitly told the game how to match the rows.

The problem with that is that you match on a "random" number. Modding when a certain type of UFO appears in a mission would require hoping the number never changes.

It's already in my todo list to fix both of these correctly for 0.26.

[kabill]

The "ID" workaround is possible because only the merging code refers to columns by their number, everything else in the game refers to them using their names. So you basically explicitly told the game how to match the rows.

The problem with that is that you match on a "random" number. Modding when a certain type of UFO appears in a mission would require hoping the number never changes.

I'm not sure why the numbers would change, though. There's only two circumstances I can think of: some extra variables are added into the files (but you could add them in with different numbers as identifiers, e.g. 0.1, 0.2 etc. and this would work fine without disrupting anything); and compatibility with other mods.

However, the system you've described above doesn't help with multiple mod compatibility either, as the numbers appended to the UFO types are equally arbitrary. E.g. if one mod adds in scout/50 somewhere and another mod wants to add in corvette/50 on the same mission type, these will simply stack with one another rather than overwriting.

(I'm actually not convinced that trying to combine multiple mission-chaning mods is ever going to work well, save for ones which change the variables at the top.)

[kabill]

Sorry for double post, but didn't want this to be missed:

Just modding Strings.xml, I noticed that the top row is one of the strings. Since the top row always needs to be the same, it might be worth adding a header row. It's not a big issue - it only matters if you want to change that string - but I can imagine it causing some confusion for modders who don't realise and simply delete the top row because it's a string like all the others.

EDIT: Ugh, another thing:

In the xml spreadhseets, you can't merge information in the same cells. This is problematic for the researches file, since it's quite possible that mods will want to add addition tech/lore unlocks to existing research projects.

EDIT: And another thing:

It doesn't seem to like me adding in new weapons. I get a CTD every time I try to open the tab with the relevant weapon.

To check it wasn't a data-entry error, I copy/pasted the entries straight into the base weaons.xml and items.xml files and it worked fine. In another test. I copied one of the existing game items and gave it a name change to create a new item. Again, in the mod file this didn't work.

I also tried changing the file path for the weapon images to account for the mod placement, but that made no difference either.

My hypothesis is that it's related to the Items.xml file. When adding an item to weapons.xml but not items.xml in the base files, I got (what appeared to be) the same crash, but when I I tried putting an item in items.xml but not weapons.xml, it loaded fine (there was no item, but it didn't crash either).

I'd give more information using the modinfo output, but I don't know where to find it.

EDIT: And another (although I thought this might be the case) - it doesn't work with submaps. I tried my Enhanced Crash Site mod in the mod-folder and it didn't work; shifting the maps into the default folder instead made it work fine.

Edited July 12, 2014 by kabill

[llunak]

(I'm actually not convinced that trying to combine multiple mission-chaning mods is ever going to work well, save for ones which change the variables at the top.)

Some might not work well, one way or another, others might. If somebody wants to insert another UFO type at a certain ticker value, my way will work, while using ID would be problematic (e.g. if there would be two mods, they could clash).

Just modding Strings.xml, I noticed that the top row is one of the strings. Since the top row always needs to be the same, it might be worth adding a header row. It's not a big issue - it only matters if you want to change that string - but I can imagine it causing some confusion for modders who don't realise and simply delete the top row because it's a string like all the others.

Actually strings.xml is special, because it's read by the engine and not by the game's code, so there really is no header. There could be, or I can mention this in the docs.

In the xml spreadhseets, you can't merge information in the same cells. This is problematic for the researches file, since it's quite possible that mods will want to add addition tech/lore unlocks to existing research projects.

The question is, how exactly would you want the cells to be merged. A person can look at it and figure it out, but computers tend to be simple-minded. I suppose "When finished" could reasonably well work by just appending the cell contents, since it's a sequence of commands, but "Requires" is a logical expression and merging that is not going to be trivial.

It doesn't seem to like me adding in new weapons. I get a CTD every time I try to open the tab with the relevant weapon.

...

I'd give more information using the modinfo output, but I don't know where to find it.

Nowhere, seems Windows just discards it. It may be a bug in the code or a problem in the mod, but since you can't debug it until I change the output to show up somewhere, I can have a look if you provide me with the mod .zip file.

[kabill]

Nowhere, seems Windows just discards it. It may be a bug in the code or a problem in the mod, but since you can't debug it until I change the output to show up somewhere, I can have a look if you provide me with the mod .zip file.

Here's two:

[ATTACH]4922[/ATTACH]

[ATTACH]4923[/ATTACH]

The first only has a modded gamconfig file in it. It changes the radar range (which I was using for testing) and should also change a number of other variables related to UFO missions and funding. However, some of them don't appear to work. For example, in this test I've tried turning off Research mission events entirely and the continue to occur. Similarly, UFOs should be able to spawn up to 6 hours after the beginning of a wave, but they continue to spawn together like in vanilla. So while some changes (i.e. radar range) are working, others are not.

The second is the one indicated above related to items. In theory, it should add a breaching charge to the "other" equipment tab but the game crashes when you go to that tab.

GAMECONFIG_ODDNESS.zip

ITEMS_ODDNESS.zip

GAMECONFIG_ODDNESS.zip

ITEMS_ODDNESS.zip

Edited July 12, 2014 by kabill

[silencer]

I have a question - how this application will react if there are two mods that modify same thing but with different values?

[kabill]

It will overwrite them with whichever mod is given higher priority.

[Solver]

That is answered in the first post.

Mods will overwrite what the game provides, higher priority mods overwrite lower priority mods (currently mod priority is in their alphabetical order, later in alphabet is higher priority).

Of course the alphabet solution is temporarily while in development, mod priority will become something that is easy to select.

[llunak]

kabill: X:CE 0.25 has added another column to items.xml , so the merging fails. The gameconfig.xml problem was a bug in the code, it should merge properly in 0.25 hotfix (which should also put debug information somewhere you can read it, for future problems). Thanks for testing this.

[kabill]

Ta. Totally missed the fact that there was a new column added. As for testing, no trouble. Thanks for making the thing in the first place!

[skaianDestiny]

I have a question: I know that higher priority mods overwrite lower priority ones if they have the same values, but how does it deal with two mods modifying Researches.AleniumExplosives? Mod A adds UnlockKnowledge("Researches.ExampleA") while Mod B adds UnlockKnowledge("Researches.ExampleB"). Would it add both of these or overwrite one of them?

[Solver]

That is a case of data being in the same attribute, not just the same element - it will be overwritten, not merged.

[skaianDestiny]

Dang. Oh well.

[Solver]

It's not really feasible to merge those automatically though, as llunak explained.

[skaianDestiny]

Yeah, I'll just do that manually then.

[kabill]

Is it not possible to add something some kind of function in the cell which will tell it to add to rather than replace the contents of the cell? I can understand this being a problem with the prerequisite cell, since that's going to be too complex to add to, but the ones for unlocks are basically just lists of things.