Jump to content
Chris

Desura Ongoing Issues (& Build V16.1)

Recommended Posts

OK, so I suspect Desura has hit breaking point with our latest version (Build V16.1). The mirrors issue that meant nobody could download the game yesterday was caused by my uploads to the branch being corrupted. It seems that Desura is crashing when it's generating the updates on my PC, and when these files are uploaded they cause the mirrors issue.

Thus we've rolled back V16.1 to V16. The Desura guy is going to upload the patch from his end tomorrow and try to get it working. It may be a minor issue causing this, or it may simply be their upload system has melted under the sheer weight of files in our game.

Please just use the standalones for the next few days. We are working on a permanent fix for the Desura client, but all the solutions are a bit drastic. Essentially the problem is that when Desura updates the game, it performs an MD5 check on every file in the player's file directory to see if it needs to be updated. Because we have so many files, this takes a very long time and often causes crashes.

My wider concern with this is that I think Steam works in a similar manner. It's more than possible that Steam would be having similar issues if we were using them, because games simply don't have 80,000 files in them these days. It's a result of us using a woefully inadequate game engine which we don't have the source code for, and it'll only get worse when we add in all of the alien sprites for beta.

Two things we're looking at to potentially fix it up:

1) I'm talking to our ground combat programmer about consolidating all the files into a a few big single files. I'm not sure this will be possible, but if so it's clearly the best option. Not having access to the engine source code means what would be straightforward on other projects is not so here.

2) The first time you download the game on Desura is absolutely fine because the MD5 checks aren't needed. In an emergency, we can just set Desura to download a full new copy of the game every time there's an update. This is our fallback option (it also means that if the game is really stable when we release it on Steam and no patches are needed, the number of files won't be an issue).

The post-install unzip system we proposed recently doesn't look like it'd help the situation at Desura's end, unfortunately.

Anyway, my main concern here is the sheer number of people who don't read the forums etc and are buying the game and not having it work. It's really not a good place for us to be in, and we'll get it sorted one way or another in the near future.

For those curious about Steam, I did check with them and their position is still that they'll only sell the game when it goes gold (ie. when it's finished).

Share this post


Link to post
Share on other sites

Chris,

we know sh.. happens.

But, as I mentioned before in another tread: it's OK waiting for the update a few more days.

But no access to the game at all - hurts!

Share this post


Link to post
Share on other sites

Yes ideally you need a way to consolidate your files, you said it yourself other games just don't have the vast amount of files you have.

I hope for your next game you pick a better engine.

We are grateful for all your hard work though.

Share this post


Link to post
Share on other sites

Ah, that's really unfortunate. Hope you guys can find a solution!

I'm not sure if it would be too much to ask, how big a job it would be, or if you're allowed to at all, but is it possible to release a simple zip with the updated files in it that we could simply extract to the game folder? Can't afford to download another 2 gigs! :(

Share this post


Link to post
Share on other sites
For those curious about Steam, I did check with them and their position is still that they'll only sell the game when it goes gold (ie. when it's finished).

Is this because you are a smaller developer? It's just I pre-purchased Fallen Enchantress through Steam a few weeks ago (due out 23rd October) and Steam has been allowing people to download and play the beta from their client for a few months now, and when Stardock patch the beta Steam downloads the latest patch. The only requirement is that you pay for the game in advance.

On the subject of Xenonauts I haven't bought the game yet, and the Desura situation is putting me off. I'm waiting for the game to be available through Steam. One way or the other though I will be buying your game - can't wait for the release!

Share this post


Link to post
Share on other sites

I know it's very late in the day to ask this, but if engine troubles due to file count issues are going to continously keep plaguing you (and affect end users) and probably get worse when all the content is actually finally added in beta, and will likely affect all distribution platforms, is it perhaps worth a serious look at moving to a newer engine like Unity?

I know it would be a lot of work and you possibly don't have the money or will to do it, but you've already got most of the art assets along with the 3D models, you could move away from having mass amounts of sprite sheets, and you'd have a lot more freedom with how you go about adding or reimplementing other content.

Share this post


Link to post
Share on other sites

Um, changing engine is not that simple im afraid. your looking at adding months and months of extra work to change engine, at this late stage its possible to create more bugs then before possible to break features. it be like a programers worst nightmare. i mean unlike normal programming, you don't know where certain elements have broken and you have to crwal though the code to find them. and then things that do work might not work well and you need to rewrite them and then that might break other things. honestly thats like a path no programer should take this late.

But chris don't worry about it. im sure your sort something out.

Im cheering for you :)

Share this post


Link to post
Share on other sites
Um, changing engine is not that simple im afraid. your looking at adding months and months of extra work to change engine, at this late stage its possible to create more bugs then before possible to break features. it be like a programers worst nightmare. i mean unlike normal programming, you don't know where certain elements have broken and you have to crwal though the code to find them. and then things that do work might not work well and you need to rewrite them and then that might break other things. honestly thats like a path no programer should take this late.

Yeah....I'm well aware of that. Hence why I qualified I knew it would be a lot of work, and possibly not financially doable as well. I wouldn't have suggested it at all as normally it's a very very stupid decision to make late on in a project (DNF is a good example of why not to do it), but the distribution problems may force the the issue.

Share this post


Link to post
Share on other sites

Nah, changing the engine is not an option sadly. We'll see how it goes, the coders may have found a ray of light on this issue but we'll see in the next few days.

Share this post


Link to post
Share on other sites
Nah, changing the engine is not an option sadly. We'll see how it goes, the coders may have found a ray of light on this issue but we'll see in the next few days.

Keeping my thumbs pressed :) as per my request a few days ago: http://www.goldhawkinteractive.com/forums/showthread.php/3394-Technical-Question-Resource-Management

Wishful thinking regarding game-engine would be to look at the way ProjectZomboid has implemented their isometric engine including map streaming: http://projectzomboid.com/blog/index.php/2012/09/of-pz-payments-map-streaming-and-status-updating-ness/ -- Please don't bash me for pointing it out but some thinking outside the box may sometimes help with your own situation ...

Share this post


Link to post
Share on other sites
Keeping my thumbs pressed :) as per my request a few days ago: http://www.goldhawkinteractive.com/forums/showthread.php/3394-Technical-Question-Resource-Management

Wishful thinking regarding game-engine would be to look at the way ProjectZomboid has implemented their isometric engine including map streaming: http://projectzomboid.com/blog/index.php/2012/09/of-pz-payments-map-streaming-and-status-updating-ness/ -- Please don't bash me for pointing it out but some thinking outside the box may sometimes help with your own situation ...

Although I have been aware of Xenonauts for a while, what prompeted me to actually buy it was the positive comments in the PZ forum.

http://theindiestone.com/community/viewtopic.php?f=8&t=9232

Both good communities from what I can tell.

:)

Share this post


Link to post
Share on other sites

Yeah, though some people here are bit too fanboyish with their true successor comments. But yeah, besides that this is good community.

Share this post


Link to post
Share on other sites

I wonder if its possible to stack Xenonauts' files into file-like folders, that would act like ".jar" format does? It would make them easy to correct and modify but would look like a single file to applications like Steam and Desura.

Share this post


Link to post
Share on other sites

I'm playing with a binary patch tool from my previous life as a game-dev/design/infrastructure guy. If it goes well, I'll be sure to get some folks to test and more than happy to provide the tool and key to Chris and co.

Share this post


Link to post
Share on other sites

For those curious about Steam, I did check with them and their position is still that they'll only sell the game when it goes gold (ie. when it's finished).

I guess you have to know someone who knows someone, because there are many beta and some even alpha games that are, and have been sold on Steam.

I don't know the what's and why's but Steam might be full of **** :)

Share this post


Link to post
Share on other sites

Not just Desura and Steam, 7zip gets notable lag when I open the standalone archive just from counting all the files.

Good luck to Chris and the coders on consolidating stuff, but why don't they have the engine source code?

Share this post


Link to post
Share on other sites
Not just Desura and Steam, 7zip gets notable lag when I open the standalone archive just from counting all the files.

Good luck to Chris and the coders on consolidating stuff, but why don't they have the engine source code?

This is answered in this thread featuring a quote from Christ himself. Also, here's another with a comment from Giovanni and a bonus explanations of the difference between the game engine and the game code.

Share this post


Link to post
Share on other sites
Yes ideally you need a way to consolidate your files, you said it yourself other games just don't have the vast amount of files you have.

I hope for your next game you pick a better engine.

We are grateful for all your hard work though.

fully 3d xcom same mechanics as here :P would be awesome :D

on another note this is ALPHA!!! i used quite some time figuring out how to get the new version and that there was one and an issue with v16.0 but it doesnt bother me at all since this is alpha, the fact we can even play is freaking awesome and we thank you for that :D (not many devs allow their fans to play around in alpha versions and they will just have to wait while we get to try out this awesome game already now ^^ very impressed with it btw keeping the true mechanics and spirit of xcom alive and still adding some nice new features which allows for the old features to be used in a much more fluent and nice way :D )

Share this post


Link to post
Share on other sites

Two things we're looking at to potentially fix it up:

1) I'm talking to our ground combat programmer about consolidating all the files into a a few big single files. I'm not sure this will be possible, but if so it's clearly the best option. Not having access to the engine source code means what would be straightforward on other projects is not so here.

Well, but I guess you can get a hold on the documentation of the SDK, right? From what I gathered from the Reddit thread you're using the Playground SDK, which obscure or not, isn't as obscure as not warranting someone to upload the documentation to scribd

http://www.scribd.com/doc/88352047/PlaygroundSDK-2#outer_page_30

Checking the Chapter "Playground Fundamentals" I quickly bump into this:

2.24 Flat File Basics

Playground supports grouping most of your assets into a single flat file. The support is completely transparent toyour game—when you open up a file in the assets folder, it will behave identically regardless of whether it existsas a file on the disk or embedded in a flat file. One difference your users might notice, though, is that your gamewill install more quickly with a flat file—and that it will load more quickly when you run it. On Windows we usememory-mapped file IO to access the file, meaning it’s particularly fast to open and read small files.A flat file also provides a small amount of obfuscation to your assets—it’s no longer possible to just open up the.png files in your game by double-clicking them. In fact, PlayFirst does not provide a way to open up a flat file, though of course a serious developer could come up with a way to produce such a tool. Flat files in Playground are given the .pfp extension—and files with that extension will be auto-loaded by Play-ground from the assets root, user: root, or common: root folders, so be sure not to name your files with a .pfp extension unless they really are Playground pfpack files.

2.25 Creating Your Own Flat File

The way you compress your folder is to run the included pfpack.exe tool on the folder:

pfpack assets assets.pfp

This will create a file, assets.pfp, that you can drop into your assets folder instead of all the files you currently include there. But there’s a catch: Two specific files, strings.xml and settings.xml, can’t be included in assets.pfp. In addition, Flash can’t read any .swf file from assets.pfp. So you’ll need to create a folder with those files (strings.xml, settings.xml,∗.swf) excluded, pack that folder, then reintroduce those files to the combined folder. Additionally, strings.xml and settings.xml need to be read in before the flat files are initialized, so those files need to be excluded. As an example, a functional flat-file assets folder might look like this:

assets/assets.pfp

assets/strings.xml

assets/settings.xml

assets/splash/playfirst.swf

2.26 Appending Files

You can also use .pfp files to add new downloaded modules to your game. When Playground starts up, it scans the assets folder, the user: folder, and the common: folder for any files that end in .pfp. It then loads them inalphabetical order, with later files appending to overriding earlier files.This means that if you have a file game settings/levels.xml that appears in three .pfp files, the instance of gameset-tings/levels.xml that appears in the latest .pfp file alphabetically will be the one that the game sees. The folders are scanned in the order:

1. assets

2. user:

3. common:

So your game assets.pfp can be overridden by any .pfp files you download into user: or common:. Finally, anyfiles that physically exist in the assets folder will override any files with the same name in any of the pfp files thatare found in the assets folder; .pfp files found in the user: or common: folder will override filenames present int he assets folder (snipped)

If you have the SDK guys (headers, library binaries, and command line tools) you should be able to pack this stuff easily into these .pfp files. If anything, you might need to rewrite parts of code accessing sprites, etc. (which might not be a bad thing at all). Of course, this only applies to Playground 5.0.71.

So cheer up Chris! :-)

Edited by Bletchley_Geek
Fixed the mangling done by the clipboard on line breaks

Share this post


Link to post
Share on other sites

On the other hand, certainly this Playground thing has some truly :facepalm: stuff in its C++ Coding Standards:

You SHOULD NOT use third-party libraries (other than those shipped with Playground) in your code.

You MUST get permission prior to using any third-party libraries in your game.

That's quite hamstringing, for starters. From what I gather from other "suggestions", seems like the code goes through a third-party pre-processor which probably inserts OS-dependant #include directives and conditional compilation... this kind of big scale macro magic is bad, especially when the code generated gets mixed up with the code one writes (Qt, after 15 years, has got it almost right).

Share this post


Link to post
Share on other sites

I have a suggestion.

Why not do the main updates through Desura, but drop any updated/patched files into dropbox.

That would make hotfixes so easy to acquire.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×