Technical

How to shrink down a git(hub) repository

Starting point

With my Vulkan C++ example github repository approaching 200 MB in size I decided it was about time to shrink it down to a reasonable size again. Shrinking a git(hub) repository isn’t just about deleting locally present files but requires cleaning up the history as files that have been removed are still present in the repository’s history and therefore still contribute to it’s size.

A big chunk of the repo’s size is caused by binary assets like textures and 3d models. When I started out with my Vulkan example there were only a few assets so I just added them to the repository. In hindsight this was the wrong decision, so one of my primary goals was to remove all those assets from the repository and it’s history. I already stopped adding assets while I did some examples using HDR textures and moved them into a separate asset pack that needs to be downloaded to actually run these examples. After removing the assets I’ll no longer add any of them to the repo but rather put them into the separate asset pack.

So in this article I’ll try to describe how to shrink down a long running repository without having to recreate it. For my Vulkan examples this resulted in a much smaller repository that’s a lot faster to clone.
Continue reading

Vulkan from the POV of a hobby 3D developer

68747470733a2f2f7777772e6b68726f6e6f732e6f72672f6173736574732f75706c6f6164732f617069732f67646673676b6a6834353574342f676c6e6578742d6c6f676f2e706e67

 

As there have been lots of new information on Vulkan, Khronos‘ new graphics and compute API I decided to do a little write up of the new API from a hobby 3D developer’s point of view.

Although I’ve been writing games, demos and applications with OpenGL for roughly 15 years now I still consider myself a hobby developer in terms of 3D graphics. My job is not depending on pushing pixels, maxing out draw calls or swizzling shader commands, and most of my work on 3D (primarily OpenGL) is done during my spare time (mostly late at night).

vulkanScene

During this years SIGGRAPH and GamesCom, Khronos showed off some stuff related to Vulkan (LunarG e.g. uploaded a video of LunarXChange, the developer platform for Vulkan) and several IHVs (e.g. PowerVR) and ISVs demoed applications and implementations.

And while it’s still a way to go before Vulkan is released to the public (should be by the end of the years) I’d like to write down a few words on the new API for hobby (OpenGL) developers that may be uncertain on whether to switch or not, or even on what Vulkan actually is.

 

Continue reading

Status update

Just as the last blog post is starting to collect dust (time flies by oO) :

I’m still doing lots of 3D development during the late hours in my spare time, still with C++ (Visual Studio rocks 🙂 ), but most of that development is done under an NDA so that’s the main reason I haven’t been updating this blog lately.

Other than that I’ve also been working on a dungeon crawler prototype using modern C++ (C++11/14) and modern OpenGL (4.4). Not much to see yet, but the first screenshot shows the latest version with omni-directional shadow maps, randomly generated dungeons, normal mapping and 3D models (using ASSIMP). I’d also like to mention G-Truc‘s excellent glm and gli libraries that I’m both using in this (and other projects).
I’ve been playing around with randomly generated dungeons a few times in the past but never got much further than some simple tech demos, so this may never become a game but rather a playground, though it already looks pretty nice.

2015-07-07 22_46_38-Dungeon Crawler Prototype - (c) 2015 by Sascha Willems 2015-06-17 21_48_05-Dungeon Crawler Prototype - (c) 2015 by Sascha Willems 2015-06-12 23_29_57-Dungeon Crawler Prototype - (c) 2015 by Sascha Willems 2015-06-05 23_07_32-Dungeon Crawler Prototype - (c) 2015 by Sascha Willems

 

So right now there’s not much new stuff on OpenGL (ES) or WebGL, but you may check out my twitter account for screenshot and ramblings on my current 3D endeavors without me getting into details. Most of the stuff I post there is not worthy of a complete blog post, so for quick updates, twitter pretty much replaced my blog.

 

Javascript repository

Following Java (on Android) and C++ (on Windows), I just released my first Javascript sources over at my bitbucket repository. My first (ever) public JavaScript demo is the random dungeon generator that I wrote an article about (a long time ago).

It generates rarnddungeon06ndom dungeons of different sizes and can be tested directly in your browser over here.
Included are the JavaScript sources for the random dungeon generator that use HTML5 for drawing a simple representation of the randomly generated rooms and corridors.

My plan now is to add some more randomness, some game-related features and to write a WebGL-renderer, so you can walk the dungeons in your browser, with maybe a game to follow.

From Delphi, Windows, OpenGL and Newton to…

box2dtest_01 …Java , Android, OpenGL ES and Box2D. That’s what I’ve been doing the last few days during my (currently rather rare) coding sessions. I wrote a simple physics playground using JBox2D for my phone that allows me to drop different objects by the touch of a finger and also uses the orientation sensor to change the gravity vectors. Putting lots of dynamic objects inside a confined box, and seeing them fall over by just tilting the phone is a fun affair 😉

I also wrote a (very simple) OpenGL ES renderer to display the physics bodies. Luckily OpenGL ES is pretty close to recent OpenGL versions (if you skip all the legacy immediate functions), and getting the renderer up-and-running only took a few hours.

And I actually don’t miss Delphi that much for my hobby development (Though I’m still earning my money with it), Java has come a long way, is a modern language that actually evolves (whereas Delphi, in terms of the language hasn’t changed a lot in the recent years) and writing quality code kinda seems easier, even though I’ve actually just started writing code with Java. I’m not yet feeling as comfortable with it as with Delphi, but writing Java code gets easier and easier by every line of code.

So for the next few weeks (or months) I plan on playing around with realtime physics on Android, and especially with all the different input methods and sensors that simply aren’t present on PCs. I’m still not sure about what type of game I’m gonna do for Android, but I have some initial ideas that I’d like to implement. Maybe I’ll even port Trugbild to android, just to get some more experience developing for mobile platforms using OpenGL ES.

 

New target : Android

I’ve been using a symbian “smartphone” (a Samsung SGH-i560) for ages now, but finally decided to swap it for a new and shiny Android device, mainly to code on.
So I recently bought myself a Huawei Ascend G510. It’s a middle class Android handset with a dual-core CPU (1,2GHz) and a 4.5″ screen that supports OpenGL ES 2.0 that doesn’t cost a fortune (~140€) and condidering the pricetag it’s a pretty good phone.
So once the battery was loaded I attached it to my PC and started to develop on it right away. I started off with the native FPC armv7-cross compiler using native Android Controls, went over to integrating Free Pascal (JVM) into Eclipse and did some basic OpenGL ES demos with both of them.

And though it may be appealing to code in pascal for the Android platform, I decided to go with Eclipse and Java. Java has a lot in common with Delphi, and getting used to it’s syntax only took me a few days and I’m already creating productive sources. Actually I kinda like coding with Java (and Eclipse). You get lots of modern language features that Delphi has been lacking for years, and the code features (auto completion, correcting problems, etc.) are just great. And best of all is that you get it for free, which is no comparsion to the expensive entry into the Android world with Delphi XE5 (which btw. forces you to use FireMonkey).

So after getting used to Java and Eclipse (in the form of ADT) I started work on an Android version of the glCapsViewer, called “glESCapsViewer”. It’s already up and running, getting the implementation details. It reads out information on the device, the renderer, all available extensions and all supported texture compression formats. Next on the line is generating XML files for the repots and then uploading them to my online database.

I’m not sure yet on wether to just add the mobile OpenGL ES reports to the current caps database, or to add a sparate database. But I guess I’ll go for the latter way, as it’ll allow for more flexibility. This would allow me to e.g. also add information on the device iteself and wouldn’t inflate the desktop report database with OpenGL ES versions and extensions.

It’s not quite ready yet, but expect an Android version of the glCapsViwer in the near future. And also expect to read lots of postings on Android and Java. Once the glESCapsViewer ist done, I plan on creating my first mobile game, and it may be a port of Trugbild, as the gameplay itself is perfect for touch based devices. I’m even thinking about using the motion sensors to move along the scenery 😉

Going multi-platform, part 6 : Mac OSX

After hard weeks of work, february finally saw the release of a linux version for Projekt Weltherrscher – Phase. This marked the first step on my journey to go multi-platform, something to too easy for a game that was made on and for windows (with an IDE that actually only runs on windows).

But those that are into multi-platform development know how close Mac OSX and linux are in terms of coding (OSX is based on Darwin which is based on unix), so after getting the game to run on linux the next (natural) step would have been porting it to Mac OSX. But different to linux you don’t just set up a virtual machine, install a free distro, your IDE and start porting, no, you need something that actually runs a Mac OSX operating system, and that something is ideally a real Mac. But now I finally have access to a running Mac OSX with all my development tools (lazarus, FPC, xcode) up and running.

So I used the last weekend to start porting the game to Mac OSX, and as I initially thought most of the stuff that makes the game run on linux also works on OSX. I had to comment out a few things that I still need to implement, like iterating display modes, changing to fullscreen mode, etc. but the game is up and running native on OSX for the first time!

So yes, Phase 2 is coming to Mac OSX some time in the not too distant future, and I hope that there are some Mac users out there that are up for some serious strategics 😉

The challenges of a game’s sound design

Usually I’d add some nice screenshots to show off progress on “Phase 2“, but this time it’s an area of game development that’s pretty hard to show off in screenshots :  Sound design.

I find this is one of the hardest parts as far as game design goes, at least for me. I do all the parts of game development on my own (design, code, content, etc.) but when it comes down to sound design, the result is usually so bad or bland that (I guess) most people just turn it off. I remember searching for and creating sounds for a shmup some years ago that took weeks, and when I heard the combined result of that effort running in-game I knew that I just had wasted a lot of time. It was just horrible. Though I don’t have any problems imagining how certain aspects of the game should look and feel, I usually don’t have a clue about “how” a game is actually supposed to sound.

As for the sound design for Projekt “W” – it’s pretty much non-existant. All the sounds in the current build are the same as on the first release of “Phase 1”, and these few sounds don’t add anything to the atmosphere and some of them (like the menu swoosh) are downright annoying. So during the last few weeks I played through some smaller indie games (mostly from IndieDB) and payed close attention to the sound design of these titles. I then quickly noticed that even a few simple GUI sounds can add a lot to the atmosphere of a game, and if you combine them with a fitting soundtrack (I’m still searching for new background tracks, more in the ambient vein) they quickly become a real bonus to the game’s atmosphere.

And since people think that the graphics are pretty good for a one-man hobby project I’d like to get the sound design to a similar level. So for the last few days I’ve been collecting and creating new samples for the user interface (with samples for actions, units, projects, etc. to follow) and even though I only changed and added half a dozen basic UI sounds (button clicks, menu transitions, confirmations, etc.) the sound design has already become a much closer fit for the game’s visuals.

So hopefully the  new sound design (that’ll at least partially come with the next release) will add even more atmosphere to the game!

And since I’m sure that a few (hobby) game deves read this : How do you handle sound design? Do you have the same troubles, or is there any easy way to get this part of game design right?

Going multi-platform, part 5 : Finishing touches

phase-2-linux After weeks of hard work and learning a lot about the differences between coding on windows and linux, the current source revision of Phase 2 is completly functional on linux and playing the same as a windows version. So if you’d put them side-by-side in fullscreen mode you wouldn’t notice any differences!

Currently I’m putting the finishing toches on the linux release and only a very few minor things have to be fixed and adjusted before I’ll release the linux version into the wild, so you guys can play with it and provide me with feedback, the linux world is far more fractured than the windows world, so I’d love to know how the game runs on different distributions.

Since the last posting I did the following for the linux source path of the game :

  • Switched to the BASS audio system
    In my last posting I wrote about the problems finding an fmod linux library that would work with my game and free pascal. The problem here is that delphi/fpc support has long been dropped from fmod, and since a lot of declarations have changed, using the last delphi/fpc header with a recent fmod library wouldn’t work. So I switched over to BASS, a sound library that I already used several years ago. It still supports delphi/fpc, is leightweight, updated on a regular basis, free for non-commercial use and supports many platforms. You can get it from un4seen.
    Since my different sound systems (fmod, OpenAL) are all abstracted via TSoundSystem and TMusicPlayer classes, adding in BASS only took me 15 minutes, so the game is now running on a different sound API and the linux version now also plays music and samples.
  • Added resolution selection / switchting
    This was a lot harder than I intially thought. For windows this is pretty simple (as I’ve been using it for years), but for linux I had to do a lot of research before I started implementing it. In the end I went with xrandr and you can now change resolutions and windowed / full screen on linux like on windows.
  • Implemented performance fixes
    I hinted at this in one of my recent postings. There are several performance differences between free pascal and delphi, so code that ran pretty fast on delphi made the game crawl on free pascal. One example was the transfer market. On delphi I had all texture names for all available staffer picutes in a stringlist and on drawing each staffer entry in the selection listbox, I looked up the texture ID of the staffer’s face via this stringlist. If that image was already loaded into VRAM I draw it, if not it get’s loaded from the virtual file system. On free pascal that code made the game go down below 10 fps when opening up the transfer market window. So now I load up all staffer pictures when loading the game (only takes a few ms) and directly assing the texture ID to a staffer. Sicne this is faster in general, this will also make the delphi version run faster.

So the linux version is looking fine, and I hope to have a public release ready within the next two weeks, so the linux crowd is finally able to play my game outside of wine or virtual machines. And though it was a hard piece of work (partially making me wonder if it was worth all the hours of coding) I think it was worth it as I learned a lot about coding on linux. So for my future projects I’ll take multi platform support into account from the very beginning.

Going multi-platform, part 3 : First throwback

Last weekend I worked on getting the lazarus / free pascal build of “Phase 2” to look and play exactly the same as the delphi build. I had to change some small things here and there, but after a few hours you can now no longer distinguish between the two builds, maybe except for performance. It looks that some stuff that’s pretty fast with delphi is kinda slow with fpc, for example accessing stringlists and stuff. But that’s nothing that can’t be easily optimized, so the first goal of getting the different compiler builds on par is done. This also means that the game is fully playable under linux.

But during my playtestings on linux (I ditched my ubuntu installation with it’s horrible ubuntu one interface btw., and now use mint) I noticed that all descriptions of nations, units, technologies, etc. were missng, as well as the whole credits text. And after hours of debugging, checking encodings, creating test applications, etc. I found out that NativeXml is not able to load text contained within XML nodes on linux. So a node like this :

[xml]<unit name="test">
<description>This is a test description</description>
</owner>[/xml]

Will return an empty node value on Linux. No matter what I tried, accessing it directly, via the first childnode, etc. it was always empty. At first I thought I’d be doing it wrong (though it worked fine on windows), but then I did a very simple test that failed : I loaded up the XML with NativeXml and directly saved it to disk afterwards. And guess what? All texts within the xml nodes were removed by the library!

So it seems that I can’t use NativeXml for linux due to this severe bug. Their forums seem dead (registered there the weekend, but haven’t even been unlocked there so I couldn’t post the problem), and finding people that use it for linux is almost impossible. So all the thousands of lines of code I replaced to get the xml stuff to work on windows were just a waste of time, and I’ll have to remove this library again. I took a look at other xml libraries, but most either only supported delphi or were too bloated (dozens of units, huge source files, different dependencies, etc.). But well, that’s part of the learning process that makes programming so much fun (and sometimes so frustrating), and for the next external library I’m going to use (I prefer to write most of the stuff myself anyway) I’ll first check and see if it works on all my intented platforms.

And what now? Is the linux port in danger? No, rest assured, it’s not! Instead of relying on an external library for loading and saving xml I have written my own xmlwrapper that abstracts free pascal’s and delphi’s different xml interfaces into a single structure that I can use in my game (and coming projects). So now I load up an xml document via my own wrapper and compiler directives control wether delphi’s xml interface or free pascal’s DOM is used to fill this dynamic xml structure from the given file. The syntax is leaned towards the style of delphi’s xml implementation and parts of the game have already been changed to use my new xmlwrapper. So in my code it looks like this to get values from the xmlwrapper :

[delphi]VFS.LoadXML(‘data\xml\nationinfo.xml’, TmpXML);
for i := 0 to High(Nation) do
begin
Nation[i].History := TmpXML.Root.NodesByIndex[i].Nodes[‘history’].NodeValue;
for j := 0 to LocalizationDB.LangID.Count-1 do
Nation[i].Name[j] := TmpXML.Root.NodesByIndex[i].Attributes[‘name_’+LowerCase(LocalizationDB.LangID[j])].AsString;
Nation[i].Stronghold := TmpXML.Root.NodesByIndex[i].Attributes[‘stronghold’].AsInteger;
with TmpXML.Root.NodesByIndex[i].Nodes[‘basefactor’] do
begin
Nation[i].BaseFactor.Growth := Attributes[‘growth’].AsSingle;
Nation[i].BaseFactor.Loyality := Attributes[‘loyality’].AsSingle;
Nation[i].BaseFactor.Ressources := Attributes[‘ressources’].AsSingle;

[/delphi]

So much for an open source xml library that was supposed to work with linux. So I guess this weekend will pretty much look like the second last one with me replacing lots of lines of xml-related code 🙁