Designing RuneEngine V2

This is something I’ve been meaning to discuss wider scale, but haven’t taken the time to do so. This post is going to be all about the ideas and design theories I am using as features are added to RuneEngine V2. I am also going to do my best to explain why these theories are proving to work well and how they will improve experience for use of the engine later when it is more completed.


Interfaces and Polymorphism

One of the biggest concepts that have stuck hard in the new structure of the engine is strong use of Polymorphing topics and interfaces. This is extremely important for users who would plan to extend the engine and add objects without having to write crazy wrappers around my rendering and content systems. Almost every object in the engine implements an interface, and almost all of the internal systems use the interfaces and NOT the objects directly to handle core features. For example, all content reference objects are interface driven and apply specific rules that the rendering pipeline requires. So, the rendering pipeline only requires the interface and proper implementation. Therefore, this can be extended very easily so long as the interfaces are implemented correctly. The most recent IO systems are nearly 100% interface driven, and probably show the extensibility of RuneEngine at it’s peak. The physics system exposes no objects, only interfaces and attaches to physics engine wrappers, so to add a new custom engine is trivial(well as far as making it work with RuneEngine goes).

As you can see those are some major area’s and they are very simple to plug into the existing API. This makes my life especially easier, but other’s using the engine could gain the same benefit as these interfaces are not internal and can be used by anyone.

Key reasons for the high focus here are obviously for extensibility and generic behaviors. This benefits mostly programmers, but that high benefit could allow a team to do much more with RuneEngine than what I allow them to.

Internal Code Cleanup/Optimization

This is a huge reason I started restructuring the engine. My goals for what I wanted it to be changed a few times during the first structure so the internals were getting haggard and the calling conventions were inconsistent.

So what are the new guidelines and why? First a little history of what it kept shaping around originally. When I first started the engine it was designed to be built like C++, well I took that a little too seriously. So, I backed off that and went way the other direction for a while. The object oriented model has always been very important through development of this, but some pieces were being written poorly, and as the ideas changed the legacy code was not getting removed. This made the code base fairly hard to work in as there was a mix of all sorts of styles. Before I started the restructure I gained an even better handle over C++ development and stay well practiced with it building many prototypes first in C++. The C++ iteration was started for RuneEngine. This is when I saw how poorly they were going to correlate. Some parts would work fine while others would need full rewrites. Therefore, the engine began restructuring. I use the term restructuring because I have not rewritten much of the internal code, just optimized and sorted it better. Much of it has not changed aside from areas where the objects used were changed greatly.

Ok, so let’s focus more on what has happened internally now. Originally there were a lot of methods calling methods and deeper callstacks, well now unless completely necessary every method is completely self maintained and does not internally call upon another. I’ve made my method argument lists more appropriate for C++ porting. However, I have also added higher level overloads which do not look as much like C++. Naming conventions have been held true, and many fields are marked internal now. Using statements have been cleaned up(this can actually be a huge optimization). All of this has made the performance better and the cleanliness much greater.

Exposure of High Level and Low Level methods

Before I only really exposed high level methods to access features. Although, I did have internal methods to do much of the low level portion as well. The biggest change here is that the low-level methods are now public and cleaned up. This also gets back into extensibility of the engine. If I don’t expose these methods then users of the engine will still have to handle things the XNA way. My methods just make that process a little simpler, even the low level ones.


IO vs Content Pipeline

This is a big change more so than original design at all. Originally there was a high focus on the IO system to load files which were unique to RuneEngine. Well, to adhere better to XNA and what XNA developers are used to, I am extending the ContentPipeline for many proprietary formats. Of course the C++ iteration will need some major differences here, but it should be manageable. So, pretty much all Rune specific content types are extended in the pipeline. And therefore could be used without any feature in RuneEngine at all.


Compartmentalized Design

This has been the plan since day one. Each DLL has been designed to be completely self maintained and features are being better designed in this new structure to hold true that they can work without the rest of the engine. I do this by writing majority of the objects completely independent from the rest of the engine. I had made some mistakes in the original structure, primarily the content systems. The content was reliant on every other part there, and the Graphics wrappers I had built as well. Now they’re self maintained and as stated above can be loaded through the content pipeline and be used with normal XNA projects.


Platform Target considerations

This part is simple, I’ve focused towards XBOX360, because it has more restrictions than PC and it’s not a mobile device. Not to hark on mobile, but just because market studies show it as the better revenue stream doesn’t mean it’s my primary focus. I prefer to focus on maximum quality I’m capable for. I’m not going to downgrade the output by adhering to mobile. This is a personal stubbornness of mine and I admit that, but I am NOT ignoring mobile. Just I will build the engine focused against everything else and make a mobile build later. Adhering to 360 and REACH profiles are going to keep me closer to what mobile will expect, so I am genuinely not concerned.


Development Tools

There is a fairly large concern here as well, but none of this matters if the engine itself behind it is not backing it. So, I’ve rebuilt a lot of the editor and have a lot of plans for working new features into this side of the engine. There is also a strong feel for extensibility here.




That about wraps it all up, there is not much more to point out at this time. Of course there has been a lot of time spent some days determining what a method should be called, and where it should be located. I didn’t think any of that detail was that important ;-D. Anyways, until next time…

Project Organization and File Structure

I was just thinking on this and realized that I preach about
this to everyone, all my student, anyone I get into dialog with about
programming development. This is something
I’ve gotten better about over the years and have also seen how big of a problem
it can be not doing so. ORGANIZE YOUR
FILES!!!!! Make a bulletproof file
structure. I can’t stress that any more
than I am right now. If you do not get
your files organized in a way you can find things quickly, you will hurt
yourself later.

Now, I don’t believe I have taken the time to give some big
tips and tricks that have worked before for me on a regular basis, so I’m going
to put them up here now. When I create a
new project I generally spend about 2-3 hours putting it together BEFORE
writing a single line of code. You may
say “WOW that’s a lot of time for preparation,” but I say 2-3 hours on a
project that you’re going to spend more than a week on… that 2-3 hours is
nothing. And if you didn’t do it I
wouldn’t be far from promising it would take 2 weeks instead of 1.

So, what I start a new project what do I do? First of all first and foremost even before
starting VS. I create a master directory
for the project. Typically on the root C
drive of my PC at home, if I had a D partition it would probably be there. For example RuneEngine V2’s master directory
is located at “C:\RUNE2.” Now what goes
in the master directory? The solution
right? NO!!! I stress that the solution
is not created here because with the rest I’m going to cover creating backups
would be nearly impossible or an extremely painful process otherwise.

The next folder I create is a child folder of the Master
directory “Bin”. This is where ALL my
projects output goes. In C:\RUNE2\Bin I have these sub directories:

REACH
HiDef
XBOX360
Mobile
DOCS (Yes I actually build documentation, it would not be
inappropriate for this go to outside of the bin in the master directory if you
prefer)
Editor
EditorAPI

As you may see above, these are all the platform targets,
not debug and release like VS typically creates. This is entirely intentional; my
Debug/Release folders are contained in these.
And this is where my projects target their output. This is why we do NOT put the solution in the
Master directory. When we create the
projects and configure their target paths we do not have to redo this for any
backups we generate. Also, Source
control is easier to… well control this way.

So, where are the projects created? They’re created in Subdirectories of the
Master Directory. The current working
solution is contained in C:\RUNE2\RuneEngine_2D_MAIN, but there are 4 other
backups and iterations of the engine there as well as the initiation of the
Native code etc etc blah blah. The
Master Directory contains anything and everything related to the project in
this setup, and it isn’t burrowed in C:\Users\Me\Documents\Visual Studio
2010\Projects….. Yep, not my favorite place to put things.

A note on source control, IF YOU ARE NOT CAREFUL YOUR
PROJECT CAN BE CORRUPTED, hence why I have multiple backups at all times. Usually it only gets corrupted if you’re using
it incorrectly, but better safe than sorry with large projects.

A note about creation of code files in the solution explorer
and handling folders. In C# you’ve
probably noticed that it adds namespace when you add to the folder. And sometimes you don’t want that. Of course you can just delete it and fix it
that way, but my tip is to just always add files in the namespace you desire
and if you want to move the file somewhere else to be more organized, it’s a
simple drag operation.

I hope this is a good read for other developers out there,
everyone has their own styles of handling files and I’ve tried quite a few
variations and this is the way I plan to stick to for a while. Below is a pseudo copy of my file structure
in RuneEngine v2’s project directories:

C:\
  RUNE2 – Master
    Bin
      REACH (Debug/Release)
      HiDef (Debug/Release)
      XBOX360 (Debug/Release)
      Mobile (Debug/Release)
      DOCS
      Editor (Debug/Release)
      EditorAPI (Debug/Release) – FYI I’m going to be
talking A LOT about this soon.
    Lib
    Includes (Lib and
includes are for the native build ignore them if you’re developing fully
managed)
    RuneEngine_2D_MAIN
(There are
actually multiple solutions here which load different combinations of the
projects)
      RuneEngine
      RuneEngine.Graphics
      RuneEngine.Content
      RuneEngine.Physics
      RuneEngine.ContentExtensions
      RuneEngine.Math
      RuneEngine.Scene
      RuneEngine.Threading
      RuneEngine.Components
    RuneEngineOLD2
    RuneEngineOLD1
    RuneEngineBACKUP
    RENativeDX

I wanted to make one last modification to this, though I suggest this methodology highly if it doesn't work for you don't do it. This ulimatley is a reminder of how important this is and a suggestion in what works for me. It's mostly about what you benefit from most. -Happy coding

DevConnections 2012 - Las Vegas

So, I was sent off to the DevConnections conference in Las Vegas recently. I was there as an exhibitor, but I also had a full attendee pass and was told to attend sessions as much as I could while there. There was a lot to learn there specifically on Metro and HTML 5. I'm writing now to summarize some of the things that were key points accross the sessions that I attended.

New C++ Features

I'm actually going to start with the new features of C++ in Visual Studio 11 as it is the shortest and probably in my opinion the most important.

One of the biggest surprises here was Microsofts statement regarding the state of C++ in software development today. IT'S BACK! That's right, as far as Microsoft claims to be seeing the interest and desire to work in C++ is growing again. It was thought for quite some time computers were surpassing the software we were putting on them so there was no need to go through the extra effort of C++/Native development.

The biggest things for me and I assume many developers with an interest and desire to work in C++ that are very good to hear are these two things:

• Intellisense has greatly improved
• Types are now color coded

I know these are not as big of a deal as they seem, but for many years C++ has been a painful experience while these two features greatly reduce just how painful that is. If you have used C++ in the VS 11 Beta already you have probably seen this.

There are a few things that are both really great and a little weird at the same time. For example there are a few new keywords added, one specifically I am not in full agreement on is the "auto" keyword. If you're familiar with var in C# or a few other various languages it basically does the same thing. Auto does not require a type definition, it requires a specific template based instantiation, but does not works very similar to var as it does not specify a type. It does delete itself, so it behaves a lot like a managed memory allocation. I refuse to use automatic type resolution myself so this was not appealing to me personally, and chances are if I'm developing in C++, it is to get away from managed and this sounds too much like managed behavior, without the CLR.

The last big implementation is that XAML can now be used with C++ when using WinRT. Yes, that's right Metro apps can be developed with C++ as well if you didn't already know this.

Something else really neat that threw me off a bit, mainly because they did not show much of what they meant was that supposedly the new Libs for C++ are more DirectX friendly and allow writing 2D/3D graphics in standard C++ libraries. It seemed to me all they really did was include DirectX in the standard Libs/Headers and that's just all it really seemed to be, still cool, but I don't know if it was a HUGE thing for me.

WinRT / Metro

This is where things start getting fun. WinRT and Metro are "practically" the same. So, let that sink in a moment...

Ok, unlike .NET WinRT communicates directly with the Operating System, but is also exclusive to Windows 8 and will not be ported for compatibility in Windows 7, which I think most everyone is aware of. What exactly is WinRT though? It is a new API for development which is NATIVE not managed for development of Metro applications in Windows 8.

WinRT is actually Native COM to be more specific. The DLLs/Assemblies are compiled with CLI metadata the same as .NET and actually... the code you write will very much resemble .NET. Another big kicker is that many of the .NET 4.5 objects ARE WinRT objects. Just in .NET they hit the CLR BEFORE going to WinRT.

INFORMATION OVERLOADED, take a breath...

Ok, now given the fact that WinRT is Native COM what does this give us? Unlike .NET the "common language runtime" WinRT is ACTUALLY more so a common language runtime than .NET by a thousand times. WinRT can be used in C#, VB, C++, Javascript/HTML, and pretty much anything else that can access COM. Another big factor is that ALL WinRT is XAML ready, sooooo... C++/XAML applications are possible where they were not really before.

There are a few downers to this though, WPF/Silverlight developers are not going to be as happy as they originally may have thought. WPF and Silverlight use XAML and so does Metro, but as one of the speakers at DevConnections stated XAML needs to be stated for what it really is and not associated directly with WPF/Silverlight.

XAML as he puts it is a language for object instantiation. Yes, this is true, but there is more to it than that. It defines data bindings as well and some extra syntax for handling event mapping as well. Basically think XML with a twist.

That being said, XAML in Metro communicates with WinRT, while WPF and Silverlight communicated with .NET or the Silverlight specific binaries. The objects and properties may be different... Actually you will find that only the MAJORLY used pieces of each were ported exactly as they are. So, just because you know WPF or Silverlight does not mean it's going to be easy, but it is NOT as bad as some are making out to be. General development should be rather simple, but most Metro apps will need a rewrite anyway to comply with the proper paradigm for Metro.

HTML 5

I really didn't take as much from these sessions. This is mainly because all they were really saying was it's an idea, you can use polyfills to add support to browsers that don't have it. There was some mention of more specifics, but there was nothing too glamorous or interesting to me aside from a little more realization to what the big fuss is. Oh and Opera is the most feature filled browser at this time. HTML 5 is very friendly will cell phones.

LOOK UP MONDERNIZR, this site has a ton of useful polyfills and scripts for making HTML5 work everywhere, even IE6.

Aside from that, I really don't have a whole lot here.

Conclusion

Ultimately, this article is not nearly as glamorous as I wanted it to be, but it's hard to get too deep into the specifics without losing majority of the people who may read this. Either way, I hope there was something here that sparked interest enough to do some of your own research.

ThreadPool vs Custom Pool : MultiPipeThread

This is actually something I did research on a while back (6 months or so). The ThreadPool object in .NET is a fairly powerful concept which reuses existing threads registered to the pool and allocates new threads as necessary dependent on your machine specs. Like all High Level objects this has some overhead to it, but probably less than what most could create on their own with exactly the same behavior.

But, is all of that truly necessary? My concept was to build a threading system which allocates and starts threads at initialization. It stops them when they're not necessary, but doesn't destroy them. It works with a piping concept system. Basically, in my example I describe 4 pipes associated with the MultiPipeThread object. This object receives information from the 4 threads and handles them appropriately.

Okay, so unlike the .NET ThreadPool, these threads are instantiated in the beginning. They are started immediately and block themselves when their inactive. How do we use them though? Ultimately we create an interface which defines the process method of an item that our threading will handle. Of course this could potentially be a simple delegate, but in my implementation it is not.

When we push items towards the thread piping system, it collections them and places them in a queue. It can potentially be expanded to collection them in buckets to gain more control over order etc. Once there are threads available to accept the information of a given type, if that type is prepared and ready to be added, it pulls the bucket or range of collection matching this type. Adds them all at once. This is where we gain our advantage over the ThreadPool. In the thread pool, every delegate is pushed towards and individual thread, existing or not. In our implementation items are pushed in bulk collections. The thread stays active until that thread pipe is complete. When it is complete it is marked ready for information, and as long as the type this pipe is set to accept is available the process starts over.

Does the tests prove the concept? Well yeah, the concept defines creating something very similar but pushing items in larger collections so there is less thread synchronization. Aside from this they are closely the same object, but we have a lot more control with a custom concept. We know exactly what is being pushed where and have opportunities to control order.

The above diagram shows the classes that are implemented. I know there is little you can see from this vantage point, but these objects are not that complex and easily implemented with some knowledge of synchronizing thread logic in .NET.

My favorite example of this object in use, is like so:

I have different phases of logic that need to be processed, some which can be overlapping some which cannot. Lets say 3 types, 2 of which can overlap and one that cannot. But the one which does not overlap must be first. So we mark it for priority and type ALONE. All pipes are marked to availability to accept ALONE, and we also add types GROUP1 and GROUP2, GROUP1 is allocated to pipes 1 and 2, where GROUP2 is allocated to pipes 3 and 4. Now we push a collection of items containing a mix of all. All objects of type ALONE fill up all 4 pipes immediately. Even though a pipe may open up before completion of ALONE, we restrict groups 1 and 2 from pushing in until all items currently in the pipe system complete. Once all is complete items in GROUP1 push to pipes 1 and 2 equally, and GROUP2 pushes to pipes 3 and 4. So they process simultaneously. Let's say we had a GROUP3 which could process at the same time as GROUP2, but not GROUP1. If GROUP2 finishes first it will still wait for GROUP1 to finish, however if GROUP1 finished first GROUP3 would allocated into the available pipes. We could even mark pipes 1-3 for GROUP3, so if pipe 3 is completed even though it's used by GROUP2, items from GROUP3 will go there.

This control is something we don't have with the ThreadPool, and even if we built a wrapper, we would not gain the same prowess we gain from implementing our own similar but different system.

Polymorphism: Structs, Interfaces, and Huh?

Ok, I know odd title, but I promise this will all get explained.

First, let’s start with some definitions of these concepts.

Polymorphism-

I like the Wiki version of this definition so here’s the link: http://en.wikipedia.org/wiki/Polymorphism_in_object-oriented_programming

For those who prefer not to click links and stay here, I will explain in a 1000 foot overview way. Basically in object-oriented programming, this is the ability to create a value, variable, function, or object that has more than one form. Shapeshifting so to speak, just as it would mean if not referring to coding. WOW! Simple enough right?

Structs-

MSDN does a fairly good job describing this:

http://msdn.microsoft.com/en-us/library/saxz13w4.aspx

Here, a struct is essentially the same as a class but it’s a value type. Just a little more limited, but can be less expensive, especially when using smaller objects that may be contained in large collections for example.

To get deep with it, a class is allocated in memory and allocates pointers (IntPtr) to the object wherever you place a field for reference type objects, classes.

However, structs, or value types, are allocated where you place them and do not generate pointers. This does mean they will copy and become unique allocations moving through the call stack and around the different scopes of your application, this can make them both easier and more difficult to control.

Interfaces-

MSDN also does their job well here:

http://msdn.microsoft.com/en-us/library/ms173156.aspx

Interfaces can be a complex topic for some, but to try to sum it up as simple as possible. They contain a reference to an object which implements the same methods, properties events that the interface defines. If you’ve dealt with C++ these will look a lot like class definitions in header files, and ALMOST behave like them. Interfaces cannot be instantiated and cannot define code implementation for it’s members.

Alright, Polymorphism in general is a huge topic of coding, so we’re not getting too deep, just showing one AMAZING example, though it may be hard to see all the benefits without trying it yourself.

If you’ve dealt with structs before you’ll know a few things about them, such as the inability to use inheritance, and the frustrating ref keyword when trying to pass them into methods as a reference type instead. So, their benefits typically get outweighed by the aggravations of these things. Here’s a tidbit that a lot of people forget, though structs cannot inherit they can implement an interface. It gets better though. Interfaces can inherit other interfaces….

Due to the two above facts, we can use polymorphism techniques to build a collection of structs which inherit each other, and can be passed as reference types without the ref keyword.

My example, which I’ve included the class diagram for, uses the following structs and interfaces:

• IVector
• IVector2
• IVector3
• IVector4
• Vector2
• Vector3
• Vector4

Since the structs, the one’s without the ‘I’, cannot inherit each other all the inheritance tree is in the IVector-IVector4, and it just goes incrementally.

Ok, got that part, but what does this allow me to do?

void RandomMethod( IVector aV1 ) { }

The above method declaration accepts IVector, so what can be passed into this method? This method can actually accept ANY Vector type as it’s interface description. It could use IVector.AxisCount to determine which one was actually passed and handle it appropriately.

That’s one use, which is pretty cool. What about this one?

Vector4 v = new Vector4( 1,1,1,1 );

IVector2 v2 = v;

Will this even work? YES, this just gave me a reference to the Vecto4, the object is obviously still the same object, but rather than converting the value, we made it accessible as a Vector2 interface. Of course you would need methods that accept the interface instead of the struct to use it. Hence why IVector2.ToVector2() exists. This is a simple convert, where if the internal object is not a Vector2, it converts it before returning it.

I know this is a lot to take in so I will stop here. Have fun!!!

Introduction

Hello All,

I just wanted to take a moment to explain why I’m even creating this blog before making my first real posting. This blog I am going to use to pull all of my development activity into a single location.

Those who know me already and have been watching some of my other posts on the RuneEngine v2 Facebook Page should know I post almost everything there as it is. However, there are things I reserve posting there as they are unrelated to the RuneEngine project or any of it’s inner workings.

So, basically anything I post there or anywhere else, will likely end up here as well.

A little about me though. My name is Danny Helms and I am a full time support agent for an Imaging SDK provider in Charlotte, NC and a part time faculty for CPCC SGD Department, as a programming instructor. I have been coding since I was 8 years old, so been a pretty long time now. One thing you may see as I write is that no matter how long I’ve been developing or how far I go, I’m always learning new things. It’s not only because the field changes and advances on a daily basis, but also because it’s already so vast it’s hard not to come across new things all the time.

You may see me mention use of languages in:

• -C#
• C++
• WPF
• Silverlight
• SOME VB.NET
• Javascript
• May start some Java soon
• ASP.NET is also something I will likely piddle with in near future.

Common SDKs I use:

• Microsoft’s XNA Framework
• WeiFenLou Docking API (May not use this as much as I am moving further into WPF development)
• BEPU Physics Engine

Current Projects:

RuneEngine V2 Projects----

• RuneEditor.API (Simple API for addons to the RuneEditor application)
• RuneEngine (2D)
• RuneEngine (Native)
• RuneEngine (Reach, Mobile friendly)
• RuneEngine (HiDef, the core project and most of my efforts go here)

Other Projects---

• BitbucketReporting.API ( Utility API for working with Bitbucket's WCF API, included in RuneEditor.API )