Some math about that "digital interface to rule them all"

I remarked in my future OS post that we should standardize on one digital interface format for everything digital.  One way to solve the most pervasive problem in that domain (that the bandwidth use cases of devices differs quite a bit) can be solved with dynamic frequency clocking of the interface buses.  An easy way to solve this is to have a hot-plug handshake between devices - when a connection is made, each side sends standardized information about itself (what it does, what frequencies its capable of, etc) and the interface uses the lowest frequency both support that sufficiently fills bandwidth requirements of the interconnect.  So you could in theory have a multi-gigahertz interconnect where low bandwidth devices (like input devices) could run at only a few hundred hertz.

Some general numbers:

The most bandwidth intensive activity I can conceptualize for this interconnect, besides just going all out maximizing the bandwidth for usage as a supercomputer interconnect, is about 10 GB/s or 80 gb/s.  I get this number using 16:10 Ultra-HD at 48 bit color depth (12 bits per color channel + 12 bit alpha channel) at 180hz (this comes from the idea of using 3d displays - 90hz is a good standard that most human eyes can't differentiate very well, just like 4k is a good resolution where most eyes won't see the difference at 250 PPI from a distance of 2 feet - I wouldn't ever use a 3d display, but the standard should anticipate that).  Given that current Displayport can reach 18gb/s, increasing that 5 fold in time for this "concept" to actually matter is completely feasible.  Worst case scenario, you just include more packet channels.  Comparatively, we have pretty much clobbered the limits of audio and codecs like Opus just make compression even better over time, so I'm not worried about 15 speaker surround sound being the next big thing.

But just as a consideration, if you were to go completely audio-phile crazy and used 20 mbit/s lossless audio on 33 speakers.  That still only constitutes 660 mbit, less than a gigabit.  In practice, if you are transferring raw video without some kind of encoding compression, it will be the absolute dominator in bandwidth utilization.

So a target of 10GB/s sounds good, especially considering that also works as an interface bandwidth target per lane inside the chip architecture.  If running at peak throughput, you would be effectively running a cpu-northbridge interconnect over a wire.

If we use fiber channel for this connector standard, it can be the ultra-low-latency needed to be as multi-purpose as it needs to be.  However, while it could be ultra-high bandwidth with low latency when needed, you could also disable all but one data channel and run it at a few hundred hertz for a few megabytes of bandwidth, which is also a keyboard or mouse should need.

I could also see color coded cables indicating the peak frequency on the line supported - a 100' cable probably wouldn't be able to run at whatever frequency is needed to supply 10GB/s without active modulation, whereas a 20' cable should be able to.  This also means it makes a good ethernet standard, because realistic network bandwidth won't pass a gigabit for a long while, and getting a gigabit on a multiplexed cable like this will be cakewalk at really low frequencies.

I really don't even want to consider analog interfaces with this configuration.  It should be all digital video, audio, etc.  You could always stick an analog converter chip on top of this interface like a traditional PCI device anyway.

I also would only want one standard connector, preferrably as small as possible, with an optional lock mechanism.  Just from my personal view on the matter, but having 3 standards of usb, micro usb, hdmi, mini hdmi, micro hdmi, mini displayport, displayport, etc is just absurd.  If the maximum connectivity can be obtained on a smaller connector foot the extra few cents to build a better circuit.

Just as a footnote, real world 100gbit Ethernet should eventually hit market, and that would be the perfect standard for this.

Software Rants 6: How to Reinvent the Wheel of Servicing

In my post about new OS paradigms, I remarked about how you won't replace IP as *the* network protcol and how we should all just bow down to its glory. 

However, one thing that can easily change (and does, all the time, multiple times a week) is change how we interact over that protocol.  It is why we even have URI's, and we have everything from ftp:// to http:// to steam:// protoctols over IP packets.  I want to bring up some paralells I see between this behavior, "classic" oprating system metaphors, and the relatively modern concept of treating everything as files circa Plan 9 and my stupid ramblings. 

If I was writing an application for a popular computing platform, I would be using a system call interface into the operating system, some kind of message bus service (like dbus) for communicating with most service, I would personally use some kind of temp file as an interchange but I could also open a Unix socket as a means of IPC.  Or maybe I go really crazy and start using OS primitives to share memory pages.  Any way you slice it, you are effectively picking and choosing protocols - be it the Unix socket "protocol", the system call "protocol", etc.  In Plan 9 / crazy peoples world, you forgo having protocols in favor of a a file system, where you can access sockets, system calls, memory pages, etc as files.  You use a directory tree structure rather than distinct programmatic syntaxes to interface with things, and the generic nature improves the interchangeability, ease of learning, and in some cases it can be a performance gain since you are using significantly less overhead in using a kernel VFS manager to handle the abstractions.

If I took this concept to the net, I wouldn't want to specify a protocol in an address.  I would want the protocols to be abstracted by a virtual file system, so in the same way I mentioned /net/Google.com/reader should resolve as the address of Google's reader, you could be more specific and try something like /net/Google.com:80/reader.https (this is a generic example using the classic network protocols) where you can be specific about how resources are opened (in the same way you use static file system typing to declare how to handle files).  But this treats Google.com as a file system in and of itself - and if you consider how we navigate most of these protcols, we end up treating them as virtual file servers all the same.  The differentiation is in how we treat the server as a whole.

In current usage, interacting with ftp://mozilla.org and http://mozilla.org produces completely different results, because ftp requests are redirected to an ftp server and http ones are directed to an http server.  But https doesn't inherently mean use a different server, because it just means sticking a TLS layer on top of the communication, but the underlying behavior of either end resolves the same - packets from one are generated, boxed in an encrypted container, shipped, decrypted on the receiving end, and then processed all the same.  That is in many ways more elegant than the non-transparent designation of what server process to interact with at an address based solely off the URI designation.

So what I would rather see is, in keeping with that VFS model, a virtual mount of a remote server under syntax like /net/Google.com producing a directory containing https, ftp, mail, jabber, etc, where an application would be able to easily just mount a remote server, and depending on the visible folders derive supported operations just off that.

Likewise, authentication becomes important.  /net/zanny@google.com would be expected to produce (with an authentication token, be it a cached key or a password) *my* view of this server, in the same way users and applications would get different views of a virtual file system. 

This leads to a much more cleaner distinction of tasks, because in the current web paradigm, you usually have a kernel IP stack managing inbound packets on ports, where to send them, and such.  You register Apache on port 80 and 443 and then it decodes the packets received on those ports (which now sounds even more redundant, because you are using a url specifier and a port, but the problem becomes that ports are not nearly as clear as protocols).

So in a vfs network filesystem, determining the available protocols on a webserver should be simpler, by just looking at a top level directory of the public user on that server, instead of querying a bunch of protocols for responses.  Be it via file extensions or ports, it would still be an improvement.

Lifes Too Short, or why using anything but C++ is insulting to your users

I was watching a presentation from altdevblog about C# for games (I hope it takes the gaming world by storm, C# is beautiful).  Repeatedly, when discussing "hard" concepts (callbacks, manual memory management, threading) the author (Miguel) would qualify the difficulty, and the expected debugging nightmares and resulting ripping out of hair of doing hard things in software with "lifes too short".  Really great talk about Mono in the game world, it is a really ingenious platform especially since it supports everything ever.

I think it is a dangerous sentiment.  It basically says do things the easy way because it is a mountain to climb.  While I will agree on the intent, that developers should trade efficiency and determinism for time and ease of development.  Which is great, especially in games.  Maybe not in services.  And definitely not in kernels.  It is not a black and white case of just use the high level language to save time, it is a debate between the difficulties of building a project and the amount of work it is expected to perform.

This actually applies to games.  A single player without multiplayer can reasonably just throw C# at the problem and crank the game out quickly.  If it only lasts 30 - 40 hours, in a computing environments lifetime, that is peanuts.  But when you build a game meant to last hundreds of hours, you are actually not just costing cycles on your users machines, but you are wasting their time in exchange for your own, and the resources to power the redundant cycles, and the time of those who produce the power to run the machines.  It isn't just about developer productivity, but about everyones optimal use of time.

So I look at opportune shortcuts as a question of cost - it is why I would almost always look at C++ when thinking of some mass market product, but I'd look at Python for anything niche.  And I'd look at C# to produce business software, or non-service continually running processes.  Everything has a use case and a cost, so just thinking about ones own life being short might be a short sighted justification for paying a performance cost.

Also, I think this was the most flamebait title I have written yet.  Yay.

Document and Serialization Format Thoughts and Examples

I made a few posts on reddit in a thread about Python in the browser about why HTML / xml suck and almost anything else is better. Assuming that, I explored some options I would like to describe here as alternative document syntaxes.

1. JSON-like whitespace-insignificant functional orientation:

jsondoc {
  title : "Ninjas Are Awesome",
  p(id:"fact") : {. : "A ninja's natural enemy is a",
                  strong : "PIRATE", "!"}
  ul(id:"ninja_weapons") : { li : "sword", 
                             li : "kung fu", 
                             li : "throwing star"}
}

This I feel has the most chance to catch on - or something like it. JSON-esque maps. The only real difference between standards JSON and this syntax is the introduction of arguments on keys that act as attributes. The map element . denotes the wildcard content of a tag if you use a map instead of value for a "tag". Alternately, you could derivative from JSON standard more, and that leads me to my next example.

2. YAML-like whitespace significant:

title :: Python
list ::
- article : id=1 :
    author :: Xanny
    date :: 2012-12-15
    title : id=pirates : Ninjas are awesome
    post : id=bacon,class=words : >
        I talked about this in another fork of this 
        comment thread a bit.  You absolutely need to 
        have the capabilities of the language...
- article : id=2 :

This syntax is YAML derived but introduces attributes between the key - value syntax.   It is a little verbose with all the double colons, and in my considerations of this syntax it became apparent you didn't need to have the : > syntax for multiline descendant text bodies, so I refined it with a few changes. > is a synonym for ::, and a value section that only contains a : is considered an indicator of a descendant multiline element.  Also, instead of using minus signs as an array element delimiter, I use commas instead, since it uses comma separated values in the arguments.  Also, a keyless argument is assumed to be the id, and arguments without quotes are ,>: terminated.

title > Python
list >
  , article : 1 :
      author > Xanny
      date > 2012-12-15
      title : pirates : Ninjas are awesome
      post : class = words, bacon >
        I talked about this in another fork of this 
        comment thread a bit.  You absolutely need to 
        have the capabilities of the language...
  , article : 2 :

I like this, but I can't feel it would fly very well, so I also have a whitespace insignificant dialect of this syntax:

doc {
title > Python,
  list > [
    article : 1 : {
      author > Xanny,
      date > 2012-12-15,
      title : pirates : Ninjas are awesome,
     post : class = words, bacon : I talked about this in another fork of this comment thread a bit.  You absolutely need to have the capabilities of the language...
  } ,
  article : 2 : ...
  ]
}

Here, commas are string and element delimiters everywhere, maps are curly braces denoted. Because of a concise grammar, you would only need to escape commas, colons, and the two kinds of braces. One thing to note is this language never utilizes parenthesis. I feel something like this might easily become more popular. An alternative might be to keep the argumentative behavior from the json dialect, and reintroduce parenthesis:

doc {
  title : Python,
  list : [
    article(1) : {
      author : Xanny,
      date : 2012-12-15,
      title(pirates) : Ninjas are awesome,
      post(class=words, bacon) : I talked about this in another fork of this comment thread a bit.  You absolutely need to have the capabilities of the language...
  } ,
  article(2) : ...
  ]
}

The big deal here is that by introducing overhead of parenthesis the key : value syntax remains succinct, and it easily allows for a data serialization format to be used where you just discard the arguments syntax, or maybe even have the parsing behavior definied that arguments are just key:value pairs to be added to the constructed map (here, in python syntax) like so:

title(pirates) : Ninjas are awesome, 
>>> 
"title" : {"id" : "pirates", "body" : "Ninjas are awesome"}

The same could work for the yaml syntax, where colons past the first are disregarded (except ::\w+\n which denotes multiline text follows). Or you could use a glyph exclusively for multiline text, like & and * which go unused in yaml).

It really comes back to that vision of one data format to rule them all (that isn't the current ruler, xml) for documents, serialization, message passing, etc. Both JSON and YAML are significantly better than xml, and in keeping with that unified protocol ideology, unified textual language is a natural extension.

As a footnote, I had to rewrite this blog in completely manual html since it kept malforming the code -> pargraph transistions and inserting redundant spaces with tons of unneeded tag duplication. So the source of this should be pretty. Come on Google, get your shizzle together.

And as a final note, I'd probably go with a choice between the last two. I'd easily see this better-markup-language (bml) have extensions .bmlw for the whitespace dependent version (better markup language (with) whitespace-significance) and .bmlb (better markup language (with) brace-significance) for the whitespace agnostic.

GPU Fan Controller Thingy Post-Mortem

Forgot to write a blog post about this.  Took about an hour of coding, and around 4 reading Python documentation on proper usage of the subprocess module.

It works, I guess that is the point.  I'm not going to try to integrate it as a service into upstart or systemd since it runs fine unprivileged as a startup application.  I would try to make a nice GUI for it but Kepler GPUs don't support fan speed control through coolbits anymore (go figure) so it seems user fan speed control is now unsupported by Nvidia, which defeats the purpose of the project.

It is easy to use though if anyone wants to grab it since most GPU default fan profiles are awful.  It is just a array of temps to percentage speeds that gets built into a temperature map and every recheck time (default 5 seconds) the script will poll for the temperature, index the temperature in the array of fan speeds, and if the speed it gets differs from the current one it will update the speed.

I worked to minimize calls into nvidia-settings, because it almost certainly has a lot more overhead than my little script to incite.  So I only change the fan speed when necessary.  I found that it made more sense to constantly turn fan speed control back on every update than to potentially silently lose the ability to change fan speed (or to implement a way to parse for errors on speed update, and then to turn it back on - that might work, but I didn't get conclusive evidence it will actually error out if fan speed controlling gets disabled in some circumstances so I didn't persue it further).

In the end, it works for purpose.  It is to my knowledge the most efficient implementation on the net (even though it has a lot of memory usage with a temperature array, but looking into trying to do an array by 5c changes in temperature the math just gets more complicated than finding an index in an array of temp : speed tuples that crosses a thermal boundary to use).  It doesn't have a neat GUI but I lost the drive to persue that when I found out Nvidia no longer supports fan control.  I will need to investigate more thoroughly how AMD is doing on that front, because to me it seems completely unacceptable to not allow the user to control fan speeds - the defaults on these GPUs are absolutely awful and in my case my gtx 285 will crash on the default profile from overheating.  I even get severe graphical errors from it running near 80c and the fan doesn't even kick in at all until then.

Also, using gitorious instead of github for now, I like the site design more and prefer open source to closed.  Github seems to be getting too big for its own good in my book, and are starting to fork away from standard git in a lot of ways.

<<Add Link to gitorious repo when I reupload it to my original gitorious account>>

Ubuntu, mysql-server, apparmor, and symbolic links

If you can't start mysql-server on Ubuntu (at least version 12.10, where I encountered the error, but I imagine it might be pervasive) apparmor might (will) be denying the daemon from running due to mysql using absolute paths and skipping symbolic links on any major system directories (in my case, /var and /tmp).  Apparmor doesn't like that, since mysql-server doesn't have permissions on /mnt/data/var or /mnt/data/tmp, and the apparmor instructions only state the main system directories.

This bug only happens if you symlink directories apparmor will manage access to and from.  And it will only happen if an application, for some dumb reason, traces the absolute location of its operating directories and tries doing system calls on file operations using the paths of absolute file location, rather than using system prescribed paths such that it references major system directories through absolute paths.

The solution is to edit /etc/apparmor.d/tunables/alias and add in aliases such as:

alias /tmp/ -> /mnt/data/tmp/,

alias /var -> /mnt/data/var/,

If you do this, apparmor treats references to those directories like references to the symlink versions.  It is advisable to do this with any default dir you symlink on an apparmor based system so that the security suite doesn't bitch about random applications using arbitrary paths to get to the same folders.

GPU Fan Speed Controller Project Notes

I'm going to be writing a gui / cron job daemon for regulating the fan speeds of the proprietary nvidia / amd drivers for graphics cards.  Mainly because with Steam coming to Linux and the burgeoning interest in the platform, there really needs to be an easy way (a la Overdrive / EVGA Precision on Windows) to create a software fan curve, and the Nvidia X Server / amdcccle are insufficient in purpose for that.  Fortunately, Nvidia and AMD provide means to access the gpu temperatures and fan controls from aticonfig and nvidia-settings.  So I'll just be writing a configurable GUI frontend, a config file, and a background cron daemon to monitor it and update temperatures accordingly.  I'm basing this project off the work of 2Karl's bash scripts to accomplish the same, but they are a little cumbersome to modify (no one point reference to the names of the scripts, no table of fan speeds, etc).

I'm just writing down my game plan with this little project here:

• Using Python3 and pyQT, the latter because QT is a nicer GUI framework than GTK at this point, and Python because this isn't processor intensive and doesn't need low level optimizations.
• One python GUI application (maybe gpu-fan-control) and one background process to watchdog the gpu (gpu-fan-monitor).  The GUI should be writing to a configuration file, probably under ~/.gfancc/profileX.cfg, and the monitor should be set as a startup application that will read this file, monitor the gpu temperatures via pipelining with aticonfig / nvidia-settings, and at a user-defined interval check for temperature thresholds to change fan speed at.
• The initial goal is to get the daemon and configuration running by hand, then write a nice GUI to set it up.
• My primary GPU is a GTX 285, so I'll have this working with Nvidia first.  The configuration file should specify the gpu type, and the daemon should have some autocreate functionality in case of a missing configuration.
• The GUI will probably start with just a few data points of temperature: speed that the background will just build a one to one function out of of fan speeds.  If the temperature hasn't passed a threshold, we shouldn't update the fan speed (to avoid calls into the control software whose performance I can't directly influence).

So I'll be putting this up on github soon.  Hoping to use this as a way to get into qt, packaging in Linux, and it seemed to me like something needed with gaming coming to Linux.

Gaming Rants 2: Why WoW Went Downhill

As a 5 year WoW veteran with over 365 days played across 3 mains (rogue, druid, and priest) I quit in patch 4.0 after raiding for a few months because the game had honestly turned into complete crap and I was wasting my time on it.  Here's why.

1. Progression as a whole went to shit in Wrath, and stayed awful in Cataclysm: Progression and goals in MMOs are so essential.  I stopped playing GW2 mainly because of it.  In Gaming Rants 1 on GW2, I outlined the reasons I kept playing WoW - when I ran out of them, I burned out and quit.  In PVE, they completely changed the progression system of classic and Burning Crusade, where each raid was a progression of the last, basically requiring some farming of the previous tier to progress much into the next due to gear and skill requirements, to a system where each major content patch reset gear and progression and made each raid the singular end game pve experience.  In the raid world, instead of having tiers of progression from MC to BWL to AQ / Naxx, you would just have ToC, or Firelands, or ICC, or Demon Soul.  And that is so boring because you are always in the last dungeon, you are always a few bosses away from done, and you never have no sights to see.  4.0 was almost a breath of fresh air with 3 raids, but the real missing aspect is progression where you go from instance to instance (they are "trying" that in 5.0, but it looks like its failing - I'll get into that).  You are nearly done with one raid, and you can look forward to another to get your feet wet in.  So good.  And it meant the people who do get to the end feel really good because it means they are the best of the best.  They are hardcore, they have commitment, and everyone else, including me, can look up to them, want to be like them, to achieve that.  Today?  Everyone has end game gear, they all look the same, the difference between an idiot with raid finder loot and a hardcore raider is probably that the hardcore raider reforged BC / Classic gear over their new crap.  Wonder why.
2. They also killed PVP progression: I would have loved to get into Rated BGs, but never had the "crew" to do it.  So it never happened.  They sounded ok, but without reasonable balance (another issue) they would have always been FOTM anyway.  Without negative rating, they were just grinds anyway.  Back in my day, you went from 1500 to 2300 rating and fought progressively harder opponents.  It meant arenas were actually a ladder, rather than fake rating on top of hidden rating on top of personal rating on top of team rating on top of rating rating.  The personal rating and MMR systems ruined arenas for me because there was no progress anymore - and specifically, MMR forces you to a rating and actively keeps you there.  Progressing becomes hard after you play a bunch, if you switch team mates your MMR is almost permanently fixed.  That is such crap.  Arena participation dived from usually over a hundred gladiator teams in each bg in BC to a few dozen in Wrath and sometimes single digits in Cata.  And it was because the progression was gone, replaced with a 0 to 1500 grind to make the progress artificial.  Rating on gear was fine though, to gate pvp gear.  I think the ratings could have been lower - gear should never have been an "issue" progressing at any point, just a way to keep the percentages of who gets what gear nearly even between pvp and pve.
3. Dungeon content went from challenge to crap: They made dungeons faceroll to facilitate the dungeon finder system, so there was no fun in doing the 5 man content anymore.  In classic and BC, there were incentives (rare drops, actually gasp useful 5 man loot like Blackhands Bredth, badges, or reputation) that made going back to this content after getting raid gear enjoyable because it is fun to crush your old challenges.  And the oldest 5 mans were legitimately difficult.  BC heroics were really hard until ZA and 2.4 badge gear trivialized them.  In 2.2?  Shiz was serious.  The normal modes weren't even a walk in the park - they required CC, resource management, kiting, and knowledge of fight mechanics.  That is funny today when any random new max level player can just chain queue heroics, spam a dps / heal / tank rotation, and expect badges to pop out at the end.  No challenge means no fun, and dungeons going to crap was symptomatic of the greater issues.
4. The world became small: With raid finder, dungeon finder, portals to every city, and flying mounts, the game world went from massive and mysterious to mastered and boring.  The large game space is worth nothing if you never have to endure it or brave it.  Without guardless neutral towns, wars never broke out.  Everyone felt safe and pacified.  Terrain was designed to support player flows, rather than the classic MC walkways that would get saturated with war.
5. Flying mounts and god-guards killed world PVP: I loved world pvp in classic.  Unknown odds, wars, assassinating - it was a roguey thing and I reveled in it.  It completely died in BC.  With few exceptions of daily quest hubs, Isle of Quel Danas, Halaa, and Auchindoun, people would just be flying everywhere with no engagement with other players.  World PVP, in its raw and unsterilized ways, kept the game fresh.  In Classic, you could never be safe outside a capital city, in contested zones many neutral towns weren't even safe in the early days - it was a truly dangerous world.  And danger and challenge make games fun.  Maybe not for casuals.  Maybe not for the "target demographic".  But they did, and do, for me - and I miss that.
6. Destructive content kept the game flat and dull: When every expansion (and more recently, every patch) reset every player, not only did it demoralize the entire population by making often years worth of work worth naught, it also means that content loses all value.  Nobody comes close to raiding the old raids anymore, the old dungeons might be done once for an achievement.  There are dozens of dungeons and raids that are now dormant and empty because people race to max level - they are never a challenge anymore.  All those Cataclysm raids will now be relegated to the trash heap, with someone raiding them maybe for a transmog or achievement, and steamrolling it with a few friends.  What a waste of developer time.  If instead of constantly resetting the field, new content was injected where the playerbase was most concentrated (ex: when BC came out, Karazhan would have still been a 10 man, and would have acted a bridge from UBRS to ZG / AQ20, Gruul / Mag / SSC would have been between ZA / AQ20 and MC, and TK could have been between BWL and AQ40 / Naxx, where the major bottlnecks were... - and then BT could have been past Naxx, and Hyjal could have bridged MC and BWL).  By now, we would have dozens of raid instances, all of them "overlapping" difficulty and gear (bigger raids were give higher level stuff for less effort, but you could still have really hard 10 or 20-25 mans giving top tier loot).  Rather than 3 Pandarian raids that are gated over 3 months because they can't make raid content fast enough, and they are just going to toss these raids out in 5.1 anyway with new heroics that completely supplant their loot.  And in 6.0 all these raids and dungeons go in the dust bin as people skip ahead to level 95.
7. The sterile game world is soulless: In classic, caves existed that were empty, mountain peaks with nothing, glyphic signs referencing developers in random places, a sign in an "unreachable" zone that said it was under construction, with the entire world tree already built, entire regions of the game world not practically useful - the deep Silithid hives, Deadwind Pass, southern Blasted Lands besides Kazzak - but they existed to have something challenging in the world, waiting for players.  Same reason world bosses existed.  They should have had roaming ones, it would have been fantastic (they did have some!  some random elite packs would wander some zones, like the Fel Reaver, but I wanted raid bosses like that!).  The inconsistency in design and dangers of the unknown made the original game alive, like a real world, even when every mob stood in one place and aggroed at 20 yards.  It was so much more alive than the dumb scripting and cutscenes of today because now, your challenge is presented to you with a big sign and a raid guide page entry with a ready check, not in a random cave in a new zone.
8. Itemization has become trite, dull, and bad: One of the greatest reasons I fell out of love with PVP was a two fold reason - the introduction of flat damage reduction from resilience, and the removal of critical damage reduction.  The first basically meant spells and abilities would scale even more out of control because they could just tweak the magic "players hit players for less" button and make the experience extremely jaded between pve and pvp.  The second is just bad balance and it promotes RNG.  They were trying to lessen RNG in Wrath, and then threw a curve ball and got rid of critical damage reduction - crits are the greatest RNG in WoW PVP, and have been pretty much forever, and was pretty much the reason they added the stat in the first place - but you can see how the design direction changed.  They cared less about balance, skill, and enjoyment, and more about bigger numbers and skinner boxes to maintain subscriptions.  And humorously, it failed - they are below their peak subs, they barely got back over 10 million with Pandaland, and I guarantee it will hit 8 million or less by 5.3, because people get tired of not having challenge or goals.  And heroic modes are the most obtuse "goals" ever - harder hitting bosses for larger numbers on the same items.  That is nothing on new loot with original art, crazy itemization, and custom special effects and procs. That was loot, what is there now is a mockery of progression.  The fact gear has become extremely sanitized, and players can reforge, level up, regem, etc the items makes item levels just a statistic you want to get higher.  No more Onslaught Girdle being insanely good into Naxx even though it drops from the first raid because it was itemized perfectly - no more Dragonspine Trophey being the best trinket for 3 tiers.  Itemization became flat and boring, and that makes loot and progression less interesting.  Arbitrary stat allocation made content fun because the value of a boss kill varied with the drops beyond just who wants the new +5 ilevel pants.
9. As an addentum to why gear + pvp suck, stamina is useless: In classic, stamina was used in place of secondary stats on high warlord gear to make it pvp gear - it made you more tankey and take longer to die.  You didn't become more efficient to heal, you just lived longer from 100 to 0.  Today, nobody stacks stamina because the stat has been devastatingly eclipsed by every other stat - mainly, the revamped primary stats that give bonuses to multiple other stats.  It is still 10 hp per 1 stamina, even when damage / healing / mitigation per stat point in every other class has only gone up every expansion, maybe except Cataclysm.  Today it is so awful that nobody ever itemizes it anymore.  And that is the contributing reason pvp sucks, and why every patch balance is thrown through a blender - players gain tremendous damage and healing increases, often from gaining points in 3+ stats when each of them alone outpaces stamina, and they expect the singular stat of resilience to counteract it.  And when it doesn't they have to rebalance vast swathes of classes (if they ever do, they usually don't and just let every arena season go to the dumps) because every time they add another tier of gear, resilence isn't enough to offset damage gains, healers get stronger to counteract the damage, but it means burst is stronger, people die faster, and they live shorter because stamina doesn't keep pace.  Above all else, the scaling of player health pools has always turned the game into crap.  The only time player health was ever acceptable was in BC, when base health was high, and stamina was high on pvp gear and scaled the best it ever has since arenas started (it didn't scale as well in classic without resilience offsetting other stats).  When health pools don't scale, the game gets progressively more zergy and ping-pong like, and it gets stale.
10. The game is still pay to play: After buying 3 expansions, I don't get why they don't just make non-current content free to play.  They already let people play forever at level 20 - and I know hardware.  It costs them dirt to support players on their servers, and they make runaway profits from the game and so little goes to maintain the servers.  They could have tons of free to play players at lower levels, maybe on legacy servers.  They could be buying item shop loot like pets and what have you, but they are not.  The reason I couldn't keep playing was not just because the game turned bad, but if I'm paying $15 a month for something, I expect to get my monies worth - and that required, for me, a truckload of dedication.  I didn't want to keep it up, but felt conflicted staying subscripted and effectively wasting my money if I wasn't playing the game.  So I did the obvious - I quit.  And I won't pay for almost any game anymore, because free to play is such an earnest budget proposition.  I have a large enough catalog, people are awesome enough to make enough mods and custom maps for what I have, and I have too little free time to want to pay money to waste my free time.  I'd play WoW again, despite all its modern day failures and faults.

That's enough for now.  For all its worth, the modern game is almost always better than its predecessor versions - I wouldn't really want to play classic again, combat is too slow and flat, mana is terrible on everything, and while itemization was better back then, stats themselves were boring with only crit and hit mattering.  Random resists on spells and abilities were awful, and it is good they were taken out.  Random low percentage procs also sucked, and they are not as common (cough, 5 stack taste for blood, cough).  Really though, if you retrofitted the better game-play decisions from recent expansions into classic or BC and balanced bosses accordingly, without giving every player the kitchen sink of spells so they can do everything, the game would probably be amazing. (NO ROGUE AOE, NO HUNTER HEALING, NO MAGE PERSISTENT PETS... etc).

Gaming Rants 1: Why Guild Wars 2 Failed My Expectations

So I bought Guild Wars 2 after playing a beta weekend.  I was hoping it got... better.. that what I got in the beta at higher levels.  I only hit around ~15 in the beta, and ~25 on live.  The problem in both was that I just ended up grinding the same areas over and over trying to level up, didn't enjoy it, and just quit. 

So here are my reasons for why I didn't like GW2.  Not saying someone might be completely opposite and have loved these features, but it really gets at what I want in a persistent world nowadays.

1. Progression didn't matter: the gating system of leveling in traditional games was / is meant to give you a sense of progression of power.  In GW2, this never happens, because you get downleveled or upleveled anywhere you go.  You can never go back to the newbie area and one shot mobs because you get turned into a level 5 when zoning in.  Immersion wise, it basically implies you can go from killing world dragons to dying to a bunny.  It made me not care about getting anywhere in the game, so I.. didn't.  Any MMO I play basically needs to give me a reason to keep playing... (con't)
2. Personal character combat and playstyles were dull: I tried a bunch of classes up to at least level 10 - the warrior, guardian, engineer, necromancer, thief, and elementalist.  My highest level characters were my guardian on live at level 25 and my beta elementalist at 15.  All these classes had the exact same setup: 1 - 3 were damage abiltiies, spam on cd and rarely ever requiring thought.  The last 2 were usually 20 - 30 second cds you also used on cd because why not, they are only cooldown restricted.  6 is the heal that you also spam on cd if you aren't at full hp, or if you get it for some other effect... you spam it on cd.  7, 8, and 9 were usually the highest dps abilities you could get, that you spammed on cd.  And 10 was an "ultimate" that you would use as a get out of shit card.  One ability you don't spam.  Yet you would often spam it!  If it was an offensive cd, you spammed that shit like it was in style!  You never had to stand and cast a spell, you could cast while moving.  There were no spell interrupts, so you never had to worry about casting in someones face.  The only CC effects were tiny stuns (tbh, long cc is dumb, so this is fine) that didn't have any timing element to them.  Without healers or tanks, you basically spammed damage into whatever is nearest you until it died or you died.  Some weapons sets on some classes could be "tanky" or "healy" in that the 2 - 5 spells would often be tank or healer centric, and you could get heal / tank 6 - 9 spells, but you could never actually "tank" or "heal" as a role, you were just a shitty damage dealer with tanky / healy utility.  So no matter how you sliced the game, you were spamming abilities on cd (maybe if you were microoptimizing your playstyle you would stack debuffs to maximize dps, but in general it was just button spam till things die nonsense).  You had a dodge, but because of latency and the horrible animations (honestly, monsters flail around in idle more than they do when swinging) and no delay between animation start and effect firing, you basically just randomly rolled around hoping to avoid something.  In GW2 favor, you could usually roll a projectile in flight to immune it, so I found it actually useful versus enemies at range.
3. PVP is a spamfest: With everyone being a dps class, nobody have legitimate damage "rotations" or ability priorities, and there never being a tradeoff or choice in how you engage in combat, it becomes a cluster fuck of throwing every ability you have into a pile of bodies, trying to get away, and then doing it all over again.  Since synergy was... limited, at best, you would usually only care to beat on the same thing as everyone else hoping some debuff benefited you.  With so many arbitrary weapon combos, and pretty much every combo producing a different set of 1 - 5 spells, any one player has way too many raw abilities to memorize from other profession + weapon combos, but that gets into the next point...
4. There is absolutely no diversity of choice in how you play: One weapon set per class was, in my playtime, always completely better than every other at some role.  A guardian / warrior not using a 2h sword for mobility / damage was dumb, an elementalist without a staff for aoe was bad, etc.  The only semblance of choice was when classes with weapon swaps would pick a secondary weapon to supplement their only true primary weapon choice, because they could either pick another weapon to have more faceroll damage ability spam by constantly weapon swapping, or use something tanky / healy in case they get focused and want to tank it / get health back.  Also the choice between melee or ranged weapons did matter.  But in pretty much every profession, one set would be definitively and mathematically better than everything else, and everyone just goes with that.  So all that weapon combo diversity goes out the window since something has to have raw numerical superiority at whatever you are doing, and since that is always trying to zerg damage into something, guess what wins out.
5. Groups don't matter and players exist in a vacuum: Monsters scale off how many people are engaging them, so grouping up has no benefit besides getting unique loot per kill.  If you don't kill things faster, you aren't getting any benefit, but since xp isn't split, it doesn't really devalue grouping either.  It is a completely neutral proposition.  Better than Borderlands 2 in that regard!  Not better than WoW circa 2004, or freaking Super Mario Bros circa the 80s.  More players should have tangible benefits since it is hard to get people to play with you.  Without a trinity, or any class synergy (hell, something like affliciton locks and shadow priests having shadow damage dot buffs is synergy that GW2 is severely lacking, sure, almost every class has a debuff it provides on some weapon that buffs everyone else, but it is rarely up, and barely a damage boost, and not coordinated since everything else is already inherently zergy) it feels like a single player game with other people.  There can be no really competitive and interesting pvp or pve to me, because I can't get into the lone wolf gameplay of CoD instead of Team Fortress, or Fighting Games instead of coordinated synergetic gameplay like WoW Arena's RMP.  I feel like persistent worlds require an aspect of each player being an imcomplete package or else the gameplay becomes boring and playing with friends loses flavor because supporting the weaknesses in others with your strengths makes these games.
6. The plot is obfuscated and inconsistent: By level 25 I still have no idea what is going on.  I was playing an Asura that hunted down a green armored man that killed some rookies, who was in league with an evil faction of Asura.  I then went into a set of forests and caves consisting of friendly unnaturally smart trogg rip-offs and got bored.  I had some "epic" quest where I had to open a portal to a ruined city on an island that was overrun by the old dragon the main plot has you kill, but as a storyline the whole thing falls flat for a few reasons - 1. the character I play is personified just enough to make me not feel in the story but not enough for me to see my "character" as an agent in this world.  Also, its an MMO, you will never have immersive characters with agency because you are too busy engaging with other people treating your avatar as you.  Strangely, SWTOR didn't have this problem, because the story had a consistent goal throughout that I could care about, the scale was sufficiently epic for me to engage with, and it came in discretized chunks where each ending brought new beginnings (just an example, Sith Inquisitor is training -> foundations of feud with Thannaton -> murder plot -> artifact hunt and power acquisition for 10 levels -> betrayal by master and Thanaton -> hunt for more artifacts and power -> kill Thanaton.  Each part led into the next, and made me care about going forward, because I always wanted to kill some jerk in the Sith at some point and looked forward to it.  The enemy of Guild Wars 2 was never put in my face for me to care about, and he never did anything I could personally want to kill him for, so I just didn't care.
7. The streamlined experience makes it flat: Waypoints are completely contrived, and you can just fade in and out anywhere in the world whenever you want.  So the world had depth until you find waypoints, and then you are pretty much done.  Since almost everything important is on the map, there is no exploration (with rare, honestly well done exceptions like the Lion's Heart cave and a few aerial jumping puzzles) and while some zones are thematically consistent, some are just too big and become a colorful mess of inconsistency.  
8. There are no long term objectives: In WoW, I started playing and saw Bloodfang Armor - and I wanted it so bad that I played for a year to get it.  By the time I finished the set, BC came out, and I wanted gladiator and a netherdrake so I played shaman / rogue with a great player in the first season as a rogue and got gladiator in 2s.  Then I wanted to be the best rogue on the server and in 2 seasons was nigh the undisputed best player left on Agamaggan, mainly because Gummi Bears and their entire rank 1 crew left.  In Wrath, I wanted to try playing GM, so I did.  After that I wanted to try pvping hardcore again, but never could get the players on the shitty server.  That pretty much eventually led to me quitting, combined with the reasons WoW fell apart for me.  Hey, another blog idea!

Those 8 are pretty much it though.  If the combat was fun, grouping was beneficial and engaging, and I had goals, I would have liked it more.  The last day I played it was basically just logging in to chat and attempt to engage with my only friend also playing near my level before just getting bored and quitting after gaining a level.  I had no reason to keep going, I had my entire skill bar and every weapon unlocked, and the gameplay wasn't fun enough to keep me going.  I could farm BGs in WoW for years or grind instances forever because I liked the way WoW played in the old days.  I just couldn't get into GW2 personally, so I gave up.

Software Rants 5: Architecting a 21st Century OS

In multiple posts on this pile of cruft and silliness I've spoken about how I don't like the design of most modern operating systems.  I've even written some poorly construed ideas on the subject.  So how would I go about building something that, comparatively, has taken billions of man hours to create in a couple cases? (although subjects like Haiku make me feel a bit more optimistic). 

So I said microkernel.  This theoretical system has one goal being the minimization of kernel-space code - and anything platform specific that doesn't directly tie into the ability to execute software should definitely be userspace.  Linux has the philosophy that one source archive needs to compile with a configuration to work on any system ever, from the lowly phone to a massive supercomputer cluster of thousands of nodes.  You will get tremendously different binaries and KO sets depending on your configurations.  The difference between a PPC64 supercomputer all necessary modules compiled in kernel and a udev based dynamic module loading ARM kernel for phones makes them barely resemble each other - besides some basic tenants, the same command line arguments, the idea of loading an init payload, initializing devices, and using the same (similar?) memory management and scheduling algorithms (there are some builds of the kernel that use different ones...), the resulting system will be using entirely different tracks the source tree.

I disagree with that, in that those aren't the same software projects anymore, and keeping them in the same source tree is the exact opposite of the do one thing right, well, and concisely UNIX ideology.  And that is fine, because it means the Kernel is ultraportable, you can get an upstream codebase, build it, and run it on practically anything, and with a source clone you can customize the configuration to suit almost any use case. 

But there is also a reason Linux takes multiple levels of management and a ton of organization behind it - they have everything from device drivers to hardware translation layers to language documentation to binary blobs being merged into one program and it becomes impossible for any one person to understand or conceptualize such divergent tech.  I don't call myself smart, but I do think smart people are averse to complexity where possible, and this is definitely one of the places I see ample unneeded complexity.

So here is my proposition : a microkernel core, designed in and written in L, that only handles virtual memory management, preempted process scheduling ("virtual execution"), a virtual filesystem abstraction layer (so that aspects of this kernel can be mapped into the final root filesystem without awkward dependencies on user space filesystem managers), a virtual socket layer (for communication purposes, the sockets themselves would be managed in userspace, but the kernel would initalize this system so that some user space network daemon can manage socket communication - but the kernel itself will be using sockets as well to communicate hardware state and with the daemons it connects to!), and a hardware abstraction layer that allows user-space privileged daemons to take control of hardware components (disks, video devices, busses and usb hubs, etc, basically everything divergent from the RAM/CPU mix*).

*: I would be interested in exploring if there would be a way to have the memory controller in user space.  In that the kernel could start and initialize itself only in processor space, but it seems to be incessantly complex... you can't even establish any other virtual systems without system memory to use.  It would have the same issue a filesystem host would have, that the kernel would need to, in advance, start up this daemon, and then fall back to its own boot procedures.

The traditional boot philosophy is on -> firmware is payloaded and initialized by hardware -> firmware does primitive device detection and scans persistent storage buses for something with a recognizable partition table and payloading a bootloader from such -> giving it a "direct media interface table" even though that is Intel jumbo, with some hardware memory mapping used to provide devices.

UEFI is like this, with more complex device loading (including a 3d driver of some sort with the graphical setups) and the ability to scan not just partition tables but fat-32 filesystems for bootable executables.  It is pretty much an OS in and of itself considering how it behaves.

In my grand delusions, we could scratch the unnecessary parts - the important aspects of a boot cycle are device initialization, error detection, and searching for a payload.  Device initialization is already pretty well "passed on" to the resulting OS.  Error reporting is more complicated, because you are dealing in a world where the most you have access to may be some primitive bus signals to indicate problems, such as beep codes, error panels on the board, or keyboard signals to blink the ps2 port.  BIOSes and EFI boot procedure are obscured by platforms - each new chipset does things differently, merging more parts onto the cpu, or handling device channels differently.  In terms of payload searching, EFI actually does a really good job - given a boot device, and a system partition on it, load a binary.  No need for traditional bootloaders (which is nice).

On -> check for cpu / memory errors, on error try to signal all devices with some special hardware failure signal and a payload error descriptor.  Devices would need to be designed to handle this signal if they have some way to display errors (vga, keyboards, sound systems) or drop it.  The expectation is that "on" means devices power on, and the independent units like network controllers  come online at the same time and await the cpu to initialize them.

Check errors -> initialize devices.  Given no catastrophic errors, check if any other device has errors, and if not build a table of devices to provide the payload device to boot.  If a device has an error, broadcast another error payload signal to let anything capable alert the "outside" of the problem.  But don't stop booting if the error is recoverable, just indicate a failed device in the provided "table".

Initialize devices -> payload.  You have a table of devices, and the firmware needs to be aware of where we can find something to payload.  In terms binary byte ordering, that is an open question - we would prefer big endian for readability if it doesn't incur a hardware complexity cost.  Nobody should almost ever be working with binary data representation at this scale anyway, but if we can do Big End without circuitry overhead, we should, otherwise, keep it simple stupid and use Little E.

Since we have effectively an integrated bootloader, we need some very simple file system to just store binaries.  Now here a point of contention - have complex file system analysis machines that can read our "stock" FS type (which would absolutely be in the btrfs vein of COW auto-compressing snapshotted filesystems) or have a discrete filesystem just for the bootloader to read off.   We need to think of MBRs here - and logical and physical sector sizes.

I want to propose variable sized sectors, the same way there are viable sized memory pages - 4K memory and 4K disk are great defaults, but you can always use larger contiguous blocks of.. both.  In both cases, page size and sector size implicate overhead on the managers of these spaces while having larger sizes implicates overhead of its own when boundaries are not nicely met.

For one, traditional storage media just can't have variable sized physical sectors.  Having different logical sectors seems silly because it is a great simplication of work to have 4k sector sizes in both cases.  That will continue to work well for some time, and a sufficiently smart operating system can optimize sector utilization to minimize wasted space.  That is a device driver problem, though.

In terms of hardware memory pages, I still think swapping has value even if traditional desktops don't need it anymore - too many problems just require tremendous amounts of memory to work with beyond the bounds of traditional computing concepts, and we like embedded systems with low memory.  Even if you could theoretically implement something akin to paging with file system abstractions (writing to and from disk once you approach the physical memory limit) having the option there has proven to be worth it.

So page sizes - we don't want too many sizes, and we want them to scale nicely if possible.  This would require research and insight I don't possess, but you definitely want to support variable sized pages.

So we assume disks have 4K sectors, pages are at least 4K, and we may or may not have a dedicated partition standard for bootable binaries with some specialized file system.  We will need a disk partition table, and if were mandating 4k sectors, we have 4k bits to store info on the device.  I like how GPT has a backup copy of itself, so we want one of those, so its really 8k bits total, the first and last sectors.  In terms of sector addressing, I'm starting to think about 48 bit as an option - the overhead of 64 bit just for 64 zetabytes seems unnecessary when 48 bits gives an exabyte of storage.  Currently, the largest storage device is approximately 4tb, up from last years 3tb, the year before that at 2.5 tb, the year before that at 2, the year before that at 1.5, etc.  So if we go with a terabyte a year (which is about right for the last 5 years) we have a few decades before this becomes an issue, and we can just add in a 64 bit addressable 2.0 version anyway since we want userspace drivers.

Similarly, I'm not entirely sold on pure 64 bit cpus either.  The real big argument for large memory cpu sets is that server farms with shared memory need to address all that space, but you still get 281 petabytes on 48 bits.  I'd probably make the 1.0 of this system 48 bit, and make sure the language and instruction set inherently are valid converting to 64 and maybe even 80 / 96 bit.  This is actually really easy in the conceptualization of L, because you would have pointers as objects, and their size would be compile specific to the platform pointer size.  Integers are decoupled from words, so you can create ints of all sizes from 1 to ~16 bytes (I don't see why you wouldn't implement 128 bit integers on a modern architecture).  This also brings up the potential to just do the 64 bit virtual 48 bit physical word sizes Intel uses, but there is a translation in there that adds unnecessary complexity in my opinion.

So 48 bit CPU, 48 bit sector addresses, 4k standard pages and sectors.  Big endian unless there is a complexity or performance hit, in which case we just use little.  We want point to point memory rather than dual channel, and I already talked about CPU architecture earlier - having a heterogeneous collection of registers, alus, fpus, simd cores, etc to run dedicated parallel instructions.  This way the traditionally discrete gpu and cpu cores (even on a shared die) can be more tightly integrated.  You could also use one single ram, and not have to reserve it for the gpu cores.  As long as the instruction set is designed around supporting SIMD instructions for the parallel processing cores, we should be sound.

So we have our payload, we run it, and we set up a process scheduler that I will look into more in the future (really, this is the kind of decision that takes a ton of reading up on, to figure out the best preemptive scheduler for purpose, but CFS I guess is the industry standard).  We have the virtual memory, and we need some way for the kernel to initialize physical hardware virtual memory mappings.

So we don't want the kernel explicitly dealing with device management, but it needs to initialize device memory, so we can just add that into the device table the firmware provides - one of the hardware signals can be for the memory map size.  The kernel can then map memory addresses at this time to the given tables memory requirements for applications, and when an application access device hooks, it also gets control of the virtual page table referring to that device.

So scheduler and memory is up.  We want a virtual file system now - no hardware folders are even initialized yet, but we can provide a virtual hardware file system for device access.  This would be an elevated privilege folder - as an abstraction, you can have device control servers open devices for writing to take exclusive control over them, and the writable file is their memory map.  I proposed a file system layout earlier, so here we are talking about something akin to /System/Hardware/*.  The VFS would propagate a node per hardware device provided, the folder would require level 1 permissions to access, and once a device server has control of the virtual memory map with the device opened for writing, the only thing other servers can do is read it.

So this virtual file server needs the concept of permissions at the kernel level - we want executables running to have a level of privilege, beyond the bounds of kernel vs user mode processor execution state - this is a software privilege, where level 0 is the kernel, level 1 is init and systemwide access services and daemons, and has a "pure" view of the file system.  Level 2 would be the usual run mode of programs - restricted access to system essential files, restricted views of the file system, and each application would have device privileges specific to it given to it by a level 1 service.

Some examples - the vfs, socket layer, memory mapping, and scheduler are operating in level 0 kernel mode.  A gpu driver, device controller, usb host, a file system daemon, or a dhcp host, smb server, virtual machine service.  You want two levels of indirection here in most cases - a sound server to manage audio device servers running, a display server to manage display adapters running, a network server to manage networking devices, etc.  Access to these servers is restricted in the vfs and vss through executable privileges, probably by a level 1 executor server that wraps kernel level execution behavior.  Basically, kernel sets up permissions that the executor server manages, since level 2 permissions can't access level 0 directly.  Level 3 is the "pure sandbox" - it would be started by level 2 programs (like a VM) and has no device access directly, and only has its own restricted view of the vfs and by default has no write permissions outside its execution context.  You could thus host users from level 2 session managers (maybe run by an administrator) and they would be unable to manipulate the outer system by design.

So we have 4 permissions levels right now, and you could theoretically add more and more to add more levels of virtualization.  A virtual machine could thus just be a level 2 program that pipes commands from level 4 devices through the virtual memory map of a level 3 kernel into the level 1 devices above it.  Very slick I think.

The other major revelation is the idea of display management.  In the absence of dedicated video hardware to control by a level 1 daemon, the video daemon could itself emulate a video server.   Or you could set up the userspace where Kernel is level 0, device controllers are level 1, device managers are level 2, and user applications are level 3, so that user applications never interface with devices directly but only through abstraction layers.  I actually like that somewhat more than the other model in some use cases.

And then of course the traditional server model can just run everything at level 1.  It isn't kernel mode, but it has device mapper access so you can set up the traditional ultra-fast interconnects.  This alleviates the problem of FUSE and its ilk in Linux because you don't have to emulate the device controller without any hooks and pipe them, you inherit devices into user space.

So I'll talk about the device / manager services more next post.