Software Rants 8: The "Linux" Desktop

This is mostly a post for the sake of copy-pasta in the future.

For one, I have fallen to the darkside: qt5 and KDE have won me, after spending a bit of time tweaking a KDE install I can't get over that the ideology underlying all the eyecandy is what I'm after. I am still hesitant by the stupid reliance on having their own libraries for everything, but in recent years the KDE camp seems to be getting more inclusive, so I guess now is the time to jump on that bandwagon. I know that Plasma Active and Plasma Workspaces is the future, even if one needs maturity and the other desperately needs optimizing.

But there are other realities to the Linux desktop that are coalescing - and I think we are in the home stretch of the final maturity of the platform, at least when Wayland hits the ground running. All the old and ill thought out technologies of the 90s are mostly gone, and as we plateau the efficiency of the classic computing paradigm, the requisite technologies to support that reality are finally emerging. Here I want to talk about what they are, and why I feel that way.

• Linux Kernel: For its monolithic, toss everything in kernel space, write it in Assembly, and make it the most illegible mess of a massive project ever, it does work, and now that pretty much every device and server exists in some elevated kernel-daemon unsolicited love affair, it is fast and stable. Be it the DRM and DRI for video, ALSA for audio, the IP stack and IP tables for networking, the file system implementations, or the core CPU scheduling / memory management, hardware is finally coming under the control of the kernel for the first time. The only real great leap I still see on this front is the implementation of openCL pervasively throughout the kernel and support infrastructure to utilize graphics hardware on consumer products appropriately. We still run all this stuff cpu side, and a lot of it can see some gpgpu optimization. This is still a few years out, and it will require the usage of gallium throughout the underlying system as an emulation layer on server and ancient hardware without openCL capable gpu hardware, but a slight memory overhead and constant instruction cost across the board will absolutely be worth taking advantage of the most pervasive modern hardware inclusion - a large collection of weak, high latency, high bandwidth parallel compute units, that are becoming increasingly generic for purpose.
• Comprehensive user space daemon collection: This is a meta topic. I Think the Linux space is finally stabilizing on a collection of portable servers to intermediate the kernel hardware control that are sufficiently feature dense to support any use case. Their continued active development means they are much more likely to adapt to new technology faster than having KDE / Gnome / XFCE / LXDE / Openbox / etc try to do everything their way and have no portability.
• Wayland: Once X dies and Wayland takes over, video on Linux becomes "complete". Video drivers then just need to implement the various dialects of openGL and CL, and not worry about integrating with a rendering technology from the 80s. The simplification of visuals will be the greatest boon since completely fair scheduling. I absolutely want to get involved on this front - I see a cohesive, beautiful and simplistic delegation of responsibility emerging that can and probably will prove revolutionary, especially as gaming comes to the platform in force. I hope Valve motivates the adoption of Wayland quickly as a consequence.
• Pulseaudio: It might be latency heavy, but that can be optmized. Having a central audio manager though is essential, and the generic nature of Pulse means it is pretty much an unstoppable force in audio at this point. As long as it continues to let audio nuts stick Jack in (maybe they could even cohabitate the space better, letting certain applications get Pulse passthrough to Jack, or supporting Jack as an independent audio sink). 
• Systemd: Old init styles are out, and systemd does a lot right by being significantly based in file manipulation. To enable and disable services, you just add or break file system links. Systemd might be a kitchen sink, but considering it sits on top of a kitchen sink kernel, it seems appropriate. Systemd is rapidly becoming the user space kernel, which isn't necessarily a bad thing. However, the configuration and speed is superior to the competition.
• Dbus: The last two used dbus as its IPC channel. KDE has adopted it, Gnome made it, it is here to stay as the main protocol for IPC. Message passing is the way to go, and dbus can optmize itself enough internally to make it perfectly reasonable in 99.999% of use cases. It might not be that generic socket layer I liked in my earlier postings, but a lot of about Linux isn't pure, but it still works.

On top of these resources, we put fille managers, compositors, desktop managers, and other goodies. I'm siding with KDE - qt is much better a framework than GTK because C++ is a much more usable language than C is with its superior syntax and support for almost every programming concept under the sun. While I think it is a horrible break in Unix philosophy, the integration of webkit and v8 into the qt to support web rendering internally and javascript via QML means it is ready for the integrated web world of the future. GTK is still struggling along on this front, and the entire Gnome project is floundering in the face of trying to jump into a mobile market already saturated with Android.

I think KDE has the right ideas. It might be slow, the default theme might be ugly as hell, but it is so freaking configurable in a really intuitive way that I can't fault it. XFCE, LXDE, and any other project that aims at "simplicity" I feel is really copping out nowadays. Simple can't mean "doesn't provide a complete package" but it is being used as an excuse. Simple is when you have good defaults, and multiple layers of configuration (in XFCE's defense, it nails this - you have the defaults, you can tweak the panels by context menus, you can then go into a system settings gui tree, then you can enter a gconf database editor and tweak even deeper, there are 3 layers of configurability, each incrementally more fine grained, larger, and less comprehensive, but it nails the format).

KDE is not golden - I have avoided it for a long time. They depend a lot on eyecandy but the optimizations just aren't there. But the ideology is in the right place, having targeted generic computing and mobile computing as completely different use cases, having configurability as a top priority, and having reasonable defaults. Except for the dumb desktop Cashew, I have no idea why that isn't just a removable panel, and it is concerning that the devs care to keep it so badly when it has strong community backlash. But it is the right direction to go in. I don't think the Ubuntu model is going to peter out much longer on the fumes it runs on - the end game just isn't there. Ubuntu TV might get good when it lands, so maybe it can take a niche as a DVR OS that also functions as a powerful desktop, but it won't make it in the mobile space, and it alienates its core users by stuffing ideology down ones throat. Also, Linux's strength is in a collaborative platform like KDE or XFCE, not in some principled elite council delegation platform like Gnome or Unity.

So I'm going to put my OSS contributions towards the KDE stack in the future. I prefer C++, so qt is a natural fit. razorQT is a neat fork, but ultimately it is too similar to the kde platform to compete. I feel anyone who would migrate over to razor would be better suited just optimizing KDE to make it run better than to try to start over from scratch again.

If KDE becomes the first comprehensive desktop with Wayland support, that will be the final nail in the Gnome coffin. The future of the Linux desktop is pretty much here, and it isn't a distribution, it is a software suite.

I don't really like how much effort KDE puts into competing in the application space though. In the end, the desktop is there to frame applications, not permeate them, and while Calibre is a great book management framework, I think the KDE project spreads itself too thin trying to provide the desktop and the user experience front. So while I may be running a KDE desktop, I'll be using Thunderbird, Firefox, Deluge, Skype, Clementine (which is a fork of Amarok... heh) and Libre Office rather than the underdeveloped KDE alternatives, because theose other projects have focus on one product that makes it all the better. That is what makes FOSS best - people congregate around projects with clear objectives and goals, and make them happen, not nebulous archives of thousands of projects trying to reproduce the work of another million developers in other projects. So if I end up getting a KDE development setup going, it will be working on the core. Though Dolphin seems to be a pretty good file manager.

Also, GTK applications in a kde dark theme look like ass. I'm going to have to blog about fixing that.

Software Rants 7 : Function Signatures

While thinking about Al, one thing that would be really nice if all object definitions behaved the same, akin to C++11 universal initializing syntax.  For one, classes and functions should be defined like normal data, so if we use a syntax like int x = 5, you want to have class foo = ?.  The first question on this front is what is the minimum syntax to define any data.  The best way to deduce how to best go about this is to look how it is done in other languages, and it isn't that complex.

• In the C syntax languages with static typing, an integer is always int x = 5, or if you want to heap applocate it, you do int *x = malloc(sizeof(int)); *x = 5;
• In Python and Ruby, you use dynamic typing and forgo the int part, so it is x  = 5, but in map declarations it is x : 5.
• Perl uses $ name declarations, so it has $x = 5.
• Haskell just defines numbers like Int foo -> 5.  
•  In Shell, it is x=5.
• In Javascript it is usually var, let, or nothing x = 5, but in maps it is x : 5.

As a consistent notion, the equals sign is used, except in maps, where colons are used.  As a grammatial lex, : defines an is relationship, and = defines equality.  I am absolutely considering : as the = statement of definitions and letting = just be logical equals like == in most languages.

Regardless, the syntax is consistent.  In Al, you would have static strict typing, in that int x : 5.3 will error of an unspecified cast from float to int with loss of precision.  int x : "s" fails.  auto x : 5 resolves to an integer type, and you can drop the auto and just have x :  5 which behaves like auto.

As an aside bitwise operations like | & and ^ are becoming more and more unutiized next to their logical counterparts, I'd definitely reserve | & and ^ for or, and, and exponentiation respectively.  If I were to have glyphic bitwise operations I'd use || && or ^^ for those, if anything.  I'd probably just reserve bitxor, bitand, and bitor as keywords and forgo the glyphs since they are so situational.

So if we have a syntax of int x : 5, a function would be func foo : ?.  We want templated function definitions, so our function should be something like func< int (int) > in C++ but the syntax int ( int) isn't conductive of a comma separated value list like a template specification.  Once again, we want to minimize the syntax, so the smallest definition of data with all unnecessary information remove would be something like:

int < int, int foo : int < int x, int y { }.  If < wasn't less than, it would just be the function signifier to deliminate a return value and the function arguments.  This syntax leaves something wanting though, so we try the verbose optimally readable way:

func[int]<int, int> foo : [int](int x, int y) {
}

This looks a *lot* like C++ lambdas.  On purpose.  The capture group just fills the roll of the return type declaration, but if we want a function signature we need that information.  Function templates get uglier:

template<Z> func[Z]<int, Z> foo : template<Z>[Z](int x, Z z) {
}

This happens because the rvalue requires a template definition but so does the l value because that acts as the signature.  We redefine the signature twice.  This is absurdly redundant, so what we want is the concept that if we have an rvalue function we never redefine a signature because the static typing of the function declaration already brought that up.

template<Z> func[Z]<int, Z> foo : func(x, z) {
}

So the arguments types were defined in the signature, and we named them in the definition.  The problem here is that if you were to declare but not define foo for some time (given we even allow that, we would probably want this language to forbid nulls, and all functions are inherently a type of reference into code space) then when you actually do define it, you end up with something like:

foo : func(x, z) {
}

And that definition gives no information to the original signature.

Of course, in practice, you can't do this.  You can't have late binding on a function because it occupies built up code space, not stack or heap space.  Defining it in two places is practically worthless because the compiled code locks foo to be its function at compile time and you can't reassign it because it means you have code pages no longer named.  That means that raw function declarations are inherently constant and final.  Meanwhile, something like :

ref<func> bar; bar : ref(foo) is valid.  You are taking references to defined functions, but you must name them at least once and they can't go unnamed.

The same thing happens with classes.  It might be why in traditional language syntaxes classes and functions don't behave like the data types they represent, because the definitions are disjoint from implementations.  While classes and functions are data types, they are data type definitions - you instantiate instances of them.  They are inherently constant and final and bound to their definition names.  So if you use a global declarative syntax like foo : func, you introduce ambiuity without making the full definition something really ugly like:

const final template<Z> func[Z]<int, Z> foo : func(x, z) {}.

So let us save some time and call it template<Z> func foo : Z(int x, Z z) {}.  Maybe have state defined like func:private,extern,static foo(int x, float y) : Zoo {}.  It is kind of backwards, because it implies func : type of return is the signature rather than thing : value, but the value of a function is its return time if it is pure, for example.

Hello World in Al and why?

I wrote up a pastebin post that specifies a bit of my little pet language idea.  Just to get into the nitty gritty compared to the status quo languages and new contenders:

• C: C is way too old, and has a ton of problems.  Glyphic syntaxes like pointers, textual inclusion, no baked in template or polymorphism to facilitate dynamic code, no objects, no access modification to help team based development.  Function objects are convoluted and error prone due to their existence as effectively a void*.
• C++: While an improvement on C, a lot of what C++ does also falls flat.  It tacks on object orientation rather than being systemic with it, so interacting with C data types like ints is convoluted.  Structs are preserved as just default-public versions of classes that are default-private.  The smart pointers I feel properly implement memory management, but it is a little too little too late.  There is a lot of duplication of purpose in the standard because it is very committee driven.  As a result, this language is like a bathtub for a kitchen sink with a tendency to spring leaks.
• Haskell: The adherence to the functional ideal is admirable, but I fundamentally disagree with the treatment of the process of constructing tasks for the execution of physical transistors as if it were only math.  Statefulness is an inherent property in computers, and while it introduces overhead, it is necessary for any programming model, and Haskell is no exception in the line of functional languages that try to hide state in a leaky way.  Also, being single paradigm, it isn't all encompassing enough.
• Go: The syntax is pretty cumbersome, it lacks static typing, its memory model isn't fine grained enough for what a real systems language needs to guarantee behaviors.  It throws all its proverbial utility into parallel programming, but by doing so they make the entire language a pain in the butt to use and full of non-determinism.  So it is a good niche language for massively parallel projects, but otherwise insufficient for a generic compiled language.
• OCaml: lacks static typing, and has a really glyphic syntax.  I really do think this is a big deal, static typing makes determinism so much easier.  If you are object oriented having everything discretized into nice object boxes is a miracle for debugging and maintainability.  I do like how you can use OCaml as a script or as a native binary.
• Rust: again, inferred variables.  Not allowing null pointers is an interesting proposition I would definitely want to look into.  The concurrency model of message passing I feel is the appropriate solution, and it has an emphasis on async functionality for small execution units.  I'd rather build a threadpool and queue tasks into that in a program, but to each their own.

So just to summarize my goals with this thought experiment:

• The code should be deterministic.  No construct should be more than a sentence to explain how it runs when assembled, and as such the compiler should never be overtly complex.
• Garbage collection is appropriate in some use cases.  The best way to do it is let users opt into using a garbage collector, like most standard library functionality that would break the previous policy.
• A threading model based off the usage of functions as first class objects and the availability of a concise and easy to use socket interface to send efficient messages across processes.  Inter-process communication can also be done over that socket layer, done over some internalized message passing, or memory can be shared through references.  A thread would not have visibility on another threads stack, but would see global variables and could be given references to stack or heap objects explicitly.
• Smart pointers!  They solve the memory problem nicely.
• Don't fear references / pointers.  Rather than having the glyphic * and & syntaxes of the C's, we can use a ref<> template object to refer to a pointer to something.  I like the non-nullable ideal, so maybe this object needs to be constructed with a reference, and attempting to set it to null throws an exception.
• Exceptions are good!  Goto as well, none of the execution mode switchers are inherently flawed. Code is naturally jumping all over the place, just minimize the hard to trace parts.
• The glyphs should be minimized.  You want an easily read language with an optional whitespace significant dialect.
• Module based inclusion - importing a module could mean importing a text file, something already compiled, an entire archive of files or binaries, or even a function in a file or binary. This means you have a unified access view into other namespaces.
• Access modifiers!  They make peer programming so much easier.
• Designed with the intent to be both callable and interfacable with the other Altimit languages.  You can't kitchen sink all problems.  This language is meant to be able to give you the tools to maximize performance without needing to have overly complex syntax or requiring more work on your part than necessary.  But by default, it needs to be deterministic and safe.  The bytecode language would be easily sandboxed for application development, and the script language would be easily used as glue, easily live interpreted, and high on programmer productivity.  Using them together could give you a powerful toolkit that nobody else seems to try to build cleanly.

I wonder if I'm ever going to go anywhere with all this crazy talk.  We shall see.   I might write more pastebin samples.

Software Rants 1: Everything is wrong all the time

It occurs to me that this post was a draft for a long while.  I wrote a paragraph, tossed it out because it was rubbish (wait, I thought all my musings on this blog were) and probably should have a part 1 to a series that would be more tongue in cheek to have a part 0.

But to write this entry I have to get behind why I wrote the title, couldn't come up with anything sound for this post, and gave up on it for 2 months.  In simplest terms, I am an idiot and I have a hard time understanding complicated things I haven't spent years developing habits and muscle memory to recognize.  As a result, when in software-land, there are a trifecta of problems that contribute to this - software complexity is huge, people try to minimize it, the means by which they minimize it specialize the industry and make trying to be a "master of computers" like I keep trying to pull off a lost cause.  It is why I keep going back to wanting to do my own computing environment, because the most fundamental problem is that none of this was ever expected.  New things happen every day, and the unexpected contributes to all the complexity in this industry today.

If I wanted to hack on a software project someone else made, I would need to get it out of source control, so I need to understand the syntax and behavior of that properly.  I'm putting it on a hard drive, so I need to know how disk caching works, how the IO bus handles requests, how it determines spin speed, how a laser magnetizes a platter spinning at thousands of RPM, why they settled on 5v power (even though the damn power connector gives 3.3, 5, and 12 volt power, and nobody uses 3.3 volts, I wonder why, hey power supplies why you gotta break AC into 3 DC voltages and barely use 2 of them?).  How the file system works (b trees, allocation tables (distributed or contiguous) file distribution on the drive (aka Windows vs Linux style), how solid state storage memory works using capacitors rather than transistors like processor cache, how the south bridge and north bridge interconnect sends data, how SAS and SATA differ, how an operating system can recognize a SATA bus, determine the verison of the connection, schedule writes, treat a PCI, USB, SATA, or IDE hard drive as basically the same thing even when they are radically different.  And we haven't even gotten to the files in a folder yet, just where we are putting them.  We didn't even touch on TCP, UDP stacks, the different sockets, different variations of CAT cable, how hertz modulation over an ethernet line determines the bandwidth, how to differentiate power from data, analog vs digital signal processing.

The amount of stuff is huge.  And a lot of it is hard stuff.  Yet somehow a 5 year old can use a tablet computer to play a game, utilizing all of this and more (graphical interconnects, switching video and physical memory, caching, synchronization, SIMD instructions, opengl driver implementations, DMI interfacing) and doesn't know a wink of it.  And it may be my greatest crime that there is almost nothing I can use without understanding to a degree that I could put it back together by hand with a welding iron and a few billion dollars worth of silicon fabrication and imprinting of transistors. 

So the necessary time commitments to understand the entire beast are honestly too large.  So I prioritize the things I care about most, and often end up iteratively moving through the process - a CPU composed of ALUs, FPUs, caches, registers, a TLB, maybe a power modulator, the northbridge logic, a memory interconnect, physical RAM operating at certain timings, refresh rates, clock rates, channel sizes, how the operating system utilizes page tables to manage memory, how TLBs cache pages in layered cache on the processor. 

And just the hardware itself is overwhelming.  So you get past that and you arrive in software land, and start at a bios that exists in battery run ROM on a motherboard (and don't get me started on all that circuitry) that initializes memory and devices, and depending on if you are running EFI or BIOS you end up either executing code from a partition table or you search partitions for a system partition to find executables to run.

And on top of all this hardware complexity, we have dozens of ways of writing an OS, different binary formats, different assembly languages for different processors, different dialects of C that usually ends up at the base of it all (because we are stupidly set in our ways, to be honest.  C is so flawed... )  We pile on top of that extreme language complexity (C++) or extreme environment complexity (Java / C#) or extreme interpreting complexity (Python, JS) where something is extremely complicated to understand but essential to actually understanding what is going on.   And then you have your programming paradigms, and have Perl, Haskell, Clojure, and a dozen other functional languages or other strange syntaxes out there people use just to make your brain explode.  Yet people much smarter than myself can read these like real second languages.

It might be the fault that after 6 years of Spanish going in one ear and out the other, I am firmly grounded in English with no hope to speak anything else.  Mainly because my brain is crappy.  But in the same sense, I like my programming languages being new vocabularies rather than new languages that break my brain. But I don't think it is even my problem entirely - it is evident from the amount of failure in software space, the extreme desire for "talent", and the general cluelessness of both developers and customers in building and using this stuff that things have, just under the surface, gotten really out of hand.  And I really think that the best way to fix it is a clean slate now, so we don't end up with the Latin Alphabet problem (which coincidentally is the other blog I'm posting at the same time as this one).

So even though this post comes out 6 entries into this series on software, it does establish the baseline for why I rant about the things I do - everything is broken, and wrong, and nobody wants to take the time to just do it right.  Mainly because we are fleshy meatbags that only persist for a blink of time in space, require the flesh of living things to persist, and have an unhealthy obsession with rich people doing stupid crap.

Thinking about Alphabets

Since I've wrote a bunch about basically throwing away 30 years of work done by engineers significantly smarter than me, it occurs that you should really question everything when devising a new computer environment, besides just making a new character set that eliminates the more pointless glyphs of low order Unicode.  One aspect of that might be going as far as to redefining the alphabet used.

After all, when complaining about things being outdated and obsoleted by new technology and ideas, the 2,700 year old glyph system (albeit with added and removed glyphs over time) that forms the base of all language in the western world is a good candidate for reconsideration.  A lot of characters in the set are redundant - C and K, Y and I (and E).  In that sense, I am a fan of the International Phonetic Alphabet, which is a glyph system representing the pronounceable vocabulary of the human vocal tract.  It includes both the lung-based sounds, it has an extensions and sub-groups for clicks and other non-pulmonary sounds, and in the end it represents the goal of written language - to trans-code spoken language.  We can read a large spectrum of glyphs, and if we wanted we could encode them in a wide color gaumet, but our audible range is much more limited - the IPA has 107 characters, but in practice only around ~30 of them are significant, and if you got technical enough to create an alphabet with the discrete independent elements of spoken language, you could probably manage around that number.

But this isn't an infallible problem with a simple solution - the reason students don't hear about the IPA is because it is what many things with the word international in them the glyph system it uses is a hodge-podge mix of a dozen languages and dialects since some don't use the full range of human enunciation.  The result is that a lot of characters in the IPA are absurd multicharacter strings like ʥ, tᶣ, and ŋ̋.  Even though the modern Latin derived English alphabet leaves much to be desired, the glyphic complexity pretty much limited to a worst case of m or j.  So one objective of such a universal enunciation based alphabet, besides representing the proper human audible range, while not having redundant characters, is to have the simplest set of glyphs possible.

A good example of this is I.  A vertical bar is a capital i.  A vertical bar with a half foot is a capital L.  T is also pretty simple, as are N, Z, and V.  Thees all have 3 or fewer strokes in their structure, and have little subtle interrupt in their form.  In the same way humans read words by recognizing the entire glyph structure rather than the individual letters, having the least complex, most definitive glyphs represent the alphabet makes it the easiest to learn, recognize, and write.

The amount of work required to actually scientifically define the appropriate subset of the IPA to define all distinct audible tones of human speech, combined with the most minimalist and simple glyphic representation of that tone set, is something beyond the scope of my brain.  But in many ways it is an inevitable evolution for mankind to eventually optimize our speech and writing, in the same way most of the world is currently switching over to English as a common language.  Hopefully once we solve the thousand different languages problem, we can evolve up to a much more logical form of communication in both written and verbal form.  It makes software engineers cry less.

Some math about that "digital interface to rule them all"

I remarked in my future OS post that we should standardize on one digital interface format for everything digital.  One way to solve the most pervasive problem in that domain (that the bandwidth use cases of devices differs quite a bit) can be solved with dynamic frequency clocking of the interface buses.  An easy way to solve this is to have a hot-plug handshake between devices - when a connection is made, each side sends standardized information about itself (what it does, what frequencies its capable of, etc) and the interface uses the lowest frequency both support that sufficiently fills bandwidth requirements of the interconnect.  So you could in theory have a multi-gigahertz interconnect where low bandwidth devices (like input devices) could run at only a few hundred hertz.

Some general numbers:

The most bandwidth intensive activity I can conceptualize for this interconnect, besides just going all out maximizing the bandwidth for usage as a supercomputer interconnect, is about 10 GB/s or 80 gb/s.  I get this number using 16:10 Ultra-HD at 48 bit color depth (12 bits per color channel + 12 bit alpha channel) at 180hz (this comes from the idea of using 3d displays - 90hz is a good standard that most human eyes can't differentiate very well, just like 4k is a good resolution where most eyes won't see the difference at 250 PPI from a distance of 2 feet - I wouldn't ever use a 3d display, but the standard should anticipate that).  Given that current Displayport can reach 18gb/s, increasing that 5 fold in time for this "concept" to actually matter is completely feasible.  Worst case scenario, you just include more packet channels.  Comparatively, we have pretty much clobbered the limits of audio and codecs like Opus just make compression even better over time, so I'm not worried about 15 speaker surround sound being the next big thing.

But just as a consideration, if you were to go completely audio-phile crazy and used 20 mbit/s lossless audio on 33 speakers.  That still only constitutes 660 mbit, less than a gigabit.  In practice, if you are transferring raw video without some kind of encoding compression, it will be the absolute dominator in bandwidth utilization.

So a target of 10GB/s sounds good, especially considering that also works as an interface bandwidth target per lane inside the chip architecture.  If running at peak throughput, you would be effectively running a cpu-northbridge interconnect over a wire.

If we use fiber channel for this connector standard, it can be the ultra-low-latency needed to be as multi-purpose as it needs to be.  However, while it could be ultra-high bandwidth with low latency when needed, you could also disable all but one data channel and run it at a few hundred hertz for a few megabytes of bandwidth, which is also a keyboard or mouse should need.

I could also see color coded cables indicating the peak frequency on the line supported - a 100' cable probably wouldn't be able to run at whatever frequency is needed to supply 10GB/s without active modulation, whereas a 20' cable should be able to.  This also means it makes a good ethernet standard, because realistic network bandwidth won't pass a gigabit for a long while, and getting a gigabit on a multiplexed cable like this will be cakewalk at really low frequencies.

I really don't even want to consider analog interfaces with this configuration.  It should be all digital video, audio, etc.  You could always stick an analog converter chip on top of this interface like a traditional PCI device anyway.

I also would only want one standard connector, preferrably as small as possible, with an optional lock mechanism.  Just from my personal view on the matter, but having 3 standards of usb, micro usb, hdmi, mini hdmi, micro hdmi, mini displayport, displayport, etc is just absurd.  If the maximum connectivity can be obtained on a smaller connector foot the extra few cents to build a better circuit.

Just as a footnote, real world 100gbit Ethernet should eventually hit market, and that would be the perfect standard for this.

Software Rants 6: How to Reinvent the Wheel of Servicing

In my post about new OS paradigms, I remarked about how you won't replace IP as *the* network protcol and how we should all just bow down to its glory. 

However, one thing that can easily change (and does, all the time, multiple times a week) is change how we interact over that protocol.  It is why we even have URI's, and we have everything from ftp:// to http:// to steam:// protoctols over IP packets.  I want to bring up some paralells I see between this behavior, "classic" oprating system metaphors, and the relatively modern concept of treating everything as files circa Plan 9 and my stupid ramblings. 

If I was writing an application for a popular computing platform, I would be using a system call interface into the operating system, some kind of message bus service (like dbus) for communicating with most service, I would personally use some kind of temp file as an interchange but I could also open a Unix socket as a means of IPC.  Or maybe I go really crazy and start using OS primitives to share memory pages.  Any way you slice it, you are effectively picking and choosing protocols - be it the Unix socket "protocol", the system call "protocol", etc.  In Plan 9 / crazy peoples world, you forgo having protocols in favor of a a file system, where you can access sockets, system calls, memory pages, etc as files.  You use a directory tree structure rather than distinct programmatic syntaxes to interface with things, and the generic nature improves the interchangeability, ease of learning, and in some cases it can be a performance gain since you are using significantly less overhead in using a kernel VFS manager to handle the abstractions.

If I took this concept to the net, I wouldn't want to specify a protocol in an address.  I would want the protocols to be abstracted by a virtual file system, so in the same way I mentioned /net/Google.com/reader should resolve as the address of Google's reader, you could be more specific and try something like /net/Google.com:80/reader.https (this is a generic example using the classic network protocols) where you can be specific about how resources are opened (in the same way you use static file system typing to declare how to handle files).  But this treats Google.com as a file system in and of itself - and if you consider how we navigate most of these protcols, we end up treating them as virtual file servers all the same.  The differentiation is in how we treat the server as a whole.

In current usage, interacting with ftp://mozilla.org and http://mozilla.org produces completely different results, because ftp requests are redirected to an ftp server and http ones are directed to an http server.  But https doesn't inherently mean use a different server, because it just means sticking a TLS layer on top of the communication, but the underlying behavior of either end resolves the same - packets from one are generated, boxed in an encrypted container, shipped, decrypted on the receiving end, and then processed all the same.  That is in many ways more elegant than the non-transparent designation of what server process to interact with at an address based solely off the URI designation.

So what I would rather see is, in keeping with that VFS model, a virtual mount of a remote server under syntax like /net/Google.com producing a directory containing https, ftp, mail, jabber, etc, where an application would be able to easily just mount a remote server, and depending on the visible folders derive supported operations just off that.

Likewise, authentication becomes important.  /net/zanny@google.com would be expected to produce (with an authentication token, be it a cached key or a password) *my* view of this server, in the same way users and applications would get different views of a virtual file system. 

This leads to a much more cleaner distinction of tasks, because in the current web paradigm, you usually have a kernel IP stack managing inbound packets on ports, where to send them, and such.  You register Apache on port 80 and 443 and then it decodes the packets received on those ports (which now sounds even more redundant, because you are using a url specifier and a port, but the problem becomes that ports are not nearly as clear as protocols).

So in a vfs network filesystem, determining the available protocols on a webserver should be simpler, by just looking at a top level directory of the public user on that server, instead of querying a bunch of protocols for responses.  Be it via file extensions or ports, it would still be an improvement.

Lifes Too Short, or why using anything but C++ is insulting to your users

I was watching a presentation from altdevblog about C# for games (I hope it takes the gaming world by storm, C# is beautiful).  Repeatedly, when discussing "hard" concepts (callbacks, manual memory management, threading) the author (Miguel) would qualify the difficulty, and the expected debugging nightmares and resulting ripping out of hair of doing hard things in software with "lifes too short".  Really great talk about Mono in the game world, it is a really ingenious platform especially since it supports everything ever.

I think it is a dangerous sentiment.  It basically says do things the easy way because it is a mountain to climb.  While I will agree on the intent, that developers should trade efficiency and determinism for time and ease of development.  Which is great, especially in games.  Maybe not in services.  And definitely not in kernels.  It is not a black and white case of just use the high level language to save time, it is a debate between the difficulties of building a project and the amount of work it is expected to perform.

This actually applies to games.  A single player without multiplayer can reasonably just throw C# at the problem and crank the game out quickly.  If it only lasts 30 - 40 hours, in a computing environments lifetime, that is peanuts.  But when you build a game meant to last hundreds of hours, you are actually not just costing cycles on your users machines, but you are wasting their time in exchange for your own, and the resources to power the redundant cycles, and the time of those who produce the power to run the machines.  It isn't just about developer productivity, but about everyones optimal use of time.

So I look at opportune shortcuts as a question of cost - it is why I would almost always look at C++ when thinking of some mass market product, but I'd look at Python for anything niche.  And I'd look at C# to produce business software, or non-service continually running processes.  Everything has a use case and a cost, so just thinking about ones own life being short might be a short sighted justification for paying a performance cost.

Also, I think this was the most flamebait title I have written yet.  Yay.

Document and Serialization Format Thoughts and Examples

I made a few posts on reddit in a thread about Python in the browser about why HTML / xml suck and almost anything else is better. Assuming that, I explored some options I would like to describe here as alternative document syntaxes.

1. JSON-like whitespace-insignificant functional orientation:

jsondoc {
  title : "Ninjas Are Awesome",
  p(id:"fact") : {. : "A ninja's natural enemy is a",
                  strong : "PIRATE", "!"}
  ul(id:"ninja_weapons") : { li : "sword", 
                             li : "kung fu", 
                             li : "throwing star"}
}

This I feel has the most chance to catch on - or something like it. JSON-esque maps. The only real difference between standards JSON and this syntax is the introduction of arguments on keys that act as attributes. The map element . denotes the wildcard content of a tag if you use a map instead of value for a "tag". Alternately, you could derivative from JSON standard more, and that leads me to my next example.

2. YAML-like whitespace significant:

title :: Python
list ::
- article : id=1 :
    author :: Xanny
    date :: 2012-12-15
    title : id=pirates : Ninjas are awesome
    post : id=bacon,class=words : >
        I talked about this in another fork of this 
        comment thread a bit.  You absolutely need to 
        have the capabilities of the language...
- article : id=2 :

This syntax is YAML derived but introduces attributes between the key - value syntax.   It is a little verbose with all the double colons, and in my considerations of this syntax it became apparent you didn't need to have the : > syntax for multiline descendant text bodies, so I refined it with a few changes. > is a synonym for ::, and a value section that only contains a : is considered an indicator of a descendant multiline element.  Also, instead of using minus signs as an array element delimiter, I use commas instead, since it uses comma separated values in the arguments.  Also, a keyless argument is assumed to be the id, and arguments without quotes are ,>: terminated.

title > Python
list >
  , article : 1 :
      author > Xanny
      date > 2012-12-15
      title : pirates : Ninjas are awesome
      post : class = words, bacon >
        I talked about this in another fork of this 
        comment thread a bit.  You absolutely need to 
        have the capabilities of the language...
  , article : 2 :

I like this, but I can't feel it would fly very well, so I also have a whitespace insignificant dialect of this syntax:

doc {
title > Python,
  list > [
    article : 1 : {
      author > Xanny,
      date > 2012-12-15,
      title : pirates : Ninjas are awesome,
     post : class = words, bacon : I talked about this in another fork of this comment thread a bit.  You absolutely need to have the capabilities of the language...
  } ,
  article : 2 : ...
  ]
}

Here, commas are string and element delimiters everywhere, maps are curly braces denoted. Because of a concise grammar, you would only need to escape commas, colons, and the two kinds of braces. One thing to note is this language never utilizes parenthesis. I feel something like this might easily become more popular. An alternative might be to keep the argumentative behavior from the json dialect, and reintroduce parenthesis:

doc {
  title : Python,
  list : [
    article(1) : {
      author : Xanny,
      date : 2012-12-15,
      title(pirates) : Ninjas are awesome,
      post(class=words, bacon) : I talked about this in another fork of this comment thread a bit.  You absolutely need to have the capabilities of the language...
  } ,
  article(2) : ...
  ]
}

The big deal here is that by introducing overhead of parenthesis the key : value syntax remains succinct, and it easily allows for a data serialization format to be used where you just discard the arguments syntax, or maybe even have the parsing behavior definied that arguments are just key:value pairs to be added to the constructed map (here, in python syntax) like so:

title(pirates) : Ninjas are awesome, 
>>> 
"title" : {"id" : "pirates", "body" : "Ninjas are awesome"}

The same could work for the yaml syntax, where colons past the first are disregarded (except ::\w+\n which denotes multiline text follows). Or you could use a glyph exclusively for multiline text, like & and * which go unused in yaml).

It really comes back to that vision of one data format to rule them all (that isn't the current ruler, xml) for documents, serialization, message passing, etc. Both JSON and YAML are significantly better than xml, and in keeping with that unified protocol ideology, unified textual language is a natural extension.

As a footnote, I had to rewrite this blog in completely manual html since it kept malforming the code -> pargraph transistions and inserting redundant spaces with tons of unneeded tag duplication. So the source of this should be pretty. Come on Google, get your shizzle together.

And as a final note, I'd probably go with a choice between the last two. I'd easily see this better-markup-language (bml) have extensions .bmlw for the whitespace dependent version (better markup language (with) whitespace-significance) and .bmlb (better markup language (with) brace-significance) for the whitespace agnostic.

GPU Fan Controller Thingy Post-Mortem

Forgot to write a blog post about this.  Took about an hour of coding, and around 4 reading Python documentation on proper usage of the subprocess module.

It works, I guess that is the point.  I'm not going to try to integrate it as a service into upstart or systemd since it runs fine unprivileged as a startup application.  I would try to make a nice GUI for it but Kepler GPUs don't support fan speed control through coolbits anymore (go figure) so it seems user fan speed control is now unsupported by Nvidia, which defeats the purpose of the project.

It is easy to use though if anyone wants to grab it since most GPU default fan profiles are awful.  It is just a array of temps to percentage speeds that gets built into a temperature map and every recheck time (default 5 seconds) the script will poll for the temperature, index the temperature in the array of fan speeds, and if the speed it gets differs from the current one it will update the speed.

I worked to minimize calls into nvidia-settings, because it almost certainly has a lot more overhead than my little script to incite.  So I only change the fan speed when necessary.  I found that it made more sense to constantly turn fan speed control back on every update than to potentially silently lose the ability to change fan speed (or to implement a way to parse for errors on speed update, and then to turn it back on - that might work, but I didn't get conclusive evidence it will actually error out if fan speed controlling gets disabled in some circumstances so I didn't persue it further).

In the end, it works for purpose.  It is to my knowledge the most efficient implementation on the net (even though it has a lot of memory usage with a temperature array, but looking into trying to do an array by 5c changes in temperature the math just gets more complicated than finding an index in an array of temp : speed tuples that crosses a thermal boundary to use).  It doesn't have a neat GUI but I lost the drive to persue that when I found out Nvidia no longer supports fan control.  I will need to investigate more thoroughly how AMD is doing on that front, because to me it seems completely unacceptable to not allow the user to control fan speeds - the defaults on these GPUs are absolutely awful and in my case my gtx 285 will crash on the default profile from overheating.  I even get severe graphical errors from it running near 80c and the fan doesn't even kick in at all until then.

Also, using gitorious instead of github for now, I like the site design more and prefer open source to closed.  Github seems to be getting too big for its own good in my book, and are starting to fork away from standard git in a lot of ways.

<<Add Link to gitorious repo when I reupload it to my original gitorious account>>

Ubuntu, mysql-server, apparmor, and symbolic links

If you can't start mysql-server on Ubuntu (at least version 12.10, where I encountered the error, but I imagine it might be pervasive) apparmor might (will) be denying the daemon from running due to mysql using absolute paths and skipping symbolic links on any major system directories (in my case, /var and /tmp).  Apparmor doesn't like that, since mysql-server doesn't have permissions on /mnt/data/var or /mnt/data/tmp, and the apparmor instructions only state the main system directories.

This bug only happens if you symlink directories apparmor will manage access to and from.  And it will only happen if an application, for some dumb reason, traces the absolute location of its operating directories and tries doing system calls on file operations using the paths of absolute file location, rather than using system prescribed paths such that it references major system directories through absolute paths.

The solution is to edit /etc/apparmor.d/tunables/alias and add in aliases such as:

alias /tmp/ -> /mnt/data/tmp/,

alias /var -> /mnt/data/var/,

If you do this, apparmor treats references to those directories like references to the symlink versions.  It is advisable to do this with any default dir you symlink on an apparmor based system so that the security suite doesn't bitch about random applications using arbitrary paths to get to the same folders.

GPU Fan Speed Controller Project Notes

I'm going to be writing a gui / cron job daemon for regulating the fan speeds of the proprietary nvidia / amd drivers for graphics cards.  Mainly because with Steam coming to Linux and the burgeoning interest in the platform, there really needs to be an easy way (a la Overdrive / EVGA Precision on Windows) to create a software fan curve, and the Nvidia X Server / amdcccle are insufficient in purpose for that.  Fortunately, Nvidia and AMD provide means to access the gpu temperatures and fan controls from aticonfig and nvidia-settings.  So I'll just be writing a configurable GUI frontend, a config file, and a background cron daemon to monitor it and update temperatures accordingly.  I'm basing this project off the work of 2Karl's bash scripts to accomplish the same, but they are a little cumbersome to modify (no one point reference to the names of the scripts, no table of fan speeds, etc).

I'm just writing down my game plan with this little project here:

• Using Python3 and pyQT, the latter because QT is a nicer GUI framework than GTK at this point, and Python because this isn't processor intensive and doesn't need low level optimizations.
• One python GUI application (maybe gpu-fan-control) and one background process to watchdog the gpu (gpu-fan-monitor).  The GUI should be writing to a configuration file, probably under ~/.gfancc/profileX.cfg, and the monitor should be set as a startup application that will read this file, monitor the gpu temperatures via pipelining with aticonfig / nvidia-settings, and at a user-defined interval check for temperature thresholds to change fan speed at.
• The initial goal is to get the daemon and configuration running by hand, then write a nice GUI to set it up.
• My primary GPU is a GTX 285, so I'll have this working with Nvidia first.  The configuration file should specify the gpu type, and the daemon should have some autocreate functionality in case of a missing configuration.
• The GUI will probably start with just a few data points of temperature: speed that the background will just build a one to one function out of of fan speeds.  If the temperature hasn't passed a threshold, we shouldn't update the fan speed (to avoid calls into the control software whose performance I can't directly influence).

So I'll be putting this up on github soon.  Hoping to use this as a way to get into qt, packaging in Linux, and it seemed to me like something needed with gaming coming to Linux.

Gaming Rants 2: Why WoW Went Downhill

As a 5 year WoW veteran with over 365 days played across 3 mains (rogue, druid, and priest) I quit in patch 4.0 after raiding for a few months because the game had honestly turned into complete crap and I was wasting my time on it.  Here's why.

1. Progression as a whole went to shit in Wrath, and stayed awful in Cataclysm: Progression and goals in MMOs are so essential.  I stopped playing GW2 mainly because of it.  In Gaming Rants 1 on GW2, I outlined the reasons I kept playing WoW - when I ran out of them, I burned out and quit.  In PVE, they completely changed the progression system of classic and Burning Crusade, where each raid was a progression of the last, basically requiring some farming of the previous tier to progress much into the next due to gear and skill requirements, to a system where each major content patch reset gear and progression and made each raid the singular end game pve experience.  In the raid world, instead of having tiers of progression from MC to BWL to AQ / Naxx, you would just have ToC, or Firelands, or ICC, or Demon Soul.  And that is so boring because you are always in the last dungeon, you are always a few bosses away from done, and you never have no sights to see.  4.0 was almost a breath of fresh air with 3 raids, but the real missing aspect is progression where you go from instance to instance (they are "trying" that in 5.0, but it looks like its failing - I'll get into that).  You are nearly done with one raid, and you can look forward to another to get your feet wet in.  So good.  And it meant the people who do get to the end feel really good because it means they are the best of the best.  They are hardcore, they have commitment, and everyone else, including me, can look up to them, want to be like them, to achieve that.  Today?  Everyone has end game gear, they all look the same, the difference between an idiot with raid finder loot and a hardcore raider is probably that the hardcore raider reforged BC / Classic gear over their new crap.  Wonder why.
2. They also killed PVP progression: I would have loved to get into Rated BGs, but never had the "crew" to do it.  So it never happened.  They sounded ok, but without reasonable balance (another issue) they would have always been FOTM anyway.  Without negative rating, they were just grinds anyway.  Back in my day, you went from 1500 to 2300 rating and fought progressively harder opponents.  It meant arenas were actually a ladder, rather than fake rating on top of hidden rating on top of personal rating on top of team rating on top of rating rating.  The personal rating and MMR systems ruined arenas for me because there was no progress anymore - and specifically, MMR forces you to a rating and actively keeps you there.  Progressing becomes hard after you play a bunch, if you switch team mates your MMR is almost permanently fixed.  That is such crap.  Arena participation dived from usually over a hundred gladiator teams in each bg in BC to a few dozen in Wrath and sometimes single digits in Cata.  And it was because the progression was gone, replaced with a 0 to 1500 grind to make the progress artificial.  Rating on gear was fine though, to gate pvp gear.  I think the ratings could have been lower - gear should never have been an "issue" progressing at any point, just a way to keep the percentages of who gets what gear nearly even between pvp and pve.
3. Dungeon content went from challenge to crap: They made dungeons faceroll to facilitate the dungeon finder system, so there was no fun in doing the 5 man content anymore.  In classic and BC, there were incentives (rare drops, actually gasp useful 5 man loot like Blackhands Bredth, badges, or reputation) that made going back to this content after getting raid gear enjoyable because it is fun to crush your old challenges.  And the oldest 5 mans were legitimately difficult.  BC heroics were really hard until ZA and 2.4 badge gear trivialized them.  In 2.2?  Shiz was serious.  The normal modes weren't even a walk in the park - they required CC, resource management, kiting, and knowledge of fight mechanics.  That is funny today when any random new max level player can just chain queue heroics, spam a dps / heal / tank rotation, and expect badges to pop out at the end.  No challenge means no fun, and dungeons going to crap was symptomatic of the greater issues.
4. The world became small: With raid finder, dungeon finder, portals to every city, and flying mounts, the game world went from massive and mysterious to mastered and boring.  The large game space is worth nothing if you never have to endure it or brave it.  Without guardless neutral towns, wars never broke out.  Everyone felt safe and pacified.  Terrain was designed to support player flows, rather than the classic MC walkways that would get saturated with war.
5. Flying mounts and god-guards killed world PVP: I loved world pvp in classic.  Unknown odds, wars, assassinating - it was a roguey thing and I reveled in it.  It completely died in BC.  With few exceptions of daily quest hubs, Isle of Quel Danas, Halaa, and Auchindoun, people would just be flying everywhere with no engagement with other players.  World PVP, in its raw and unsterilized ways, kept the game fresh.  In Classic, you could never be safe outside a capital city, in contested zones many neutral towns weren't even safe in the early days - it was a truly dangerous world.  And danger and challenge make games fun.  Maybe not for casuals.  Maybe not for the "target demographic".  But they did, and do, for me - and I miss that.
6. Destructive content kept the game flat and dull: When every expansion (and more recently, every patch) reset every player, not only did it demoralize the entire population by making often years worth of work worth naught, it also means that content loses all value.  Nobody comes close to raiding the old raids anymore, the old dungeons might be done once for an achievement.  There are dozens of dungeons and raids that are now dormant and empty because people race to max level - they are never a challenge anymore.  All those Cataclysm raids will now be relegated to the trash heap, with someone raiding them maybe for a transmog or achievement, and steamrolling it with a few friends.  What a waste of developer time.  If instead of constantly resetting the field, new content was injected where the playerbase was most concentrated (ex: when BC came out, Karazhan would have still been a 10 man, and would have acted a bridge from UBRS to ZG / AQ20, Gruul / Mag / SSC would have been between ZA / AQ20 and MC, and TK could have been between BWL and AQ40 / Naxx, where the major bottlnecks were... - and then BT could have been past Naxx, and Hyjal could have bridged MC and BWL).  By now, we would have dozens of raid instances, all of them "overlapping" difficulty and gear (bigger raids were give higher level stuff for less effort, but you could still have really hard 10 or 20-25 mans giving top tier loot).  Rather than 3 Pandarian raids that are gated over 3 months because they can't make raid content fast enough, and they are just going to toss these raids out in 5.1 anyway with new heroics that completely supplant their loot.  And in 6.0 all these raids and dungeons go in the dust bin as people skip ahead to level 95.
7. The sterile game world is soulless: In classic, caves existed that were empty, mountain peaks with nothing, glyphic signs referencing developers in random places, a sign in an "unreachable" zone that said it was under construction, with the entire world tree already built, entire regions of the game world not practically useful - the deep Silithid hives, Deadwind Pass, southern Blasted Lands besides Kazzak - but they existed to have something challenging in the world, waiting for players.  Same reason world bosses existed.  They should have had roaming ones, it would have been fantastic (they did have some!  some random elite packs would wander some zones, like the Fel Reaver, but I wanted raid bosses like that!).  The inconsistency in design and dangers of the unknown made the original game alive, like a real world, even when every mob stood in one place and aggroed at 20 yards.  It was so much more alive than the dumb scripting and cutscenes of today because now, your challenge is presented to you with a big sign and a raid guide page entry with a ready check, not in a random cave in a new zone.
8. Itemization has become trite, dull, and bad: One of the greatest reasons I fell out of love with PVP was a two fold reason - the introduction of flat damage reduction from resilience, and the removal of critical damage reduction.  The first basically meant spells and abilities would scale even more out of control because they could just tweak the magic "players hit players for less" button and make the experience extremely jaded between pve and pvp.  The second is just bad balance and it promotes RNG.  They were trying to lessen RNG in Wrath, and then threw a curve ball and got rid of critical damage reduction - crits are the greatest RNG in WoW PVP, and have been pretty much forever, and was pretty much the reason they added the stat in the first place - but you can see how the design direction changed.  They cared less about balance, skill, and enjoyment, and more about bigger numbers and skinner boxes to maintain subscriptions.  And humorously, it failed - they are below their peak subs, they barely got back over 10 million with Pandaland, and I guarantee it will hit 8 million or less by 5.3, because people get tired of not having challenge or goals.  And heroic modes are the most obtuse "goals" ever - harder hitting bosses for larger numbers on the same items.  That is nothing on new loot with original art, crazy itemization, and custom special effects and procs. That was loot, what is there now is a mockery of progression.  The fact gear has become extremely sanitized, and players can reforge, level up, regem, etc the items makes item levels just a statistic you want to get higher.  No more Onslaught Girdle being insanely good into Naxx even though it drops from the first raid because it was itemized perfectly - no more Dragonspine Trophey being the best trinket for 3 tiers.  Itemization became flat and boring, and that makes loot and progression less interesting.  Arbitrary stat allocation made content fun because the value of a boss kill varied with the drops beyond just who wants the new +5 ilevel pants.
9. As an addentum to why gear + pvp suck, stamina is useless: In classic, stamina was used in place of secondary stats on high warlord gear to make it pvp gear - it made you more tankey and take longer to die.  You didn't become more efficient to heal, you just lived longer from 100 to 0.  Today, nobody stacks stamina because the stat has been devastatingly eclipsed by every other stat - mainly, the revamped primary stats that give bonuses to multiple other stats.  It is still 10 hp per 1 stamina, even when damage / healing / mitigation per stat point in every other class has only gone up every expansion, maybe except Cataclysm.  Today it is so awful that nobody ever itemizes it anymore.  And that is the contributing reason pvp sucks, and why every patch balance is thrown through a blender - players gain tremendous damage and healing increases, often from gaining points in 3+ stats when each of them alone outpaces stamina, and they expect the singular stat of resilience to counteract it.  And when it doesn't they have to rebalance vast swathes of classes (if they ever do, they usually don't and just let every arena season go to the dumps) because every time they add another tier of gear, resilence isn't enough to offset damage gains, healers get stronger to counteract the damage, but it means burst is stronger, people die faster, and they live shorter because stamina doesn't keep pace.  Above all else, the scaling of player health pools has always turned the game into crap.  The only time player health was ever acceptable was in BC, when base health was high, and stamina was high on pvp gear and scaled the best it ever has since arenas started (it didn't scale as well in classic without resilience offsetting other stats).  When health pools don't scale, the game gets progressively more zergy and ping-pong like, and it gets stale.
10. The game is still pay to play: After buying 3 expansions, I don't get why they don't just make non-current content free to play.  They already let people play forever at level 20 - and I know hardware.  It costs them dirt to support players on their servers, and they make runaway profits from the game and so little goes to maintain the servers.  They could have tons of free to play players at lower levels, maybe on legacy servers.  They could be buying item shop loot like pets and what have you, but they are not.  The reason I couldn't keep playing was not just because the game turned bad, but if I'm paying $15 a month for something, I expect to get my monies worth - and that required, for me, a truckload of dedication.  I didn't want to keep it up, but felt conflicted staying subscripted and effectively wasting my money if I wasn't playing the game.  So I did the obvious - I quit.  And I won't pay for almost any game anymore, because free to play is such an earnest budget proposition.  I have a large enough catalog, people are awesome enough to make enough mods and custom maps for what I have, and I have too little free time to want to pay money to waste my free time.  I'd play WoW again, despite all its modern day failures and faults.

That's enough for now.  For all its worth, the modern game is almost always better than its predecessor versions - I wouldn't really want to play classic again, combat is too slow and flat, mana is terrible on everything, and while itemization was better back then, stats themselves were boring with only crit and hit mattering.  Random resists on spells and abilities were awful, and it is good they were taken out.  Random low percentage procs also sucked, and they are not as common (cough, 5 stack taste for blood, cough).  Really though, if you retrofitted the better game-play decisions from recent expansions into classic or BC and balanced bosses accordingly, without giving every player the kitchen sink of spells so they can do everything, the game would probably be amazing. (NO ROGUE AOE, NO HUNTER HEALING, NO MAGE PERSISTENT PETS... etc).

Gaming Rants 1: Why Guild Wars 2 Failed My Expectations

So I bought Guild Wars 2 after playing a beta weekend.  I was hoping it got... better.. that what I got in the beta at higher levels.  I only hit around ~15 in the beta, and ~25 on live.  The problem in both was that I just ended up grinding the same areas over and over trying to level up, didn't enjoy it, and just quit. 

So here are my reasons for why I didn't like GW2.  Not saying someone might be completely opposite and have loved these features, but it really gets at what I want in a persistent world nowadays.

1. Progression didn't matter: the gating system of leveling in traditional games was / is meant to give you a sense of progression of power.  In GW2, this never happens, because you get downleveled or upleveled anywhere you go.  You can never go back to the newbie area and one shot mobs because you get turned into a level 5 when zoning in.  Immersion wise, it basically implies you can go from killing world dragons to dying to a bunny.  It made me not care about getting anywhere in the game, so I.. didn't.  Any MMO I play basically needs to give me a reason to keep playing... (con't)
2. Personal character combat and playstyles were dull: I tried a bunch of classes up to at least level 10 - the warrior, guardian, engineer, necromancer, thief, and elementalist.  My highest level characters were my guardian on live at level 25 and my beta elementalist at 15.  All these classes had the exact same setup: 1 - 3 were damage abiltiies, spam on cd and rarely ever requiring thought.  The last 2 were usually 20 - 30 second cds you also used on cd because why not, they are only cooldown restricted.  6 is the heal that you also spam on cd if you aren't at full hp, or if you get it for some other effect... you spam it on cd.  7, 8, and 9 were usually the highest dps abilities you could get, that you spammed on cd.  And 10 was an "ultimate" that you would use as a get out of shit card.  One ability you don't spam.  Yet you would often spam it!  If it was an offensive cd, you spammed that shit like it was in style!  You never had to stand and cast a spell, you could cast while moving.  There were no spell interrupts, so you never had to worry about casting in someones face.  The only CC effects were tiny stuns (tbh, long cc is dumb, so this is fine) that didn't have any timing element to them.  Without healers or tanks, you basically spammed damage into whatever is nearest you until it died or you died.  Some weapons sets on some classes could be "tanky" or "healy" in that the 2 - 5 spells would often be tank or healer centric, and you could get heal / tank 6 - 9 spells, but you could never actually "tank" or "heal" as a role, you were just a shitty damage dealer with tanky / healy utility.  So no matter how you sliced the game, you were spamming abilities on cd (maybe if you were microoptimizing your playstyle you would stack debuffs to maximize dps, but in general it was just button spam till things die nonsense).  You had a dodge, but because of latency and the horrible animations (honestly, monsters flail around in idle more than they do when swinging) and no delay between animation start and effect firing, you basically just randomly rolled around hoping to avoid something.  In GW2 favor, you could usually roll a projectile in flight to immune it, so I found it actually useful versus enemies at range.
3. PVP is a spamfest: With everyone being a dps class, nobody have legitimate damage "rotations" or ability priorities, and there never being a tradeoff or choice in how you engage in combat, it becomes a cluster fuck of throwing every ability you have into a pile of bodies, trying to get away, and then doing it all over again.  Since synergy was... limited, at best, you would usually only care to beat on the same thing as everyone else hoping some debuff benefited you.  With so many arbitrary weapon combos, and pretty much every combo producing a different set of 1 - 5 spells, any one player has way too many raw abilities to memorize from other profession + weapon combos, but that gets into the next point...
4. There is absolutely no diversity of choice in how you play: One weapon set per class was, in my playtime, always completely better than every other at some role.  A guardian / warrior not using a 2h sword for mobility / damage was dumb, an elementalist without a staff for aoe was bad, etc.  The only semblance of choice was when classes with weapon swaps would pick a secondary weapon to supplement their only true primary weapon choice, because they could either pick another weapon to have more faceroll damage ability spam by constantly weapon swapping, or use something tanky / healy in case they get focused and want to tank it / get health back.  Also the choice between melee or ranged weapons did matter.  But in pretty much every profession, one set would be definitively and mathematically better than everything else, and everyone just goes with that.  So all that weapon combo diversity goes out the window since something has to have raw numerical superiority at whatever you are doing, and since that is always trying to zerg damage into something, guess what wins out.
5. Groups don't matter and players exist in a vacuum: Monsters scale off how many people are engaging them, so grouping up has no benefit besides getting unique loot per kill.  If you don't kill things faster, you aren't getting any benefit, but since xp isn't split, it doesn't really devalue grouping either.  It is a completely neutral proposition.  Better than Borderlands 2 in that regard!  Not better than WoW circa 2004, or freaking Super Mario Bros circa the 80s.  More players should have tangible benefits since it is hard to get people to play with you.  Without a trinity, or any class synergy (hell, something like affliciton locks and shadow priests having shadow damage dot buffs is synergy that GW2 is severely lacking, sure, almost every class has a debuff it provides on some weapon that buffs everyone else, but it is rarely up, and barely a damage boost, and not coordinated since everything else is already inherently zergy) it feels like a single player game with other people.  There can be no really competitive and interesting pvp or pve to me, because I can't get into the lone wolf gameplay of CoD instead of Team Fortress, or Fighting Games instead of coordinated synergetic gameplay like WoW Arena's RMP.  I feel like persistent worlds require an aspect of each player being an imcomplete package or else the gameplay becomes boring and playing with friends loses flavor because supporting the weaknesses in others with your strengths makes these games.
6. The plot is obfuscated and inconsistent: By level 25 I still have no idea what is going on.  I was playing an Asura that hunted down a green armored man that killed some rookies, who was in league with an evil faction of Asura.  I then went into a set of forests and caves consisting of friendly unnaturally smart trogg rip-offs and got bored.  I had some "epic" quest where I had to open a portal to a ruined city on an island that was overrun by the old dragon the main plot has you kill, but as a storyline the whole thing falls flat for a few reasons - 1. the character I play is personified just enough to make me not feel in the story but not enough for me to see my "character" as an agent in this world.  Also, its an MMO, you will never have immersive characters with agency because you are too busy engaging with other people treating your avatar as you.  Strangely, SWTOR didn't have this problem, because the story had a consistent goal throughout that I could care about, the scale was sufficiently epic for me to engage with, and it came in discretized chunks where each ending brought new beginnings (just an example, Sith Inquisitor is training -> foundations of feud with Thannaton -> murder plot -> artifact hunt and power acquisition for 10 levels -> betrayal by master and Thanaton -> hunt for more artifacts and power -> kill Thanaton.  Each part led into the next, and made me care about going forward, because I always wanted to kill some jerk in the Sith at some point and looked forward to it.  The enemy of Guild Wars 2 was never put in my face for me to care about, and he never did anything I could personally want to kill him for, so I just didn't care.
7. The streamlined experience makes it flat: Waypoints are completely contrived, and you can just fade in and out anywhere in the world whenever you want.  So the world had depth until you find waypoints, and then you are pretty much done.  Since almost everything important is on the map, there is no exploration (with rare, honestly well done exceptions like the Lion's Heart cave and a few aerial jumping puzzles) and while some zones are thematically consistent, some are just too big and become a colorful mess of inconsistency.  
8. There are no long term objectives: In WoW, I started playing and saw Bloodfang Armor - and I wanted it so bad that I played for a year to get it.  By the time I finished the set, BC came out, and I wanted gladiator and a netherdrake so I played shaman / rogue with a great player in the first season as a rogue and got gladiator in 2s.  Then I wanted to be the best rogue on the server and in 2 seasons was nigh the undisputed best player left on Agamaggan, mainly because Gummi Bears and their entire rank 1 crew left.  In Wrath, I wanted to try playing GM, so I did.  After that I wanted to try pvping hardcore again, but never could get the players on the shitty server.  That pretty much eventually led to me quitting, combined with the reasons WoW fell apart for me.  Hey, another blog idea!

Those 8 are pretty much it though.  If the combat was fun, grouping was beneficial and engaging, and I had goals, I would have liked it more.  The last day I played it was basically just logging in to chat and attempt to engage with my only friend also playing near my level before just getting bored and quitting after gaining a level.  I had no reason to keep going, I had my entire skill bar and every weapon unlocked, and the gameplay wasn't fun enough to keep me going.  I could farm BGs in WoW for years or grind instances forever because I liked the way WoW played in the old days.  I just couldn't get into GW2 personally, so I gave up.

Software Rants 5: Architecting a 21st Century OS

In multiple posts on this pile of cruft and silliness I've spoken about how I don't like the design of most modern operating systems.  I've even written some poorly construed ideas on the subject.  So how would I go about building something that, comparatively, has taken billions of man hours to create in a couple cases? (although subjects like Haiku make me feel a bit more optimistic). 

So I said microkernel.  This theoretical system has one goal being the minimization of kernel-space code - and anything platform specific that doesn't directly tie into the ability to execute software should definitely be userspace.  Linux has the philosophy that one source archive needs to compile with a configuration to work on any system ever, from the lowly phone to a massive supercomputer cluster of thousands of nodes.  You will get tremendously different binaries and KO sets depending on your configurations.  The difference between a PPC64 supercomputer all necessary modules compiled in kernel and a udev based dynamic module loading ARM kernel for phones makes them barely resemble each other - besides some basic tenants, the same command line arguments, the idea of loading an init payload, initializing devices, and using the same (similar?) memory management and scheduling algorithms (there are some builds of the kernel that use different ones...), the resulting system will be using entirely different tracks the source tree.

I disagree with that, in that those aren't the same software projects anymore, and keeping them in the same source tree is the exact opposite of the do one thing right, well, and concisely UNIX ideology.  And that is fine, because it means the Kernel is ultraportable, you can get an upstream codebase, build it, and run it on practically anything, and with a source clone you can customize the configuration to suit almost any use case. 

But there is also a reason Linux takes multiple levels of management and a ton of organization behind it - they have everything from device drivers to hardware translation layers to language documentation to binary blobs being merged into one program and it becomes impossible for any one person to understand or conceptualize such divergent tech.  I don't call myself smart, but I do think smart people are averse to complexity where possible, and this is definitely one of the places I see ample unneeded complexity.

So here is my proposition : a microkernel core, designed in and written in L, that only handles virtual memory management, preempted process scheduling ("virtual execution"), a virtual filesystem abstraction layer (so that aspects of this kernel can be mapped into the final root filesystem without awkward dependencies on user space filesystem managers), a virtual socket layer (for communication purposes, the sockets themselves would be managed in userspace, but the kernel would initalize this system so that some user space network daemon can manage socket communication - but the kernel itself will be using sockets as well to communicate hardware state and with the daemons it connects to!), and a hardware abstraction layer that allows user-space privileged daemons to take control of hardware components (disks, video devices, busses and usb hubs, etc, basically everything divergent from the RAM/CPU mix*).

*: I would be interested in exploring if there would be a way to have the memory controller in user space.  In that the kernel could start and initialize itself only in processor space, but it seems to be incessantly complex... you can't even establish any other virtual systems without system memory to use.  It would have the same issue a filesystem host would have, that the kernel would need to, in advance, start up this daemon, and then fall back to its own boot procedures.

The traditional boot philosophy is on -> firmware is payloaded and initialized by hardware -> firmware does primitive device detection and scans persistent storage buses for something with a recognizable partition table and payloading a bootloader from such -> giving it a "direct media interface table" even though that is Intel jumbo, with some hardware memory mapping used to provide devices.

UEFI is like this, with more complex device loading (including a 3d driver of some sort with the graphical setups) and the ability to scan not just partition tables but fat-32 filesystems for bootable executables.  It is pretty much an OS in and of itself considering how it behaves.

In my grand delusions, we could scratch the unnecessary parts - the important aspects of a boot cycle are device initialization, error detection, and searching for a payload.  Device initialization is already pretty well "passed on" to the resulting OS.  Error reporting is more complicated, because you are dealing in a world where the most you have access to may be some primitive bus signals to indicate problems, such as beep codes, error panels on the board, or keyboard signals to blink the ps2 port.  BIOSes and EFI boot procedure are obscured by platforms - each new chipset does things differently, merging more parts onto the cpu, or handling device channels differently.  In terms of payload searching, EFI actually does a really good job - given a boot device, and a system partition on it, load a binary.  No need for traditional bootloaders (which is nice).

On -> check for cpu / memory errors, on error try to signal all devices with some special hardware failure signal and a payload error descriptor.  Devices would need to be designed to handle this signal if they have some way to display errors (vga, keyboards, sound systems) or drop it.  The expectation is that "on" means devices power on, and the independent units like network controllers  come online at the same time and await the cpu to initialize them.

Check errors -> initialize devices.  Given no catastrophic errors, check if any other device has errors, and if not build a table of devices to provide the payload device to boot.  If a device has an error, broadcast another error payload signal to let anything capable alert the "outside" of the problem.  But don't stop booting if the error is recoverable, just indicate a failed device in the provided "table".

Initialize devices -> payload.  You have a table of devices, and the firmware needs to be aware of where we can find something to payload.  In terms binary byte ordering, that is an open question - we would prefer big endian for readability if it doesn't incur a hardware complexity cost.  Nobody should almost ever be working with binary data representation at this scale anyway, but if we can do Big End without circuitry overhead, we should, otherwise, keep it simple stupid and use Little E.

Since we have effectively an integrated bootloader, we need some very simple file system to just store binaries.  Now here a point of contention - have complex file system analysis machines that can read our "stock" FS type (which would absolutely be in the btrfs vein of COW auto-compressing snapshotted filesystems) or have a discrete filesystem just for the bootloader to read off.   We need to think of MBRs here - and logical and physical sector sizes.

I want to propose variable sized sectors, the same way there are viable sized memory pages - 4K memory and 4K disk are great defaults, but you can always use larger contiguous blocks of.. both.  In both cases, page size and sector size implicate overhead on the managers of these spaces while having larger sizes implicates overhead of its own when boundaries are not nicely met.

For one, traditional storage media just can't have variable sized physical sectors.  Having different logical sectors seems silly because it is a great simplication of work to have 4k sector sizes in both cases.  That will continue to work well for some time, and a sufficiently smart operating system can optimize sector utilization to minimize wasted space.  That is a device driver problem, though.

In terms of hardware memory pages, I still think swapping has value even if traditional desktops don't need it anymore - too many problems just require tremendous amounts of memory to work with beyond the bounds of traditional computing concepts, and we like embedded systems with low memory.  Even if you could theoretically implement something akin to paging with file system abstractions (writing to and from disk once you approach the physical memory limit) having the option there has proven to be worth it.

So page sizes - we don't want too many sizes, and we want them to scale nicely if possible.  This would require research and insight I don't possess, but you definitely want to support variable sized pages.

So we assume disks have 4K sectors, pages are at least 4K, and we may or may not have a dedicated partition standard for bootable binaries with some specialized file system.  We will need a disk partition table, and if were mandating 4k sectors, we have 4k bits to store info on the device.  I like how GPT has a backup copy of itself, so we want one of those, so its really 8k bits total, the first and last sectors.  In terms of sector addressing, I'm starting to think about 48 bit as an option - the overhead of 64 bit just for 64 zetabytes seems unnecessary when 48 bits gives an exabyte of storage.  Currently, the largest storage device is approximately 4tb, up from last years 3tb, the year before that at 2.5 tb, the year before that at 2, the year before that at 1.5, etc.  So if we go with a terabyte a year (which is about right for the last 5 years) we have a few decades before this becomes an issue, and we can just add in a 64 bit addressable 2.0 version anyway since we want userspace drivers.

Similarly, I'm not entirely sold on pure 64 bit cpus either.  The real big argument for large memory cpu sets is that server farms with shared memory need to address all that space, but you still get 281 petabytes on 48 bits.  I'd probably make the 1.0 of this system 48 bit, and make sure the language and instruction set inherently are valid converting to 64 and maybe even 80 / 96 bit.  This is actually really easy in the conceptualization of L, because you would have pointers as objects, and their size would be compile specific to the platform pointer size.  Integers are decoupled from words, so you can create ints of all sizes from 1 to ~16 bytes (I don't see why you wouldn't implement 128 bit integers on a modern architecture).  This also brings up the potential to just do the 64 bit virtual 48 bit physical word sizes Intel uses, but there is a translation in there that adds unnecessary complexity in my opinion.

So 48 bit CPU, 48 bit sector addresses, 4k standard pages and sectors.  Big endian unless there is a complexity or performance hit, in which case we just use little.  We want point to point memory rather than dual channel, and I already talked about CPU architecture earlier - having a heterogeneous collection of registers, alus, fpus, simd cores, etc to run dedicated parallel instructions.  This way the traditionally discrete gpu and cpu cores (even on a shared die) can be more tightly integrated.  You could also use one single ram, and not have to reserve it for the gpu cores.  As long as the instruction set is designed around supporting SIMD instructions for the parallel processing cores, we should be sound.

So we have our payload, we run it, and we set up a process scheduler that I will look into more in the future (really, this is the kind of decision that takes a ton of reading up on, to figure out the best preemptive scheduler for purpose, but CFS I guess is the industry standard).  We have the virtual memory, and we need some way for the kernel to initialize physical hardware virtual memory mappings.

So we don't want the kernel explicitly dealing with device management, but it needs to initialize device memory, so we can just add that into the device table the firmware provides - one of the hardware signals can be for the memory map size.  The kernel can then map memory addresses at this time to the given tables memory requirements for applications, and when an application access device hooks, it also gets control of the virtual page table referring to that device.

So scheduler and memory is up.  We want a virtual file system now - no hardware folders are even initialized yet, but we can provide a virtual hardware file system for device access.  This would be an elevated privilege folder - as an abstraction, you can have device control servers open devices for writing to take exclusive control over them, and the writable file is their memory map.  I proposed a file system layout earlier, so here we are talking about something akin to /System/Hardware/*.  The VFS would propagate a node per hardware device provided, the folder would require level 1 permissions to access, and once a device server has control of the virtual memory map with the device opened for writing, the only thing other servers can do is read it.

So this virtual file server needs the concept of permissions at the kernel level - we want executables running to have a level of privilege, beyond the bounds of kernel vs user mode processor execution state - this is a software privilege, where level 0 is the kernel, level 1 is init and systemwide access services and daemons, and has a "pure" view of the file system.  Level 2 would be the usual run mode of programs - restricted access to system essential files, restricted views of the file system, and each application would have device privileges specific to it given to it by a level 1 service.

Some examples - the vfs, socket layer, memory mapping, and scheduler are operating in level 0 kernel mode.  A gpu driver, device controller, usb host, a file system daemon, or a dhcp host, smb server, virtual machine service.  You want two levels of indirection here in most cases - a sound server to manage audio device servers running, a display server to manage display adapters running, a network server to manage networking devices, etc.  Access to these servers is restricted in the vfs and vss through executable privileges, probably by a level 1 executor server that wraps kernel level execution behavior.  Basically, kernel sets up permissions that the executor server manages, since level 2 permissions can't access level 0 directly.  Level 3 is the "pure sandbox" - it would be started by level 2 programs (like a VM) and has no device access directly, and only has its own restricted view of the vfs and by default has no write permissions outside its execution context.  You could thus host users from level 2 session managers (maybe run by an administrator) and they would be unable to manipulate the outer system by design.

So we have 4 permissions levels right now, and you could theoretically add more and more to add more levels of virtualization.  A virtual machine could thus just be a level 2 program that pipes commands from level 4 devices through the virtual memory map of a level 3 kernel into the level 1 devices above it.  Very slick I think.

The other major revelation is the idea of display management.  In the absence of dedicated video hardware to control by a level 1 daemon, the video daemon could itself emulate a video server.   Or you could set up the userspace where Kernel is level 0, device controllers are level 1, device managers are level 2, and user applications are level 3, so that user applications never interface with devices directly but only through abstraction layers.  I actually like that somewhat more than the other model in some use cases.

And then of course the traditional server model can just run everything at level 1.  It isn't kernel mode, but it has device mapper access so you can set up the traditional ultra-fast interconnects.  This alleviates the problem of FUSE and its ilk in Linux because you don't have to emulate the device controller without any hooks and pipe them, you inherit devices into user space.

So I'll talk about the device / manager services more next post.