Transcript

I have been the CTO and founder of Cloud9 I have in the last two years thrown myself at the GPU and using that thing for UIs and doing interesting things. And, the solution that I've arrived at is called OneJS and I want to talk to you about that today. How many people here are graphics nerds and how many here are language nerds. Graphic nerds? Language nerds, good language nerds are good. Programming language nerds are good. So OneJS ... why?! Very simple reason, the GPU has vast creative powers. The programming languages ‑‑ the programming languages that can run on the GPU can determine the color of a picks elwith a program not with CSS background color or image, with an actual program. And what that means is that there is an enormous amount of yee creative possibility that is enabled by the use of programs on the GPU, the only problem is, it is really hard to do right now 's like a huge air grab between your UI programming environment like JavaScript and the programming environment on the GPU, which is GSL, it's a C‑like programming language that came out of Pixar, kind of sort of, but it's actually kind of a pretty language. It's pretty much just like JavaScript, if you take out the objects, and you add instead of numbers you can do vectors, so you can do A plus B, and A is a vector and B is a vector, and the result is a vector and every component is added in parallel like Sim D, and G P Us are everywhere nowadays, everybody here is walking around with a GPU in their phone that's more powerful than anything we had on PC a decade ago here's my presentation running on the iPad, it's sixty frames a second on the web. It's rally really nice that we have this kind of GP U power vail northbound lane our mobile devices, web GL is still kind of Shiy on Android, actually really Shitty on Android, it's hopefully going to get there some day when they fix the GPU reset bug. But, the main problem is that GL and GPUs are really really hard, they're very difficult and very, very elaborate to make them do things, time just scrolling a bit of GL of web GL AP I want here, it's incredibly chatty, you have to do a lot of set up to get the GPU to do something. So, what is the essence of the GPU, how does this thing work? You have to look at it in a way that you can give it a program, a bit of code that is computing fix elcolors, for instance, and, you feed it data and variables. And the variables are reference in the the program and the data streams through program in parallel and is then outputted to fix els, I'm not going to go deep into the web GL explanation, Steve Whiten has brilliant demos if you want to understand how this stuff works, go look at that. I want to focus on how to remove all this Kruft between programmer and the GPU. So, one of the things that GPU can do is massive par hash legaltion ‑‑ this presentation is running in OneJS because doing a talk about graphics without using your own graphics engine is kind of lame. So, the cool thing about the GPU is that you can feed it data and have these little programs do your styling for you, so essentially you par ram tar rise the data and then have the little programs on the GPU do style switches, do style changes, in this case I made a very simple blob of vertex data, the little triangles that go into the GPU, on every one of the triangles is a letter, is a character of a font, and in that big blob of data I encode the XY coordinates, in this case the word color is compute. And that means that this whole thing is taking like 0.2 milliseconds of draw time in JavaScript and run entirely on yourBU, there's pretty much nothing going on in JavaScript. This parametertion of styling on to GPU, I think is going to be very, very important and it's going to find it, hopefully a new way to style things on the web that CSS has sorely lacked. So the parialism idea or this picture is trying to illustrate (Parallelism) you have to imagine every fix elhaving a function call that computes the function of the fix el. In that case, the functioncal takes some arguments an X and Y coordinate, exclamation parameter T, and it decides the color of the text or the color of the "I" el, it's same concept. So, we all love of brad vector talks Whoo ‑‑ that guy's awesome. He had an interesting point about first principles, it was a bit puzzling to me, first principles, what are those things, this time around I try to find my first principles of what I'm trying to build. And, one of the core concepts that I tried to put in to OneJS is programmable beyond configurable. Programmable is better than configurable. We all know the declarative space where everything you want needs some implementation from an engine designer. We want another parameter to do something in the engine, if you do it declaratively, then those declarations have to be excueded by an engine that somebody has to build. We all know CSS spec stuff where is configurable. But configurability is an enormously limiting factor on the creative potential on what we can do with computers. But this concept, if ‑‑ so, I usual live apply like this, if you find yourself making something configurable, consider making it programmable, so instead of making an API, make a Compiler. Right. Instead of having a flag using an expression. At the very least an expression and better general function execution. And I try to put this in the details of OneJS, another one is that style and behavior orer reducible. By this I mean ‑‑er reducible ‑‑ ‑‑ Irr deucible. I'm encoding a red rectangle, I have to say it's red and I have to say it's a rectangle, I cannot make a red rectangle out of anyless data. Red button is the same thing. Everybody here has probably made a button,e in HTML ‑‑ that logic is Irr educ Ibl oh, if you're trying to introduce a button, the language is completely Irreduc Ible T only way to make UI more compact in the power of expression is by composition. Make composition the core aspect, programmability, composition, because style and behavior are Irredu doctorible 's no pipe dreams about let's make the grand unified UI kit, because someone is going to want the slide tore do something that you didn't program into it. So declarative is always going to break down. And, this is something new that has arrived with theGPU, inefficiency is okay when it's parallel. Many of you might think the GPU is so fast, well, it's not fast,it's parallel. Which means that it can run thousands of little programs that run relatively slowly but you can still get a vast result out of that that you wouldn't be able to do if it's a serial language, because JavaScript has pretty much been a serial language, we're all accustomed to four loops are expensive, well, if you do it in par are legal you can do quite a lot of things. So, (Parallel) so how are we going to attack this problem? So if you want to make theGU programmable in a way this is more accessible from within another programming environment, because we oar not doing declarative structures, we're doing programming environments and Compilers, we have to fuse GSL ‑‑ GLSL into JS as much as we can. It's a language with strict typing you have a four vector you have a float, and you have an array of this and Ie an array of that. And JavaScript, actually sin tackcally is fair live close to what GLSL looks like in terms of C, what I try to do was fuse these languages at the hip, probably. But, fuse them in a way that you can leverage the same syntax whether you're writing a shader or whether you're writing something that run in JavaScript. So, by the way, these are syntax shaders, syntax highlighting is kind of lame (Laughs) which means that I actually throw the AST on to theG and you can write little fragments that style your code. I wrote a code editor with my previous company, this was just like you want this, I don't know why you want this, but it's awesome. It's not about why, it's about the potential of exploration that you can do. If I can make an error jitter in my line of code, I can add something that is useful. This is not useful, but this looks pretty. So OneJS is JavaScript, I try not to explain JavaScript, that's annoying because I have to explain a million times why I did something. At the same time, trance Compiler I tried to fix some of the problems that are in JavaScript. And one of the things is I finally got rid of semi colons, it's very important. Visual noise, and I don't like visual noise. But what you is to do to get rid of semi colons is make it new line sensitive when it reads parens or square brackets on the next line, if you make it new line sensitive, at that point, then you can get rid of semi colons, no, semi colons are there for a reason because it blows your foot off if you do it incorrectly. Oh, now it's a cull of a sud b, shit, everyone that's learned that ‑‑ well, if you can fix that, it's no problem. And I tried to take the Ecma script 6 and 7 all the things I could find in coffee script, Julia Lisp, I tried to fuse it down to something that wouldn't break JavaScript but would still add the stuff I needed. If you're fusing two execution domains, there's something that is very useful about symbolic expressions, we'll get to that next, I'll first show you the class, because classes are another very important aspect of the language. Because, offed ‑‑ because of the composability if you have a button, and you want to specialize it to do something different, you need to be able to inherit the class and change one parameters, that's the most come pressed rep evennation of that idea. Button that's a different color, we'll take this button and change the color, that's sort of like a compression algorithm in your expression of code. Class is super important, I took ‑‑ script I changed one thing, they're not functions anymore, I hate dot prototype ‑‑ nobody gets it, it's horrible. So I just removed that prototype, my classes are just objects, I don't have a new operator, I have dot new. I tried to explain it so many times and it's just not going to work for me. Doesn't mean it's not compatible with JavaScript, because it's a super set, largely, except if you like to do function call with parens, on the next line, then that's your problem. So, the interesting part is we have struts and vector in this implementation, and we have symbolic expressions, symbolic expressions is an idea from lists where you quote and you write something in code that looks like code, but it's actually an ASD node in the language, so here on the right you see X is Y plus one, but with a colon if front of it. The colon is the quote operator, the colon eats the whole expression to the right and turns this into an Astnode, you can now start to treat expression as usable objects in your language. And heres an example of how you draw a triangle. This is not library that abstracts away what you do, these are the actual it's the actual data, the differentiator and fixable differentiator, there's no matrix stacks, there's no nothing, this is just thest seasons of what the GPU needs to draw a fry angle. You can make a vector two array, so you can use that in JavaScript, this compiles to a flow 32 array expansion, so you really in JavaScript here, I could a console log in the middle, this is just JavaScript, except it has a few little syntax tweaks, another one of the syntax tweaks is the identifier curly syntax, because I always get very annoyed when I have to express tree structure in a sort of computable way that I have to go toJON or I have go to XML, which ‑‑ (JSON) which is horrible) it shouldv a construct to express these tree structure in a way that you can mix et with your language, like a templating structure. I could put a four loop in front of draw and it would do it ten times. So, this identifier curly syntax is actually a call, it's object.call, and the curlies are a function, so this just says device.call wrap with function. That's what it expands to, and it passes in to this, because that is another important thing, OneJS does not have global, it just slaps this dot in front of anything that would be a global, which means you access properties on your object without using this dot, which also makes your code much less wide. So here we have a triangle, and you can see the mesh with thementty brackets, that pull in the data from the JavaScript space,I adopt have to write attributes and uniforms, it's generated for you (I don't have to write) by looking at what you're using in the shader, it's very simple, Compilers do it all the time, something that needed to be done. So, we have ‑‑ we have device and we have draw, and draw is just a draw call, but if we can inflater a device and make a scene and we can inherit from draw and make a layer we can use composability to use actual scene drafts, this is terribly ugly because a week ago it did nothing. This is a very simple example of how you do nested rectangles with matrixes, right. Scene graphs work in a way that you have a reck tang that will has a matrix and then something inside it that has a matrix, and matrixing are multiplied together to get the relative offset, that's how the CSS transforms work, that's how the browser does it. That's how you do 2 D nested transforms by Concaptina theing the matrixes ‑‑ simple ‑‑ just implements the multiplication of the matrix, you parameterize that matrix with WH and SX, I could call it top, bottom, left, right, all those kind of things, I like 'em short, anybody can change that and make a long name for it. And, here you can see instead of background image, you actually just make a texture fetch and you do background.sample at a certain coordinate. So this rectangle can be parameterized any way, I can twirl it, flip it, I can do anything with the code, it's programmable, the styling is programmable, the styling comes from your class model, I can do inheritance, I can do CSS‑like style class that are just applied, just apply these properties. Because shaders are now composeble, I can reference an object, I can reference another function on my class, essentially shaders have been fused with the class model. I have to hurry up. So interactivity, this is really lame, but it shows you a very important aspect of changing stuff dynamically, so, on mouse left down I change the symbolic expression of X to mouse X and Y, so I lock it to the mouse. Essentially the Compiler goes you've changed an expression, let's recompile your draw function and send it to the renderer, then the renderer does the stuff with the mouse coordinates, so I've essentially told the render today that bind, this is expression day the binding in the renderer by using a Compiler. And, it's rally cool, you can change any part of your shader function and variables interactively. I'm still working on the animation system, and all that, but we'll get to that later. So these slides are written using a class DSL because you have the nesting curly identifier syntax, you can impresent a DSL by putting ‑‑ by making a bunch of nested classes, you can do class deck and a slide is in a deck, this just means that can only exTennesseeuate a slide because it's only known on the class deck. So if I don't put the slide thing p deck, it's just going to say undefined. So now I have a domain specific language that is encoded in a class hierarchy, so, you know, I don't need all these really fancy HTML things, I can make a domain specific renderer for these slides, the slide are aDM tree,but a different kind of DOM, it doesn't reverend all their children and I'm hiding layer, a DOM tree is having a custom draw command I'm going to render the current slide and nothing else, I know slide ID, you specialize the draw I differential,instead of being C + + parameterized, it's a programmable engine that has a per node draw calls that you can just change. So, the stuff not ready for use at all, three months atwo let's do JSConf 2014, it's a nice dead line if you're building Compilers to get somewhere. It's rally far to hello world when you start with a language. But, I still need to refactor ‑‑ this is a type inferencing Compiler, it's kind of my first type inferencing Compiler, I need some refactorring time on this thing. It's multithreaded, so the code that you saw runs in a worker, and the Compiler sends the draw calls, the draw code to the main thread which is really nice, because I don't get ‑‑ or I get minimal garbage collection hick ups in my rend E. all the user stuff that's doing data generation for data visualization or whatever it's doing is not in the render thread, it's perfectly fine had a work e. It's fairly large, it's 130 K over the wire, I'm compiling everything in the browser always, I'm using browser based local storage to cachee cache the output. I don't like server based compilation solutions, it limits the composability of your programming construct if you is to precompile everything, so it always compile in the browser, it's actually really fast. The whole slide takes about 450 milliseconds to parse and compile the whole thing, which is okay for about ten or fifteen thousand lines of code in JavaScript. And if it's cached it starts up much quicker. So IOS is great, I was like, yeah, year, web GL, bam, it's work: Android is web GL, web GL, it doesn't work. The state of web GL is not that great for mobile yet, this is great that it works but on Android it's not ‑‑ you to manually enable it in chrome or you can freeze the whole operating system, that's why it's disabled,it makee makes sense it's disabled, this is a future facing technology, this is not something you want the use for a corporate client any time soon. So, the future is constrained layouts, and I have it running and because I have the Compiler I can actually, you can just write a constraint straight in your code block like this. And it can run the solver on the scene graph. So that's coming. This is all not responsive and has no layout engine because I haven't gotten to that point yet. But, I mean, I got a way with it because I have essentially expression layouts so you can do parent width minus a hundred, so position something from the right. It doesn't actually have layout engine, but it will. And I'm going to build a live code editor in this thing itself, this is a great way to build a code editor if you can do this kind of text stuff. But also the reason is that I need to bolt editor on to the ASD directly otherwise the editor is going to be this dumb thing that looks at regular expressions which is not you want to be with the code editor, you want the full ASD, and operate on the SAD so it needs to sterilize into proper code, which is what I'm going to work on next and it needs to work on this thing, the Oculus at 75 hearts, I'm hoping the browser guys can get that done, I'm fearing it might not happen. It has almost no AI expotion, so I can writtene run it in node just as well as in the browser. And I'm going to, you know, make a prototype of a new Cloud9 UI, I think it should be easily doable. It's on GitHub, you look at it and not complain about it. I mean, you can look at it and do stuff with it if you want, not something I'm throwing out in a supported way, because I'm in the middle of, you know, bolting the time inferencing Compiler into a code editor, I need my brain time for that, instead of support everything, yet. I believe if I have that editor support is going to be a lot easier. This actually runs on hello.onejs.IO you can open it up and blow your battery power. It's expensive. You need to stop animating this stuff, it's great if you want to burn through your battery, you need dirty regions and not animate. This is by the way, you can just throw your ender scene to win amp kind of visualization in five minutes, because the composability is super easy. A good friend of mine, just before I went to JSConf was like let's put pack tills ton outlines of your font. I was like Dude ... really, it's not useful, but it's so fucking pretty. Right. (applause) so, that was timed better than I hoped, I'm done, thank you (Applause (If you have ideas and questions, anything about this, seek me out, I really do want to engage, I just need my time to do some of the hard bits in this thing, still. Edit transcript via pull request.