Thanks to some motivation from Greg Weber, I've finally gotten around to working on database migrations. The code is currently only functional for PostgreSQL, but it's already being heavily battle tested: my main project relies very heavily on this feature to deal with the ever-changing specs that are thrown at me.
What I think is a relatively unique feature in persistent is database introspection. The migration code looks at your table structure, including default values, NULL/NOT NULL and type, and compares that to your entity definitions. In addition, some pretty cool features:
migrate will replace the initialize function in persistent 0.2. I still have to implement the code for Sqlite. Unfortunately, sqlite does not provide the same level of introspection and alteration abilities that postgresql does. Most likely, I will need to parse the CREATE TABLE statements and copy table contents for any migration, but it should be manageable. And when it comes to schema-less databases like Redis, migrations shouldn't be necessary.
Rehno Lindeque pointed out that we were missing two important features in persistent: getting the number of records and limiting/offsetting a result set. For the former feature, I was hoping to write something more generic, allowing any aggregate functions. Unfortunately, I haven't come up with a type-safe approach to this yet, so we'll just have a count function for the moment. If anyone has a brilliant insight, let me know.
The select function is now a bit more complicated. In addition to a list of filters and a list of orders, it now also takes an offset and limit. If the limit is 0, then no limit is imposed.
Finally, select no longer returns a list of records. For this usage, use the selectList function. select now has an enumerator interface; you provide it with a seed value and an iteratee. I'm debating whether to use the iteratee package for this interface; input is welcome.
For postgresql, we now support foreign key references. persistent figures these out in one of two ways:
This makes it exceedingly difficult to actually delete data from a database; having any references to data you try deleting will result in an exception. So I've also added a DeleteCascade typeclass and a TH helper function to automatically create "deep deletes".
Back to Greg Weber, we've been working together on cleaning up the Yesod.Form module a bit. Here are some notable changes:
day Day toFormField=YesodJquery.jqueryDayField. The YesodJquery prefix specifies that this function requires a YesodJquery instance.I've added an nginx section to the deploying page of the Yesod docs site.
One final note: I'm in the process of some big changes over here (my wife's expecting any day now, and we're probably moving apartments in the next few weeks), and I need to make sure I don't slip too far behind in my day job. Don't be surprised if I suddenly stop answering my email for a few days; it's probably because I'm either in the hospital with a new baby or between apartments without internet access.
]]>A big thanks to Gregory Collins to making some modifications to Snap so that this shim would be possible. Everything should be installable from cabal without any issues. I've added a sample to the yesod-hello repository, but it should all be pretty straight-forward.
The code itself is very simple: the only slightly complicated part was converting a WAI enumerator to a Snap enumerator. Also, since a Snap enumerator cannot be converted directly to a Source, the request body uses lazy I/O under the surface.
Below are the obligatory benchmarks for a very simple Sqlite-powered blog. The example is simply loading up a page 1000 times with 20 concurrent connections. The numbers do not necessarily reflect any performance issues with Snap itself; most likely, the performance difference between FastCGI and Snap are due to either issues in the wai-handler-snap layer or, most likely, the runtime arguments I passed to the executable. I'd appreciate feedback on this from people in the know; the code is all available on github.
]]>The major feature in this release is widgets. This allows you to create modular pieces of HTML/CSS/JavaScript and plug them together with ease. As an example, the widgets tutorial demonstrates having a jQuery UI datepicker and Nic HTML editor without explicitly calling any Javascript.
Other major changes: include a large architectural overhaul of persistent, which does not affect user code very much, but gives much more flexibility in using the library; an updated parser for Hamlet allowing some nice features like quoted attributes; a more sane subsite syntax hopefully allowing interoperation with other frameworks; and much more.
I keep the changelogs separate by project, you can see each of them here:
Unlike other releases, I don't have any major features planned next, so I'm very interested in what the community has to say at this point. I'm hoping to get a few more WAI backends working soon (waiting on some feedback from Snap before releasing a Snap-powered handler), but otherwise I'm open to suggestions.
]]>In no particular order, here are the items I've changed. I'm very happy to hear any critiques of these, so don't hold back.
I wrote a buffer function which performs badly, and I want to remove it. Nuf said.
An obvious definition for HTTP request methods is:
data Method = GET | POST | PUT | DELETE
Obvious... except I forgot OPTIONS. No problem, I'll add it. And I forgot TRACE. OK, add that too. Oh wait: you can have arbitrary HTTP methods if you like. Doh!
WAI 0.0 and 0.1 therefore had an extra constructor Method B.ByteString, which allowed you to specify any request method you wanted. All seemed well and good to me, until Jeremy Shaw commented on an entirely different topic: "The use of a type ... implies that you want to be pattern matching on the constructors in your code".
And I realized he was absolutely right. It's a bad idea to create a data type with constructors that can't be pattern matched on, and that's exactly what I'd created. After all, Method "GET" == GET, so pattern matching is out of the question. Sure enough, I've received a few patches for WAI code that mistakenly attempts to pattern match. I'd created a horrible monster which thwarted all that is well and good in type-safe programming! </drama>
So instead, the definition is now simply type Method = B.ByteString. And all is good and right in the world.
This same change was applied to the other relevant datatypes in Network.Wai: HttpVersion, RequestHeader, ResponseHeader and Status.
I'm on the fence for this one, so please let me know what you think. I've replaced the GET constructor with a methodGET function. This is simply defined as methodGET = B8.pack "GET"; and the same is done for all other constructors.
But frankly, I think this just litters the namespace. I have not implemented these functions for RequestHeader and ResponseHeader, and I don't think I will. I'm fairly certain I dislike the functions for Method, but will probably keep the definitions for HttpVersion and Status.
Gregory Collins pointed out that request and response headers should be case-insensitive. As a result, WAI 0.1 made sure their Eq instance handled this properly. Now, due to the switch to type synonyms, we define a new datatype named CIByteString (which looks strangely familiar) and use that for RequestHeader and ResponseHeader.
WAI 0.0/0.1 define a response body as Either FilePath Enumerator. We all know how wonderful enumerators are versus lazy I/O... but how good are they versus lazy evaluation. In other words, which is faster: an enumerator, or a purely created lazy bytestring? I don't have specific benchmarks, but my work on Hamlet indicated to me that lazy bytestrings win.
The vast majority of web applications will not often need an enumerator interface. Mostly, static files and lazy bytestrings will be sufficient. Hack has done a very good job with only providing a lazy bytestring interface. So I would like to add a lazy bytestring response body to WAI; not to replace enumerators, but to augment them. So I've created a new datatype:
data ResponseBody = ResponseFile FilePath
| ResponseEnumerator Enumerator
| ResponseLBS L.ByteString
This adds a bit of complexity to handler code, but I think it's worth it. I've also replaced the fromEitherFile function with fromResponseBody. As a bonus, I think it's nicer using a specific datatype than an Either.
There are three decisions in particular that I've decided to stick with, even though there's been some pressure to change this. I'd like to state these out loud to re-open the debate, because I want to make sure this is the right decision.
The first is fairly trivial: whether the Request datatype has a record for script name. Hack and the Hyena WAI both have it, but I've removed it for two reasons:
The second issue is the Source datatype as a request body. People have said it should simply be an Enumerator. I have this to say:
The final issue is the definition of Enumerator. There's lots of work in the iteratee package, and it seems to be blazing the trail for this domain. However:
Much of the complexity introduced by iteratee is totally unnecessary in WAI. For example, we can't have an arbitrary monad, we need IO. We can't have arbitrary chunk data types, we use strict bytestrings.
However, at a more fundamental level, iteratee is trying to solve problems WAI doesn't have. WAI just needs a way to allow efficient outputting of data with interleaved IO actions. It will always just send one chunk at a time, and either the entire chunk will succeed, or there will be a failure. iteratee deals with complex problems like partial input consumption.
In other words: I have not seen a single argument made why switching to iteratee is actually a win for WAI.
That said, I'm happy to hear any explanations and see some examples of why I'm making The Wrong Decision here. For that matter, please tell me if you think any of the changes I'm proposing for WAI are a bad idea.
]]>Most web frameworks provide some notion of user sessions, where you can store small bits of information for each user. In the case of Yesod, we use the clientsession package (also written by yours truly) to store data in an HTTP cookie. While this might sound insecure and inefficient at first, it's not:
Like most frameworks, the sessions are simple key-value stores. There are three basic operations on a session:
Please note that the setSession and deleteSession functions do not affect the outcome of a call to lookupSession within a single request. In other words:
do
val1 <- lookupSession "key"
setSession "key" "newvalue"
val2 <- lookupSession "key" -- val1 == val2, not "newvalue"
Messages are built on top of sessions. The situation is a common one in web development: the user performs a POST request, you make a change, and want to simultaneously tell the user that the request suceeded and redirect them to a new page. Yesod provides a pair of functions to make this very easy:
In my projects, I will often put a call to getMessage in the defaultLayout function and know that any messages I set with setMessage will be displayed on the next page the user views.
Not to be confused with a horror film, this concept is used internally in the Yesod authentication module. Simply put, let's say a user requests a page that requires authentication. Clearly, you need to send them to the login page. A well-designed web app will then send them back to the first page they requested. That's what we call the ultimate destination.
We all know how annoying it is to deal with web forms. Back in the good old days of raw CGI, you'd have to write your form HTML, write parse code separately, probably write another version of the HTML to create sticky fields, etc. We've come much farther than that, especially with libraries like formlet. However, there's still one piece of the puzzle which is tedious to set up: the Javascript and CSS.
For example, I want to have a day field. Of course, like any sane person, I use YYYY-MM-DD format. But I live in a country where the standard format is DD/MM/YYYY, and many of the user's of my sites live in a country where the standard is MM/DD/YYYY. So ideally I'd like to use Javascript to both do client-side checking and provide a nice date picker. I also would like error messages to be displayed in some shade of red.
So with Yesod 0.3.0 I would need to include the necessary Javascrip libraries in my template, include the datepicker CSS file in the template, write a local Javascript file to initiate the datepicker and the validation code, and write a local stylesheet for styling the error messages. Now that's tedious.
Instead, we're going to create composable widgets. They keep track of not just HTML to be placed in the body of a document, but also scripts and stylesheets to load, extra HTML to throw in the head, CSS for <style> tags, and the title of the page. It also should supply me with unique identifiers to use.
I've implemented this as a monad transformer stack; you can see the implementation on Github. The Yesod.Widget module also contains a bunch of functions for creating widgets. For example, addStylesheet adds a stylesheet, addBody adds some content to the body of the page, etc. There's also a cool function applyLayoutW which is able to display a widget with the default page layout.
Since a widget is just a monad stack, it composes very easily. There's also a Monoid instance to make life simpler. The trickiest part of it right now is when you want to wrap the body HTML code up somehow (for example, sticking everything in a div id=wrapper tag). For this, check out the wrapWidget and extractBody functions in the Yesod.Widget module.
The next big step is converting Yesod.Form over to use widgets. The original inspiration for this module is formlets; however, I'm veering farther and farther from the API, as my goals are quite different than the formlets API. As an example, formlets does not support inline error messages; all validation messages must be shown together. I don't find that an acceptable tradeoff.
The API is in flux right now, so I don't want to spend too much time talking about it. However, here's a high-level approach: there is a GForm monad which takes an "xml" type parameter. There are then two type synonyms: Form sets that xml type parameter to be a widget. This is what you'll usually want to embed in a page.
The other type synonym is FormField, which instead sets that xml type to FieldInfo. A FieldInfo contains information on the label, error messages, and various other things in a field. Of course, you can't display this directly; there will be various helper functions to convert a FieldInfo into a widget. The reason to split this up is so users have more control of how to layout their forms. For example, there's a function currently to display a FieldInfo as a row in a table; we could also provide a list and paragraph version.
And I still intend to have all this tie in with some typeclass for automatic creation of forms for PersistEntity instances. Just imagine: you could declare your entity structure once, and Yesod will create a database schema, generate SQL code, and create HTML forms for you automatically. I have some examples of this actually working with various versions of Yesod, and most likely the release of Yesod 0.4.0 will be accompanied by a screencast showing how to use all these features together. Most likely it will be the inevitable blog example ;).
]]>Version 0.0.0 of persistent had a typeclass called PersistEntity. Without getting into details, this typeclass essentially defined exactly how a datatype could be persisted to a particular database. Some advantages to this approach are:
However, there are some major disadvantages to this approach:
So instead, persistent 0.1.0 will split up the work into two type classes: PersistEntity and PersistBackend. The former gives information on a datatype, such as what columns exist, how it can be sorted, unique keys, etc. It contains nothing backend specific. The latter typeclass is for each backend- like Sqlite and Postgresql. It is implemented without any TH code.
One downside is that a new backend must deal with the information provided by the PersistEntity typeclass. If it needs extra information that's not there, it won't work. I'm not so worried about this one, since I'm currently the only backend writer, and if someone else wants to write a backend that requires more information than is currently available in PersistEntity, I can just add it later.
The other downside is that database keys are all required to be Int64s. In practice, I don't think this is a problem at all: the only backend I know of that would really prefer a different key is Amazon SimpleDB, and even there I'm sure an Int64 would work.
When Jeremy Shaw put together web-routes, I thought that web-routes-quasi would interact with it very well. Unfortunately, as the complexity of the package began to grow, I realized that it was really just becoming tailor-made for Yesod. This was a sub-optimal situation. And each new release of web-routes-quasi just added more Yesod-specific features.
This release (0.5.0) will be different. It no longer defines a QuasiSite datatype; we reuse the Site datatype from web-routes. It also cleanly exposes different TH functions for the four relevant declarations it provides for: creating the type-safe URL datatype, the parse function, the render function and the dispatch function.
The main reason I decided to start on this rewrite is to allow the dispatch function to return values still inside the Handler monad. Previously, web-routes-quasi would unwrap that monad itself, using a provided explode function. The plus was this provided a nice way to deal with the difference in type signature to subsite handlers (GHandler sub master) and master site handlers (GHandler master master); unfortunately, it made it difficult to provide features like authorization.
So now web-routes-quasi simply promotes the subsite handler functions into master site handler function. It needs three pieces of information to do this:
The first of these is provided by the user when declaring routes; the last two are handled automatically by web-routes-quasi.
It seems now that migrating over to ContT for the Handler monad was a mistake. In particular, it has a broken MonadCatchIO instance, which leads to a number of annoying bugs (seg faults in sqlite because it double-frees memory). Of course, I still hate the ErrorT transformer, both for the super-class requirement and the orphan Monad instance.
So I've created a new package: neither. I originally thought of the name as a joke; in fact, my wife (a Greek/Latin major) helped me come up with a great datatype:
data Neither a b = Sinister a | Dexter b
However, it now provides three useful datatypes: AEither for an applicative-style either type, which combines Lefts via mappend; MEither for a monad instance; and MEitherT for a monad transformer. And you can use them all without any orphan instances. It also provides both mtl and transformers instances of MonadIO, MonadTrans and MonadCatchIO.
Also, as a result of really hideous error messages, I've switched GHandler to be a newtype instead of a type.
These two modules currently provide related functionality: the former is for getting GET and POST parameters without generating HTML forms. The latter is for generating HTML forms and automated parsing of POST parameters. It shouldn't be too difficult to combine the two together.
I've added Facebook support to the authenticate repo, and passed that support up to the Yesod.Helpers.Auth module. It keeps track of the access token for you, so you can easily make requests against the Facebook graph API.
]]>Let me describe a project from hell. Imagine a site that has a navigational structure sometimes 8 levels deep. Imagine a client that can't write a spec, but knows exactly what he wants. And imagine that the "exactly what he wants" changes on a daily basis.
Breadcrumbs was a major annoyance on this project. Let's say we suddenly need to move one branch of the sitemap somewhere else. Using a normal, manual breadcrumbs approach, you have to change every single child page in the hierarchy to reflect things. Let's say the client decides you need to suddenly rename the root of the tree: you have to change every single page.
But if you think about it, breadcrumbs should really be very simple. For any page on your site, you need to know 1) the title of the page and 2) the parent page, if any. Turns out that type-safe URLs make this really easy to express as:
breadcrumb :: url -> (String, Maybe url)
If this is a top-level page, return Nothing in the second part of the tuple. Otherwise, return the parent. The first part in the tuple is the page title. I went ahead and packaged this into a typeclass, made the type signature a little more complex to run inside the Handler monad, and you have:
class YesodBreadcrumbs y where
breadcrumb :: Routes y -> Handler y (String, Maybe (Routes y))
breadcrumbs :: YesodBreadcrumbs y => Handler y (String, [(Routes y, String)])
The breadcrumbs function returns the title of the current page and the chain of parents along with their titles. You can easily turn that into some HTML using Hamlet:
1 (title, parents) <- breadcrumbs
2 return [$hamlet|
3 #breadcrumbs
4 $forall parents parent
5 %a!href=@fst.parent@ $string.snd.parent$
6 > $
7 $string.title$
8 |]
Just to point out some of the Hamlet syntax: the trailing dollar sign on line 6 is simply to demarcate the extra space at the end of the line; Hamlet ignores it. On line 5, the at signs denote a type-safe URL, whereas the dollar signs represent HTML content. string is a function from BlazeHtml to convert a String into the Html datatype./p>
As fast as Yesod is, it's not as fast as a static file server. It makes a lot of sense to server the static content from a different server (not to mention you avoid the cookie-transport cost). However, when developing the application, we want to be able to simply use the simple server and not set up a dedicated static file server.
That's where urlRenderOverride comes into play. Given a type-safe URL, it might return a replacement absolute URL. urlRenderOverride is part of the Yesod typeclass, where we keep a lot of optional settings. By default, the function always returns Nothing, which allows normal URL rendering to take place. However, you can set up your production code to intercept all of your static file requests and send them to a different server.
For an example of how to do static file server, see the Ajax tutorial.
One of the more recent additions to the Yesod typeclass is the isAuthorized function. Unfortunately, it was not done properly in 0.3.0, but I intend to fix that in 0.4.0. In any event, here's the idea: Given a type-safe URL, it returns Nothing if the request is authorized, or Just with a message if unauthorized. If you want the user to authenticate first, or provide some other information, you can use a redirect to send them to a different page.
The problem is that I wrote this to live in the IO monad instead of the Handler monad, so it doesn't have access to request information or the ability to send a redirect. Once this is fixed, it should provide a very light-weight access-control-list mechanism.
All three of these methods offer two serious benefits from a program safety standpoint:
I'm sure a lot of this could be implemented in a different way than type-safe URLs, but I think that we see here a natural result of a good design decision.
]]>This release of Persistent includes backends for sqlite and postgresql. The former has no external library dependencies, as it includes the SQLite C library in the distribution. The latter is built on top of HDBC-postgresql. Please see the documentation site for more details.
Yesod 0.3.0 adds some rudimentary support for Persistent, most interestingly in the Yesod.Helpers.Crud module. With this addition, Yesod is now what I'd consider a "full featured web framework", providing Model, View and Controller functionality. However, not all features have been finalized. In particular, the Yesod.Formable module is highly subject to change, and will be merged together with the Yesod.Form module in upcoming releases.
A fairly complete changelog for this release of Yesod:
I wanted to get this release of Yesod out quickly so that people could start playing around with Persistent immediately. However, as noted, some aspects are as yet unpolished. That's because I'm intending on adding a very significant feature in the next release: widgets.
Let's give a very simple motivating example: web forms. Let's say you want a form that provides client-side validation and shows your error messages in red. Currently, you need to write your Yesod code to generate HTML, write Javascript separately, write the CSS separately, and explicitly link in the JS and CSS files.
The widget concept is an attempt to combine all of that into a single place, within your Yesod code. For example, you'll be able to specify some Javascript validation code, and specify dependencies for that Javascript, such as jQuery. If you have other Javascript code on your page that depends on jQuery, jQuery will only be included once.
This will hopefully encourage the creation of composable web components. This looks to me to be a very exciting direction for Yesod to take, and I look forward to sharing my progress on it in the future.
]]>Also, I've just written the documentation for Persistent. I think the API is pretty stable at this point, and frankly the Postgresql backend is probably ready for a release. However, I'm holding off a little bit as I'm trying to solve an issue affecting the sqlite backend.
And once that's done, I think I'll be releasing Yesod 0.3.0. This release will most likely not be quite as polished as previous releases: there's currently a confusing redundancy in the API (both Yesod.Form and Yesod.Formable) which I'll most likely leave in for the moment. The reason is that I have plans for a widget system for the following release which will supercede these issues. Stay tuned for more details.
]]>