Monthly Archives: August 2009

Last week Twitter announced ‘Project Retweet’, their plans to develop a formalised API for the phenomenon of ‘retweeting’. It’s interesting that retweeting grew organically, a convention formed by user behaviour. In the announcement, Biz recognises that in this way some of Twitter’s best features are emergent – those “simple but creative ways to share, discover, and communicate” that the crucial Twitterverse have formed.

It reminded me of the ‘replies kerfuffle‘ earlier this year and Twitter’s reactive decision to review their technical decisions as a result of the amount of negative feedback they gained so quickly, their adaptability should be applauded. Biz admits they learned a lot.

What I’m getting to is that although Twitter has been near revolutionary, that now even though it’s such an important social tool – it’s still very young. Not in the sense of naivety (nor am I questioning their success), more that there is still a huge amount of potential in Twitter that we are eagerly awaiting to see come to fruition.

Here I want to talk about some of the things that I want to see, some of the things that I think are still lacking, though recognising some of the obstacles in seeing them fulfilled.

For me, the notion of conversations (as in actual conversations) and topic threads are pretty much non-existent and obviously missing. I mean ‘actual’ conversations as in direct correspondence between parties, rather than the more ‘global conversation’ reference that’s loosely thrown around when one wishes to sound informed lately.

Replies, Threads and Conversations

The function of ‘replies’ and ‘mentions’ have seen a lot of work on the Twitter platform, see their blog archives for the changes to-and-fro, but in my opinion they are still very limited.

When the ‘small settings change‘ took place in May, it turned out to be considered far larger than the heads at Twitter had anticipated. The intention was to clean up what was seen to be undesirable and confusing, though the result is arguably as vague. If you type ‘@someone’ into the status box and tweet then everyone following you will receive it, but if you click the ‘reply to’ button (which automatically puts ‘@someone’ in the update box for you) then only those following that ‘someone’ will see it. The reply button create a response message – manually typing it does not, apparently then it’s a mention. Correct me if I’m wrong.

Anyway since those changes, reply tweets now link to the original messages (facilitated by clicking the reply button on that tweet, no less). So now when viewing a tweet, the ‘in reply to [someone]‘ is actually a hyperlink to the status update to which the user has replied.

A tweet 'in reply to'

This connection though, is only one way. And that’s the main crux of everything I’m going to say here.

The reply is aware of the original tweet, but the original tweet does not know that it has been replied to.

This isn’t so much of a big thing in a simple two-tweet case of ‘question and answer’ (maybe), but this is what stops users and applications from easily following conversations that are taking place.

For example, look at the following illustration of a conversation over four tweets:

A conversation on Twitter with two participants

The messages read A through to D, chronologically, with each tweet replying directly to the previous. Without having witnessed these tweets being sent, the only way to follow them is essentially backwards. Each reply has a link to it’s original tweet, so starting with D, you can click through C, B and A to reveal the point of origin.

This is the problem. Tweets have no forward links to their replies.

In the above case:

  1. Tweet B is aware of tweet A (though A is unaware of B).
  2. Tweet C is aware of tweet B (though again, B is unaware of C).
  3. Tweet C is unaware of tweet A, even though it is linked to tweet B which is aware of tweet A.
  4. And so on, the coupling continues with D (and E and F, etc).

 
Without creating links in the forward direction of the conversation, structuring and viewing threads like this is almost impossible without some serious archive mining.

Regardless of whether users wanted these changes to happen or not, I don’t think they solved the problem they set out to anyway. They were put in place to clean up timelines, removing conversational tweets just like these because seeing without context they’re almost always irrelevant, having been written in 140 characters. But now, if someone was really interested as to the answer of tweet A but they didn’t follow @birdsigh (or any replier), then they would never know it had even been answered, not only because they don’t get @birdsigh‘s tweets but because they wouldn’t see tweet C either anymore.

If tweets had references to those that reply to it, anybody could see responses from everybody.

True, conscientious users often retweet replies that they’ve received if they are of value, but this can look a bit weird – seeing a number of usernames (including their own) then the message, like so (see the duplication of @aral):

A tweet retweet himself

So here’s another problem that Twitter are already on the heels of answering – with the new retweet functionality Biz mocked up, looking far cleaner and a lot smoother – no need to paraphrase or condense tweets to fit in n amount of usernames who deserve props.

I mentioned the archive mining – Twitter’s Search API has something almost there. If one of your results is a reply, there’s a ‘Show conversation’ button which expands to show nearby (?) tweets between the two participants. It shows the thread, yes, but it’s not that reliable – it can include tweets that weren’t direct replies to the others, rather mentions that were chronologically in between. And none of that is in the main API anyway.

Visualising conversation

That’s another thing – there’s no single place or any really decent way that I’ve seen to visualise this kind of information right now. It would be nice if the ‘in reply to’ link didn’t need a page change to see the related tweet.

Some of the desktop AIR applications try to solve this problem, for example, DestroyTwitter has a nice dialogue box – but only shows pairs of tweets, so you still have to click through:

DestroyTwitter dialogues

TweetDeck has a similar view to the Search page’s conversations, showing the latest tweets between two participants, filtered to those mentioning or replying to the other – so again it does show correspondence, but it’s uninformed – you can’t select a thread and show only that and here too you see unrelated and old tweets also.

I’ve not seen any application that shows conversations involving more than two participants.

It’s interesting that phase one of ‘Project Retweet’ introduces a ‘retweet’ button, something that’s been around in the desktop apps for a long time (even if on their part it’s a crude copy-and-paste with the additional ‘RT’ prefix added). Like Biz said, some of the best features are emergent – so it wouldn’t be so surprising if, seeing how desktop applications are already attempting to visualise threads like this, they might one day materialise on Twitter.com as a fully fledged feature.

What’s to come?

The Twitter API documentation now has some timeline methods labelled as ‘Coming Soon’, those for the new retweeting API. One of those is the statuses/retweets_of_me method, said to ‘return the 20 most recent tweets of the authenticated user that have been retweeted by others’, (here).

Woop! That does mean forward linkages right?

If tweets can be tagged to inform that they have been retweeted, surely its somewhere possible to also inform that they’ve been replied to?

Back to visualisations, it’s be great if Twitter had some pages similar to these to show retweets and conversations (respectively):

A conversation page 'on Twitter'

A retweets page 'on Twitter'

Okay, it’s easy to see one improvement and make an assumption that it can be applied ‘exactly the same way’ elsewhere. As as developer I know that and get tired of hearing those kinds of requests :(

But it’s good to hear that these improvements are but ‘phase one’ of what’s to come. Maybe all of this has been recognised, in store – maybe already in development! Who knows. I’m sure I’m not alone in wanting something like this.

I mentioned the idea of threads with more than two participants. Immediately my idyllic conversation view above falls apart right here if you want to see a whole topic of multiple threads on one screen.

I guess you could check for any replies to a given tweet and any replies to those replies exponentially, presumably listing them all chronologically, but you’d miss out on the direct correspondence between individual users.

A Conversation on Twitter with multiple participants

That kind of non-linearity is another great thing about Twitter, you don’t really get that anywhere else. Take for example correspondence on blog posts – if someone has something to say in response to a post then they can comment, but if in reply to that comment the person wants to write a second lengthier response, is the comments section the correct place to do it? Or would say, a post on their own blog ‘in response to’ be more appropriate – because that’s on their domain, under their name and heading?

Then there’s a weird loss of connection though, that post doesn’t appear in-line with the original post (other than a pingback, maybe) and users are confused where the topic continues.

That said, the ability to reply to multiple tweets also isn’t possible, only to reply to a single tweet and mention another user within that message to ensure they receive it.

Replying to two tweets

I look forward to seeing the new retweeting functionality come into effect. Hopefully the change will be well received.

Thinking about the negative feedback the changes to replies received, I recall when Twitter stopped providing their SMS service to UK members – back then I was gutted but now that they’re back, I don’t use them anyway. I’ve turned device updates off. Things are different now, there weren’t as many desktop applications (and those weren’t even half as useful as today’s) and my way of thinking about the people I follow and those that follow me is different now too, my behaviour has changed.

The way I use Twitter has changed and will no doubt continue to do so as the platform evolves. Though I’d be very happy to see anything like what I’ve written here turn up (somehow) on Twitter eventually, I think it would change change the way we use it even more.

Update (02.09.09): Since writing this I’ve been pointed to two more applications that attempt to visualise conversation threads.

The first is Tweetie for Mac, that not only shows more than two tweets in sequence, but updates from multiple users:

Tweetie conversation oneTweetie conversation 2
  
The second is Tweader, a web based tool declaring an outright ‘new way to view Twitter conversations’, clearly also in recognition of the need for one. Tweader requires an ID of a reply update and (as above) constructs the conversation backwards by retrieving each original tweet.

Both of these do the job as best they can, but are still subject to the limitations of Twitter’s API.

For example, with Tweetie you must click a reply tweet, you can’t click a tweet and see the conversation that followed – the forward links are still absent.

Same with Tweader – try this and this.

Now although it’s the Twitter API that doesn’t offer this kind of linking, these tools are standalone applications so there’s nothing to stop them from storing their own form of descriptive data internally to allow conversations to be pieced together in such a way.

For example, upon receiving a reply tweet, one could store the ID against the original tweet ID (within their own infrastructure) so if a query is made upon the original, it’s responses can be found.

But as I’ve said, what I would like is for Twitter to support threads with a formalised API call – say with a method whereby one can submit the ID of any tweet (that is part of a conversation) and get a full node list of all tweets connected to that it, either in response to or those that came before it.

This month’s YDN Tuesday was presented by Steve Marshall, he spoke about code generation, ‘Writing the Code that Writes the Code’.

A bit of a disclaimer: Going into this talk I wasn’t sure how much I’d be able to take from it, or whether what Steve would be discussing would be relevant to my own kind of development. I figured he’d be talking about working with huge projects with (literally) tens of thousands of lines of code and I don’t think that the evening was part of the ‘RIA’ umbrella which is probably closest to home for me. Anyway although Steve’s talk did cover those kinds of projects, he also went through some platform-agnostic fundamentals – the kind of transferable ideas that could be applied to any coder – and it’s really those I’m looking at here.

Code Generation in Action

Steve opened with a bit of code generation evangelism, introducing a discussion as to why code generation is something that all developers should want and should do.

Each point here I think speaks for itself, but only really having gone over each benefit staring me in the face have I questioned, hold on, why don’t I do this already? It’s a convincing argument..

The most immediate is scalability – so often development projects have large amounts of repetition, say in creating views, or sets of view/mediator/component classes, or whatever – needless to say as projects get larger, generating this kind of code from templates and base classes reduces or even eliminates the room for error.

Neatly, this this was his next point – consistency. Automated code cannot be mistyped or have any part overlooked (as could when writing by hand), unless of course the source is incorrect. And even if that’s the case, a small change can be rolled out to all generated code once corrected. Which brings us to his third point – automatic code is of course far quicker to produce.

Steve also suggests that generated code is ‘predictable’, in a way. Whether looking at code that has been generated or that written for ‘expansion’ (more on this later), on the whole it’s consequently easier to digest. Likewise, source code that is to be used by a generator or to be ‘expanded upon’ is, by its very nature, a smaller volume – so therefore easier to approach.

It’s also a single point of knowledge. Only the core source code needs to be understood. A project of tens of thousands of lines of code would be near impossible to traverse or otherwise understand, knowing that code has been generated from a smaller source that you do understand offers a reassurance that the rest of the code is good.

Another advantage is the ease of which the code can be refactored. Obviously only the source code base requires changing, the larger volume is generated automatically. This kind of code is language independent.

Outside of working with code direcly, the kind of abstraction writing for code generation offers its own benefits. Although arguably a desirable trait of good programming anyway, Steve suggests this kind of code is far more decoupled with the design process. Thinking more abstractly about coding means it has less effect on design capability, and vice-versa. Design reviews or changes should have no bearing on the coding. By being naturally quicker to produce, Steve also says that generated code is a good methodology to pursue when prototyping applications.

Steve also discussed testing, offering that generated code has (or presumably can have) 100% test coverage. Similarly, full documentation is easier achieved by way of having written a smaller code base.

Overall, cleaner code that’s easier to work with and easier to understand is undeniably a more attractive prospect to work with. Developers are more enthusiastic about working with that kind of code and so that kind of code is of a higher quality – in the first place, as well as because it has been consistently deployed, tested and documented thereafter.

So how is this achieved?

Models

Steve recognised six principle methods for generating code, each technique seemingly increasingly more ‘advanced’ than the previous and, as I said before, each progressively more suited to larger-scale projects.

He saw these as:

  1. Code munging
  2. Inline code expansion
  3. Mixed code generation
  4. Partial class generation
  5. Tier generation
  6. Full domain language

 

I’ll look at each briefly.

Code munging


Code Munging
Code munging essentially takes your code, parses, introspects and creates something new that’s not code, though still of benefit and in inherently useful.

Steve’s examples were Javadoc or PHPdoc, it’s the most basic form of generation you can introduce to your workflow.

Inline code expansion


Inline Code Expansion
This the ‘expansion’ mentioned above, where source code is annotated with keywords or variables where upon additional code is generated to replace those markers to extrapolate upon the original.

Templates are the most rudimentary example. This technique gets repetitive jobs done well, quickly and correctly.

Mixed code generation


Mixed Code Generation
This is the first ‘real’ form of generation, this expands upon Inline code expansion by allowing the output code to be used as input code once created. Instead of simply replacing special keywords for example, mixed code generation may use start and end markers, expanding code as before, but allowing the result to be worked upon once completed.

This may include code snippets, or smaller templates injected within classes.

Partial class generation


Partial Class Generation
I guess here’s where things start to get serious. Partial class generation begins to look at modeling languages like UML, creating abstract system models and generating base classes from a definition file.

The base classes are designed to do the majority of the low level work of the completed class, leaving the derived class free to override specific behaviors on a case-by-case basis.

Tier generation


Tier generation
This takes partial class generation a level further, a tier generator builds and maintains an entire tier within an application – not only the base classes. Again from some kind of abstract definition file, but the generator this time outputs files that constitute all of the functionality for an entire tier of the application – rather than classes to be extended.

Full domain language

Steve didn’t really talk about this much, he said it’s very tough – and even if you think this is what you want to do, it’s probably not (though a hand from the crowd wanted to reassure that it might not be quite that intimidating).

A domain language is a complete specification dedicated solely to a particular problem – with specific types, syntax and operations that map directly to the concepts of a particular domain.

I used the Code Generation Network’s Introduction for reference here, which looks like a good resource.

Also, all the diagrams belong to Steve. He’s put his presentation slides on Slideshare and uploaded PDF and Keynote versions on his website.

The presentation was also recorded as a podcast and is now on the Skills Matter site.

In retrospect

After all these, Steve took us through an example he’d worked on where code generation really proved it’s worth (a huge project for Yahoo!) – and offered tips and best practices to use once you’ve convinced your team to take on code generation (see the slides linked to above).

Though in my opinion here he seemed to seemed to almost contradict some of the things he’d talked about before. He pointed to some tools to use within your workflow, but spoke extensively about writing your own generator. Stressing that it can be written in any language, though should be written in a different language to that of the code being produced, that its easier than perhaps you think – though still with a fair few caveats.

But what happened to cutting down the work load? As the talk summary says, developer are fundamentally very lazy – we want to do less and less, surely this requires more effort?

One of his recommendations is that you document your generator once you’ve written it. But how would you create that documentation, with another generator? Is that meant to be a specially written generator also? And that needing documentation too?

I remember an episode of Sesame Street from my childhood, where Big Bird was painting a bench. Once he’d finished he made a ‘Wet Paint’ sign with the paint remaining, only to realise that the sign also was wet, so needed a ‘Wet Paint’ sign of it’s own. Which of course, was also wet and needed it’s own sign. Cut back a scene later and the whole street is full of signs. It was upsetting, to say the least.

I’m being pedantic, I know.

But I guess that’s where I recognise a line for me, or at least the kind of development I do. Very large-scale projects do often need this kind of approach, Steve is proof of that. For me only the first few techniques are appropriate. Though those, undeniably, are incredibly beneficial when you have the opportunity to put them into play.

On a similar note, this month’s LFPUG meeting had a presentation on UML for AS3 by Matthew Press, the recorded video for which is now online – have a look!

Tell her to push over and move them big feet.