NewsfeedModule.wiki - wiki - Git at Google

 #summary 'Sandworm' Newsfeed Module
 #labels Importance-Useful,Phase-Requirements,Contents-Draft

 [http://github.com/jamslevy/Melange/tree/master Source Code Repository]

 <wiki:toc max_depth="3" />

 = Introduction =

  	 A framework for customizable news feeds for various entity kinds that can be formatted on various site pages. Each newsfeed includes a subscription link, and through a 'subscribe' url pattern is used to render XML feeds.

 Because of the relatively simple nature of the initial project, some of the post-midterm development will include an auxiliary open source pubsub component meant to be deployed separately, and act as a hub allowing Melange instances to host chat, Google Wave updates, and other real-time activity.

 = Newsfeed Model =

 An individual FeedItem entity is created for each news feed update. A garbage collection task will delete old feed items, so they don't take over the datastore.

 The FeedItem model contains the following properties:

  * *sender* (db.Key) - The sender is the entity that was updated

  * *receivers* (db.Key) - A list of entities whose news feed is being updated.

  * *update_type* (created, deleted, modified)

  * *created* (datetime)

 A FeedItem is created for each sender-receiver pair. For instance, if a document is created and saved, it is the sender while its associated project and program and user could all be receivers.

 = Access Protection =

 Very basic entity information (a title/name and a link) is shown in the Recent Updates box and the ATOM feed. This may violate access protection, and the alternative is to just show the name of the entity kind, or a truncated title.

 Because the mail subscriptions are privately sent on a per-user basis, they contain more detailed information about the updated entity.

 = Logic =

 Newsfeed logic consists of two public methods:

  * *addToFeed* takes a sender entity, a list of receiver entities, an update type, and an optional payload (like a text preview of a document). It adds all the receivers to a FeedItem with the specified payload:
 {{{
     for receiver in receivers:
       new_feed_item = soc.models.news_feed.FeedItem(
       sender_key= str(sender.key()),
       receiver_key = str(receiver.key()),
       user = user,
       update_type = update_type
       )
 }}}

  * *retrieveFeed* takes a receiever entity, and retrieves it's latest news feed updates, with an optional count kwarg. This method is cached, with a cache handler in the /cache folder.

 {{{
   def retrieveFeed(self, entity, count=10):
     """ Retrieves feed for a given entity
     """
     feed_items = soc.models.news_feed.FeedItem.all().filter(
     "receiver_key =", str(entity.key())).fetch(1000)
     return feed_items[:count]
 }}}


 = Views =

 The newsfeed template itself is simply a table modeled after the blog template that lists each news feed update, which is added to the context dict through the retrieveFeed method.


 = Subscribe via RSS =

 I've added a new url_path 'subscribe', so that a URL for requesting an ATOM feed would look like this:

 http://melange.appspot.com/site/subscribe/somesite

 The new URL pattern is configured in soc.views.helper.params:

 {{{
   if params.get('subscribe'):
     new_params['django_patterns_defaults'] += [
         (r'^%(url_name)s/(?P<access_type>subscribe)/%(key_fields)s$',
          '%(module_package)s.%(module_name)s.subscribe',
          'Subscribe to %(name_plural)s')]
 }}}


 This address is added to pages with a meta tag:

  <link rel="alternate" type="application/rss+xml" title="RSS" href="{{ feed_url }}"/>

 There is also a graphic subscribe icon on the corner of the newsfeed GUI module.

 The 'subscribe' path will also be the interface for any PubSub interactions, but those will only be activated on POST requests with a callback URL, otherwise the ATOM feed will be returned as normal.


 = Notification Integration =

 The notifications module has been in need of work, and I've refactored it so that it has two new beneficial features:

  * Deploy to multiple contexts (the newsfeed widget, an ATOM feed for the user w/ pubsub support, and information-enriched private e-mails).

  * Any entity with a newsfeed can now be referenced in the notification so that that entity's newsfeed will be updated, etc.


 == Notification TODO ==

 * Refactor access handler to be able to perform reflective rights-based queries.
 * Improve notifications front-end
 * Licensing issues for images?
 * Improve e-mail format to take better advantage of entity-tagging in notes.
 * Tag multiple entities in notification.


 = E-mail Tasks =

 When a feed item is created, it schedules a handler to send out e-mail notification. This handler calls the sendFeedItemEmailNotifications method that fetches a list of users who are subscribed to the sender entity and e-mails them.

 E-mails are now each sent via the mail_dispatcher task. Since it is very common to get OverQuotaErrors, this task API should always be used for sending e-mails.


 = User Subscriptions =


 The subscriptions model contains a relational index entity for the user, containing a list of entity keys that the user is subscribed to. From their edit_profile page, the user can set a global preference for whether they are accepting e-mail notifications, and the user can also configure individual entity preferences from those pages using the subscribe-by-star UI pattern.


 = PubSubHubbub Support =

 <wiki:video url="http://www.youtube.com/watch?v=ewQBgbysSOQ"/>

 This demo demonstrates Google Reader updating in seconds after a blog is updated.

 It should be fairly trivial to integrate push notification support, and abstract it so that it could also be accessed by other Melange features.

 == TODO ==


 == Subscription Handler ==

 The subscriptions logic contains an UpdateLogic class that is meant to automatically compile a list of users with read-access to an arbitrary entity. Unfortunately, setting subscriptions manually is the only option right now, since the automatic subscription handler is not nearly where I hoped it would be by now. The idea is very simple - just
 subscribe users to the entities where they have read-access. But the reality
 has been much more difficult to do without creating an entirely new
 infrastructure, and I'm entertaining the idea of simply relying on a manual
 interface (subscribe-by-star). The takeaway from this is that perhaps the
 access component should be abstracted to a high-level API, because it
 appears to have gained dependencies and complexities that prevent it from
 being useful to the news feed logic.

 == Post Commit Hooks ==

 Since most soc projects involve a code repository, it would be useful to allow repository commits to enter the news feed. This is already possible through using an RSS feed, but real-time updates could be generated by using the Google Code Post Commit Hook Support:

 http://code.google.com/p/support/wiki/PostCommitWebHooks

 This would require project owners to change a setting on their projects, and would be fairly easy to implement.
	#summary 'Sandworm' Newsfeed Module
	#labels Importance-Useful,Phase-Requirements,Contents-Draft

	[http://github.com/jamslevy/Melange/tree/master Source Code Repository]

	<wiki:toc max_depth="3" />

	= Introduction =

	A framework for customizable news feeds for various entity kinds that can be formatted on various site pages. Each newsfeed includes a subscription link, and through a 'subscribe' url pattern is used to render XML feeds.

	Because of the relatively simple nature of the initial project, some of the post-midterm development will include an auxiliary open source pubsub component meant to be deployed separately, and act as a hub allowing Melange instances to host chat, Google Wave updates, and other real-time activity.

	= Newsfeed Model =

	An individual FeedItem entity is created for each news feed update. A garbage collection task will delete old feed items, so they don't take over the datastore.

	The FeedItem model contains the following properties:

	* sender (db.Key) - The sender is the entity that was updated

	* receivers (db.Key) - A list of entities whose news feed is being updated.

	* update_type (created, deleted, modified)

	* created (datetime)

	A FeedItem is created for each sender-receiver pair. For instance, if a document is created and saved, it is the sender while its associated project and program and user could all be receivers.

	= Access Protection =

	Very basic entity information (a title/name and a link) is shown in the Recent Updates box and the ATOM feed. This may violate access protection, and the alternative is to just show the name of the entity kind, or a truncated title.

	Because the mail subscriptions are privately sent on a per-user basis, they contain more detailed information about the updated entity.

	= Logic =

	Newsfeed logic consists of two public methods:

	* addToFeed takes a sender entity, a list of receiver entities, an update type, and an optional payload (like a text preview of a document). It adds all the receivers to a FeedItem with the specified payload:
	{{{
	for receiver in receivers:
	new_feed_item = soc.models.news_feed.FeedItem(
	sender_key= str(sender.key()),
	receiver_key = str(receiver.key()),
	user = user,
	update_type = update_type
	)
	}}}

	* retrieveFeed takes a receiever entity, and retrieves it's latest news feed updates, with an optional count kwarg. This method is cached, with a cache handler in the /cache folder.

	{{{
	def retrieveFeed(self, entity, count=10):
	""" Retrieves feed for a given entity
	"""
	feed_items = soc.models.news_feed.FeedItem.all().filter(
	"receiver_key =", str(entity.key())).fetch(1000)
	return feed_items[:count]
	}}}


	= Views =

	The newsfeed template itself is simply a table modeled after the blog template that lists each news feed update, which is added to the context dict through the retrieveFeed method.


	= Subscribe via RSS =

	I've added a new url_path 'subscribe', so that a URL for requesting an ATOM feed would look like this:

	http://melange.appspot.com/site/subscribe/somesite

	The new URL pattern is configured in soc.views.helper.params:

	{{{
	if params.get('subscribe'):
	new_params['django_patterns_defaults'] += [
	(r'^%(url_name)s/(?P<access_type>subscribe)/%(key_fields)s$',
	'%(module_package)s.%(module_name)s.subscribe',
	'Subscribe to %(name_plural)s')]
	}}}


	This address is added to pages with a meta tag:

	<link rel="alternate" type="application/rss+xml" title="RSS" href="{{ feed_url }}"/>

	There is also a graphic subscribe icon on the corner of the newsfeed GUI module.

	The 'subscribe' path will also be the interface for any PubSub interactions, but those will only be activated on POST requests with a callback URL, otherwise the ATOM feed will be returned as normal.


	= Notification Integration =

	The notifications module has been in need of work, and I've refactored it so that it has two new beneficial features:

	* Deploy to multiple contexts (the newsfeed widget, an ATOM feed for the user w/ pubsub support, and information-enriched private e-mails).

	* Any entity with a newsfeed can now be referenced in the notification so that that entity's newsfeed will be updated, etc.


	== Notification TODO ==

	* Refactor access handler to be able to perform reflective rights-based queries.
	* Improve notifications front-end
	* Licensing issues for images?
	* Improve e-mail format to take better advantage of entity-tagging in notes.
	* Tag multiple entities in notification.


	= E-mail Tasks =

	When a feed item is created, it schedules a handler to send out e-mail notification. This handler calls the sendFeedItemEmailNotifications method that fetches a list of users who are subscribed to the sender entity and e-mails them.

	E-mails are now each sent via the mail_dispatcher task. Since it is very common to get OverQuotaErrors, this task API should always be used for sending e-mails.


	= User Subscriptions =


	The subscriptions model contains a relational index entity for the user, containing a list of entity keys that the user is subscribed to. From their edit_profile page, the user can set a global preference for whether they are accepting e-mail notifications, and the user can also configure individual entity preferences from those pages using the subscribe-by-star UI pattern.


	= PubSubHubbub Support =

	<wiki:video url="http://www.youtube.com/watch?v=ewQBgbysSOQ"/>

	This demo demonstrates Google Reader updating in seconds after a blog is updated.

	It should be fairly trivial to integrate push notification support, and abstract it so that it could also be accessed by other Melange features.

	== TODO ==


	== Subscription Handler ==

	The subscriptions logic contains an UpdateLogic class that is meant to automatically compile a list of users with read-access to an arbitrary entity. Unfortunately, setting subscriptions manually is the only option right now, since the automatic subscription handler is not nearly where I hoped it would be by now. The idea is very simple - just
	subscribe users to the entities where they have read-access. But the reality
	has been much more difficult to do without creating an entirely new
	infrastructure, and I'm entertaining the idea of simply relying on a manual
	interface (subscribe-by-star). The takeaway from this is that perhaps the
	access component should be abstracted to a high-level API, because it
	appears to have gained dependencies and complexities that prevent it from
	being useful to the news feed logic.

	== Post Commit Hooks ==

	Since most soc projects involve a code repository, it would be useful to allow repository commits to enter the news feed. This is already possible through using an RSS feed, but real-time updates could be generated by using the Google Code Post Commit Hook Support:

	http://code.google.com/p/support/wiki/PostCommitWebHooks

	This would require project owners to change a setting on their projects, and would be fairly easy to implement.