| #summary 'Sandworm' Newsfeed Module | |
| #labels Importance-Useful,Phase-Requirements,Contents-Draft | |
| [http://github.com/jamslevy/Melange/tree/master Source Code Repository] | |
| <wiki:toc max_depth="3" /> | |
| = Introduction = | |
| A framework for customizable news feeds for various entity kinds that can be formatted on various site pages. Each newsfeed includes a subscription link, and through a 'subscribe' url pattern is used to render XML feeds. | |
| Because of the relatively simple nature of the initial project, some of the post-midterm development will include an auxiliary open source pubsub component meant to be deployed separately, and act as a hub allowing Melange instances to host chat, Google Wave updates, and other real-time activity. | |
| = Newsfeed Model = | |
| An individual FeedItem entity is created for each news feed update. A garbage collection task will delete old feed items, so they don't take over the datastore. | |
| The FeedItem model contains the following properties: | |
| * *sender* (db.Key) - The sender is the entity that was updated | |
| * *receivers* (db.Key) - A list of entities whose news feed is being updated. | |
| * *update_type* (created, deleted, modified) | |
| * *created* (datetime) | |
| A FeedItem is created for each sender-receiver pair. For instance, if a document is created and saved, it is the sender while its associated project and program and user could all be receivers. | |
| = Access Protection = | |
| Very basic entity information (a title/name and a link) is shown in the Recent Updates box and the ATOM feed. This may violate access protection, and the alternative is to just show the name of the entity kind, or a truncated title. | |
| Because the mail subscriptions are privately sent on a per-user basis, they contain more detailed information about the updated entity. | |
| = Logic = | |
| Newsfeed logic consists of two public methods: | |
| * *addToFeed* takes a sender entity, a list of receiver entities, an update type, and an optional payload (like a text preview of a document). It adds all the receivers to a FeedItem with the specified payload: | |
| {{{ | |
| for receiver in receivers: | |
| new_feed_item = soc.models.news_feed.FeedItem( | |
| sender_key= str(sender.key()), | |
| receiver_key = str(receiver.key()), | |
| user = user, | |
| update_type = update_type | |
| ) | |
| }}} | |
| * *retrieveFeed* takes a receiever entity, and retrieves it's latest news feed updates, with an optional count kwarg. This method is cached, with a cache handler in the /cache folder. | |
| {{{ | |
| def retrieveFeed(self, entity, count=10): | |
| """ Retrieves feed for a given entity | |
| """ | |
| feed_items = soc.models.news_feed.FeedItem.all().filter( | |
| "receiver_key =", str(entity.key())).fetch(1000) | |
| return feed_items[:count] | |
| }}} | |
| = Views = | |
| The newsfeed template itself is simply a table modeled after the blog template that lists each news feed update, which is added to the context dict through the retrieveFeed method. | |
| = Subscribe via RSS = | |
| I've added a new url_path 'subscribe', so that a URL for requesting an ATOM feed would look like this: | |
| http://melange.appspot.com/site/subscribe/somesite | |
| The new URL pattern is configured in soc.views.helper.params: | |
| {{{ | |
| if params.get('subscribe'): | |
| new_params['django_patterns_defaults'] += [ | |
| (r'^%(url_name)s/(?P<access_type>subscribe)/%(key_fields)s$', | |
| '%(module_package)s.%(module_name)s.subscribe', | |
| 'Subscribe to %(name_plural)s')] | |
| }}} | |
| This address is added to pages with a meta tag: | |
| <link rel="alternate" type="application/rss+xml" title="RSS" href="{{ feed_url }}"/> | |
| There is also a graphic subscribe icon on the corner of the newsfeed GUI module. | |
| The 'subscribe' path will also be the interface for any PubSub interactions, but those will only be activated on POST requests with a callback URL, otherwise the ATOM feed will be returned as normal. | |
| = Notification Integration = | |
| The notifications module has been in need of work, and I've refactored it so that it has two new beneficial features: | |
| * Deploy to multiple contexts (the newsfeed widget, an ATOM feed for the user w/ pubsub support, and information-enriched private e-mails). | |
| * Any entity with a newsfeed can now be referenced in the notification so that that entity's newsfeed will be updated, etc. | |
| == Notification TODO == | |
| * Refactor access handler to be able to perform reflective rights-based queries. | |
| * Improve notifications front-end | |
| * Licensing issues for images? | |
| * Improve e-mail format to take better advantage of entity-tagging in notes. | |
| * Tag multiple entities in notification. | |
| = E-mail Tasks = | |
| When a feed item is created, it schedules a handler to send out e-mail notification. This handler calls the sendFeedItemEmailNotifications method that fetches a list of users who are subscribed to the sender entity and e-mails them. | |
| E-mails are now each sent via the mail_dispatcher task. Since it is very common to get OverQuotaErrors, this task API should always be used for sending e-mails. | |
| = User Subscriptions = | |
| The subscriptions model contains a relational index entity for the user, containing a list of entity keys that the user is subscribed to. From their edit_profile page, the user can set a global preference for whether they are accepting e-mail notifications, and the user can also configure individual entity preferences from those pages using the subscribe-by-star UI pattern. | |
| = PubSubHubbub Support = | |
| <wiki:video url="http://www.youtube.com/watch?v=ewQBgbysSOQ"/> | |
| This demo demonstrates Google Reader updating in seconds after a blog is updated. | |
| It should be fairly trivial to integrate push notification support, and abstract it so that it could also be accessed by other Melange features. | |
| == TODO == | |
| == Subscription Handler == | |
| The subscriptions logic contains an UpdateLogic class that is meant to automatically compile a list of users with read-access to an arbitrary entity. Unfortunately, setting subscriptions manually is the only option right now, since the automatic subscription handler is not nearly where I hoped it would be by now. The idea is very simple - just | |
| subscribe users to the entities where they have read-access. But the reality | |
| has been much more difficult to do without creating an entirely new | |
| infrastructure, and I'm entertaining the idea of simply relying on a manual | |
| interface (subscribe-by-star). The takeaway from this is that perhaps the | |
| access component should be abstracted to a high-level API, because it | |
| appears to have gained dependencies and complexities that prevent it from | |
| being useful to the news feed logic. | |
| == Post Commit Hooks == | |
| Since most soc projects involve a code repository, it would be useful to allow repository commits to enter the news feed. This is already possible through using an RSS feed, but real-time updates could be generated by using the Google Code Post Commit Hook Support: | |
| http://code.google.com/p/support/wiki/PostCommitWebHooks | |
| This would require project owners to change a setting on their projects, and would be fairly easy to implement. |