#summary Melange Module Architecture
#labels Phase-Implementation,Importance-Useful,Contents-Draft

<wiki:toc max_depth="2" />

= General idea =

This section briefly summarises the functionality and design of the new architecture.
Naming

 * prefix (or at least a pattern somewhere in the url) for all urls that use a module
   * (?Pmodule gsoc)/  
   * (?Pmodule ghop)/
 * load the appropriate "soc.modules.%(module)s" module
 * modules live under the soc.modules package
 * everything that is part of Melange should live under the soc package (so no more /app/gsoc, /app/ghop/, /app/soc structure)
 * in build.py all modules as defined by settings.MODULES are loaded, and the callbacks they register are called at the appropriate times

a top level "module" that is "core" to do site wide stuff, such as "soc/user/create"
generic urls such as 'program/create' are removed and replaced by e.g. 'gsoc/program/create'

At a pinch we could just go with the naming without introducing a communication structure.  However this is a route to spaghetti code and great ambiguity in meaning of objects.  For example, what does 'App_task_limit' mean?  It could depend on the context, GHOP or GSoC.  If we restrict the communication to well designed channels, i.e. if we genuinely keep the design modular, these kinds of problems are less likely.

= Communications =

 * each module has it's own entities, that may inherit from generic entities, e.g., GHOPProgram extends Program.  Every item in 'Program' will need a comment to ensure that its meaning is clear and that it is not specific to GHOP or GSoC.
 * version compatibility will be achieved with a single API_VERSION number
 * modules can register services they offer, which can then be used by other modules
 * modules can require the presence of certain services (optionally provided by other modules) and refuse to load if these services are not present
modules talk with the core only, communication with other modules happens through the core by means of services
 * the core provides an API for:
  * registering new ACL checks
  * register new sidebar entries
  * register new urls with the sitemap
  * registering jobs that the module would like to be schedulable
  * registering user preferences the plugin would like added

= Possible Modules =

In the future we might have the following modules (this is not a roadmap, but is meant to give an impression what kind of modules we want to support).

 * gsoc module
 * ghop module
 * wiki module (wiki support with diff  tracking etc)
 * issue tracker module (could share some functionality with ghop)

<Pawel> issue tracker module ? what would it be used for ?</Pawel>
<Sverre> For example, there are no issue trackers "out there" that allow you to group issues under a main issue, currently we mark those issues as Duplicate, while instead they should be grouped. Also, rather than having a 'comment based' issue tracker, a 'wiki based' issue tracker with voting support would be a lot nicer. As said, this is not a roadmap, so I'm not saying this should be implemented by us, but I know that others _would_ want to implement it :).
</Sverre>

 * statistics module
 * survey module

= Conversion =

This section describes the conversion process from the pre-module architecture to the new module architecture.

== Time cost ==

Estimated is that the implementation of a basic framework as described in this document will take a full weekend (with no other obligations) by two developers.

<Pawel>I think it will take longer but we can bet :-) and work syncing of those two developers will be also a problem. How are you going to divide work on this new architecture between two developers ?</Pawel>

<Sverre>As you can see from the code below most of the design is already done. Dividing the work is easy, each developer focusses on one method of the API (e.g., one working on sidebar, one on sitemap). </Sverre>

The full conversion of the current system to the new architecture (e.g., so that the core uses the core callback to register it's sidebar and sitemap entries, etc) is estimated to take a week (with no other obligations) by two developers.

== Converting the datastore ==

 # push the new models
 # convert old models to new models
 # remove the old models

= Terminology =

Core: The code module that runs when Melange starts up.  It is responsible for invoking any other modules.  It contains a built in 'module' with name 'noplugin' that does things we currently expect to be required in every program.  It may turn out that later some of these move out into modules, but that is not planned at this stage.

Module: A related body of code that can register its services and requirements with 'Core'.   

= Registration Order And API =

The order in which the core invites registrations from the modules could be important at a future date.  For GHoP and GSoC the order is unlikely to be important.  We can address the issue of 'order' at a later date.

The precise details of what we want to register are also likely to grow over time.  This is our 'plug in API'.  This is the correct way to develop - organically.  We have a structure in which this can happen.

By means of the 'register capability' and 'require capability' calls described below modules can ensure that after all modules have loaded successfully each module has it's required capabilities satisfied. This way there is no need for modules to depend on one another, and order does not matter.

= Code snippets =

This section contains some general code snippets that are related to the new architecture.  

== Server side code ==

Some server side code.  This lives in the core.

{{{
def hasCapability(callback, capability):
 """Checks whether a callback support the specified capability.

 Args:
   callback: the callback to check if it supports the capability
   capability: the capability to check for
 """

 if not hasattr(callback, capability):
   return False

 func = getattr(callback, capability)

 if not callable(func):
   return False

 return True
}}}

== Callback API ==

Some callback API code.  Variants of this live in each module.

{{{
class Callback(object):
  """Callback object that handles interaction between the core and a module.
  """

  def __init__(self, core):
    """Initializes a new Callback object for the specified core.
    """

    self.core = core
    self.core.registerService(...)

  def registerWithSitemap(self):
    """Called by the server when sitemap entries should be registered.
    """

   # register our sitemap entries
    self.core.registerSitemapEntry(...)
    ...

  def registerWithSidebar(self):
    """Called by the server when sidebar entries should be registered.
    """

    # require that we had the chance to register the urls we need with the sitemap
    self.core.requireService('registerWithSitemap')

    # register our sidebar entries
    self.core.registerSidebarEntry()
}}}

== Bootstrapping ==

In order to get everything running build.py will be extended to load all modules defined in settings.MODULES. The MODULE_FMT is 'soc.modules.%s' for Melange.  The module list will be stored in a global and used from a number of places in core for different capabilities.  There will also be a subroutine for calling the callbacks for a particular capability (not shown).

One consequence of the design is that at some date in the future, if a capability has been superseded, the core no longer needs to support it.  It will not ask for that capability and callbacks will not need to supply it.

Core itself should also be part of the callbacks and use the module API to do stuff.

{{{
def getModuleCallbacks():
  """Retrieves all callbacks for the modules of this site.
  callbacks for modules without a version number or the wrong API_VERSION
  number are dropped.  They won't be called.
  """

  fmt = settings.MODULE_FMT
  modules = [__import__(fmt % i, fromlist=['']) for i in settings.MODULES]

  return [i.getCallback() for i in modules]
}}}

add in somewhere... core.triggerCapability('registerWithSitemap')

= Sharing of Models =

Currently many models are shared while they should not be.

== Shared Models ==

These are all in core
 * answer.py    
 * base.py
 * club.py
 * club_admin.py    
 * club_app.py    
 * club_member.py
 * contributor.py    
 * comment.py   
 * countries.py   
 * document.py    
 * documentation.py    
 * follower.py    
 * group.py    
 * group_app.py    
 * host.py    
 * job.py    
 * licenses.py    
 * linkable.py
 * notification.py
 * presence.py    
 * presence_with_tos.py    
 * question.py    
 * quiz.py    
 * request.py    
 * response.py    
 * reviewer.py    
 * role.py
 * school.py    
 * site.py    
 * sponsor.py    
 * user.py
 * work.py
 * priority_group.py  

== Shared But Different (Oops) ==

Looks like these need a base class and then to have specialised GHOP and GSoC versions.

 * mentor.py    
 * org_admin.py    
 * organization.py    
 * program.py   
 * timeline.py
 * student.py
 * seed_db.py   

 * org_app.py (Might be replaced by surveys)

== GSoC Specific Models ==
 
 * proposal.py    
 * proposal_duplicates.py   
 * ranker_root.py 
 * review.py    
 * review_follower.py    
 * student_project.py    
 * student_proposal.py  
 * project_survey.py
 * project_survey_record.py
 * grading_project_survey.py
 * grading_project_survey_record.py 

== GHOP Specific Models ==

 * task.py (Needs to be redone because it was wrong)

= Model Conversion Process =

== The plan ==
Converting to the new module system is going to be a multi-step process. This way we can have more control over the data. We will also be keeping all the old data until the conversion process is complete. 

The steps are outlined below:

1. Convert the current Program entities to GSoCProgram entities keeping the same keyname.

2. Convert all Organization entities to GSoCOrgs keeping their repsective keynames but with updating the Program reference (scope) to the respective GSoCProgram. The GSoCProgram can be easily looked up since it will have the same keyname as the Program entity.

3. Update all OrgAdmins to GSoCOrgAdmins and change their scope to reference to the new GSoCOrganization and the program reference to the new GSoCProgram.

4. The same as 3 but just for Mentors -> GSoCMentors.

5. Convert the Student entities to GSoCStudent entities and update their Program reference to contain the new GSoCProgram.

*At this point we are done with creating new entities, however several references still need to be updated*

6. Update the StudentProposal entities with the following references:
  * scope should be set to the new GSoCStudent
  * mentor should be set to the new GSoCMentor
  * possible_mentors should only contain GSoCMentors
  * org property should be updated to contain GSoCOrganization
  * program property should be updated to contain the new GSoCProgram

7. Update the Review entities. We should look into deleting the role reference property or otherwise we need to ensure that we convert the Students to GSoCStudents, Mentors to GSoCMentors and OrgAdmins to GSoCOrgAdmins.

8. Update the StudentProject entities with the following references:
  * scope should be set to the new GSoCOrganization
  * mentor should be updated with the new GSoCMentor
  * additional_mentors should be updated with the new GSoCMentor entities
  * student property should be updated to contain the new GSoCStudent reference.
  * program property should be updated to contain the new GSoCProgram reference

9. Update the Survey, ProjectSurvey and GradingProjectSurveys:
  * For prefix == 'program' the scope entity should be updated to contain the new GSoCProgram model.  All other prefixes should have never been in use.

10. Update the ProjectSurveyRecords and GradingProjectSurveyRecords org property to reference the new GSoCOrganization entity.

11. Update the GradingSurveyGroup model with a reference to the new GSoCProgram.

12. Update the Document model:
  - For prefix == 'org' the scope should be updated to contain the new GSoCOrganization entity. If home_for is set this should be updated to the new GSoCOrganization as well.
  - For prefix =='program' the scope should be updated to contain the new GSoCProgram enity. Same goes for the home_for property.

13. Move the app.gsoc.Timeline model entries to module GSoCTimeline and update the scope property with the new GSoCProgram entity. This one goes last because it might need to happen in two steps.

*The move to the new module system is complete*

Optional:

14. Change proposal to have the organization as scope and student as a separate property. This would require updating the references from the reviews. 

15. Might also be good to move Reviews to an id_based system as well as Comments. This will help with change 12 as well as make it a more generic system.

16. Rework document system to contain the prefix internally and work on an id-based system. This will make it easier to move documents between scopes and also makes it easier for the system for use within the module approach.

== The execution ==
Before we embark on this journey of change literally the largest parts of our data store it might be worthwhile to discuss how this process is executed.

As you can see there are clean steps in this process and that is something that can be useful when changing the data. It gives one a point of rest to check wether or not the conversion is complete and it is something that can be done again if necessary.

My proposal would be to create a Conversion module with a View and appropriate conversion tasks. The view has one simple page which shows you the conversions you can execute when you update from version X. The conversions are all accompanied by a button which sends POST?? data to the view that translates it into the correct task that needs to be fired, fires that task and the conversion is under way.